swh.objstorage.backends.winery.throttler module

class swh.objstorage.backends.winery.throttler.LeakyBucket(total)

Bases: object

Leaky bucket that can contain at most total and leaks it within a second. If adding (with the add method) more than total per second, it will sleep until the bucket can be filled without overflowing.

The capacity of the bucket can be changed dynamically with the reset method. If the new capacity is lower than it previously was, the overflow is ignored.

reset(total)

add(count)

class swh.objstorage.backends.winery.throttler.BandwidthCalculator

Bases: object

Keeps a histogram (of length duration, defaults to 60) where each element is the number of bytes read or written within a second.

Only the last duration seconds are represented in the histogram: after each second the oldest element is discarded.

The add method is called to add a value to the current second.

The get method retrieves the current bandwidth usage which is the average of all values in the histogram.

add(count)

get()

class swh.objstorage.backends.winery.throttler.IOThrottler(name, **kwargs)

Bases: Database

Throttle IO (either read or write, depending on the name argument). The maximum speed in bytes is from the throttle_`name` argument and controlled by a LeakyBucket that guarantees it won’t go any faster.

Every sync_interval seconds the current bandwidth reported by the BandwidthCalculator instance is written into a row in a table shared with other instances of IOThrottler. The cumulated bandwidth of all other instances is retrieved from the same table. If it exceeds max_speed, the LeakyBucket instance is reset to only allow max_speed/(number of instances) so that the total bandwidth is shared equally between instances.

init_db()

download_info()

upload_info()

add(count)

sync()

maybe_sync()

class swh.objstorage.backends.winery.throttler.Throttler(**kwargs)

Bases: object

Throttle reads and writes to not exceed limits imposed by the thottle_read and throttle_write arguments, as measured by the cumulated bandwidth reported by each Throttler instance.

throttle_get(fun, key)

throttle_add(fun, obj_id, content)