swh.objstorage.backends.winery.objstorage module

class swh.objstorage.backends.winery.objstorage.WineryObjStorage(**kwargs)

Bases: ObjStorage

PRIMARY_HASH: Literal['sha1', 'sha256'] = 'sha256'

uninit()

get(obj_id: bytes | CompositeObjId) → bytes

check_config(*, check_write: bool) → bool

add(content: bytes, obj_id: bytes | CompositeObjId, check_presence: bool = True) → None

delete(obj_id: bytes | CompositeObjId)

class swh.objstorage.backends.winery.objstorage.WineryBase(**kwargs)

Bases: object

init()

uninit()

class swh.objstorage.backends.winery.objstorage.WineryReader(**kwargs)

Bases: WineryBase

roshard(name) → ROShard | None

rwshard(name) → RWShard

get(obj_id: bytes | CompositeObjId) → bytes

uninit()

swh.objstorage.backends.winery.objstorage.pack(shard, shared_base=None, clean_immediately=False, **kwargs)

swh.objstorage.backends.winery.objstorage.cleanup_rw_shard(shard, shared_base=None, **kwargs)

class swh.objstorage.backends.winery.objstorage.WineryWriter(**kwargs)

Bases: WineryReader

init()

uninit()

add(content: bytes, obj_id: bytes | CompositeObjId, check_presence: bool = True) → None

delete(obj_id: bytes | CompositeObjId)

check(obj_id: bytes | CompositeObjId) → None

pack()

swh.objstorage.backends.winery.objstorage.never_stop(_: int) → bool

swh.objstorage.backends.winery.objstorage.stop_after_shards(max_shards_packed: int) → Callable[[int], bool]

swh.objstorage.backends.winery.objstorage.shard_packer(base_dsn: str, shard_dsn: str, shard_max_size: int, throttle_read: int, throttle_write: int, application_name: str | None = None, rbd_pool_name: str = 'shards', rbd_data_pool_name: str | None = None, rbd_image_features_unsupported: ~typing.Tuple[str, ...] = (), rbd_use_sudo: bool = True, rbd_create_images: bool = True, rbd_wait_for_image: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>, output_dir: str | None = None, stop_packing: ~typing.Callable[[int], bool] = <function never_stop>, wait_for_shard: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>) → int

Pack shards until the stop_packing function returns True.

When no shards are available for packing, call the wait_for_shard function.

Parameters:

• base_dsn – PostgreSQL dsn for the shared database

• shard_dsn – PostgreSQL dsn for the individual shard databases

• shard_max_size – Max size of a shard (used to size new shards)

• throttle_read – reads per second

• throttle_write – writes per second

• application_name – the application name sent to PostgreSQL

• rbd_create_images – create images directly (or wait for RBD mapper)

• rbd_wait_for_image – sleep function called to wait for an image (when rbd_create_images`=`False)

• rbd_* – passed directly to roshard.Pool

• output_dir – output directory for statistics

• stop_packing – callback to determine whether the packer should exit

• wait_for_shard – sleep function called when no shards are available to be packed

swh.objstorage.backends.winery.objstorage.rw_shard_cleaner(base_dsn: str, shard_dsn: str, min_mapped_hosts: int, application_name: str | None = None, stop_cleaning: ~typing.Callable[[int], bool] = <function never_stop>, wait_for_shard: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>) → int

Clean up RW shards until the stop_cleaning function returns True.

When no shards are available for packing, call the wait_for_shard function.

Parameters:

• base_dsn – PostgreSQL dsn for the shared database

• shard_dsn – PostgreSQL dsn for the individual shard databases

• min_mapped_hosts – how many hosts should have mapped the image read-only before cleaning it

• application_name – the application name sent to PostgreSQL

• stop_cleaning – callback to determine whether the cleaner should exit

• wait_for_shard – sleep function called when no shards are available to be cleaned

swh.objstorage.backends.winery.objstorage.deleted_objects_cleaner(base: SharedBase, pool: Pool, stop_running: Callable[[], bool])

Clean up deleted objects from RO shards and the shared database.

This requires the ability to map RBD images in read-write mode. Images will be left mapped by this process as it is meant to be executed in a transient host dedicated to this purpose.

Parameters:

• base_dsn – PostgreSQL dsn for the shared database

• pool – Ceph RBD pool for Winery shards

• stop_running – callback that returns True when the manager should stop running