swh.objstorage.backends.winery.objstorage module#

class swh.objstorage.backends.winery.objstorage.WineryObjStorage(**kwargs)[source]#

Bases: ObjStorage

PRIMARY_HASH: Literal['sha1', 'sha256'] = 'sha256'#
name: str = 'winery'#

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor

get(obj_id: bytes | CompositeObjId) bytes[source]#
check_config(*, check_write: bool) bool[source]#
add(content: bytes, obj_id: bytes | CompositeObjId, check_presence: bool = True) None[source]#
delete(obj_id: bytes | CompositeObjId)[source]#
on_shutdown()[source]#
class swh.objstorage.backends.winery.objstorage.WineryBase(**kwargs)[source]#

Bases: object

on_shutdown()[source]#
class swh.objstorage.backends.winery.objstorage.WineryReader(**kwargs)[source]#

Bases: WineryBase

roshard(name) ROShard | None[source]#
rwshard(name) RWShard[source]#
get(obj_id: bytes) bytes[source]#
swh.objstorage.backends.winery.objstorage.pack(shard, shared_base=None, clean_immediately=False, **kwargs) bool[source]#
swh.objstorage.backends.winery.objstorage.cleanup_rw_shard(shard, shared_base=None, **kwargs) bool[source]#
class swh.objstorage.backends.winery.objstorage.WineryWriter(pack_immediately: bool = True, clean_immediately: bool = True, rwshard_idle_timeout: float = 300, **kwargs)[source]#

Bases: WineryReader

release_shard(shard: RWShard | None = None, from_idle_handler: bool = False, new_state: ShardState = ShardState.STANDBY)[source]#

Release the currently locked shard

property shard#

Lock a shard to be able to use it. Release it after idle_timeout.

add(content: bytes, obj_id: bytes, check_presence: bool = True) None[source]#
delete(obj_id: bytes)[source]#
check(obj_id: bytes | CompositeObjId) None[source]#
pack(shard_name: str)[source]#
on_shutdown()[source]#
swh.objstorage.backends.winery.objstorage.never_stop(_: int) bool[source]#
swh.objstorage.backends.winery.objstorage.stop_after_shards(max_shards_packed: int) Callable[[int], bool][source]#
swh.objstorage.backends.winery.objstorage.shard_packer(base_dsn: str, shard_max_size: int, throttle_read: int, throttle_write: int, application_name: str | None = None, rbd_pool_name: str = 'shards', rbd_data_pool_name: str | None = None, rbd_image_features_unsupported: ~typing.Tuple[str, ...] = (), rbd_use_sudo: bool = True, rbd_map_options: str = '', rbd_create_images: bool = True, rbd_wait_for_image: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>, output_dir: str | None = None, stop_packing: ~typing.Callable[[int], bool] = <function never_stop>, wait_for_shard: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>) int[source]#

Pack shards until the stop_packing function returns True.

When no shards are available for packing, call the wait_for_shard function.

Parameters:
  • base_dsn – PostgreSQL dsn for the shared database

  • shard_max_size – Max size of a shard (used to size new shards)

  • throttle_read – reads per second

  • throttle_write – writes per second

  • application_name – the application name sent to PostgreSQL

  • rbd_create_images – create images directly (or wait for RBD mapper)

  • rbd_wait_for_image – sleep function called to wait for an image (when rbd_create_images`=`False)

  • rbd_* – passed directly to roshard.Pool

  • output_dir – output directory for statistics

  • stop_packing – callback to determine whether the packer should exit

  • wait_for_shard – sleep function called when no shards are available to be packed

swh.objstorage.backends.winery.objstorage.rw_shard_cleaner(base_dsn: str, min_mapped_hosts: int, application_name: str | None = None, stop_cleaning: ~typing.Callable[[int], bool] = <function never_stop>, wait_for_shard: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>) int[source]#

Clean up RW shards until the stop_cleaning function returns True.

When no shards are available for packing, call the wait_for_shard function.

Parameters:
  • base_dsn – PostgreSQL dsn for the shared database

  • min_mapped_hosts – how many hosts should have mapped the image read-only before cleaning it

  • application_name – the application name sent to PostgreSQL

  • stop_cleaning – callback to determine whether the cleaner should exit

  • wait_for_shard – sleep function called when no shards are available to be cleaned

swh.objstorage.backends.winery.objstorage.deleted_objects_cleaner(base: SharedBase, pool: Pool, stop_running: Callable[[], bool])[source]#

Clean up deleted objects from RO shards and the shared database.

This requires the ability to map RBD images in read-write mode. Images will be left mapped by this process as it is meant to be executed in a transient host dedicated to this purpose.

Parameters:
  • base_dsn – PostgreSQL dsn for the shared database

  • pool – Ceph RBD pool for Winery shards

  • stop_running – callback that returns True when the manager should stop running