swh.objstorage.backends.winery.housekeeping module#

exception swh.objstorage.backends.winery.housekeeping.AbortOperation[source]#

Bases: Exception

swh.objstorage.backends.winery.housekeeping.never_stop(_: int) → bool[source]#
swh.objstorage.backends.winery.housekeeping.stop_after_shards(max_shards_packed: int) → Callable[[int], bool][source]#
swh.objstorage.backends.winery.housekeeping.shard_packer(database: ~swh.objstorage.backends.winery.settings.Database, shards: ~swh.objstorage.backends.winery.settings.Shards, shards_pool: ~swh.objstorage.backends.winery.settings.ShardsPool, packer: ~swh.objstorage.backends.winery.settings.Packer | None = None, stop_packing: ~typing.Callable[[int], bool] = <function never_stop>, abort_packing: ~typing.Callable[[int], bool] = <function never_stop>, wait_for_shard: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>) → int[source]#

Pack shards until the stop_packing function returns True.

When no shards are available for packing, call the wait_for_shard function.

Parameters:

  • database – database settings (e.g. db connection string)

  • shards – shards settings (e.g. max_size)

  • shards_pool – shards pool settings (e.g. Ceph RBD settings)

  • packer – packer settings

  • stop_packing – callback to determine whether the packer should exit

  • abort_packing – callback to determine whether the packer should abort

  • wait_for_shard – sleep function called when no shards are available to be packed

swh.objstorage.backends.winery.housekeeping.pack(shard: str, base_dsn: str, packer_settings: ~swh.objstorage.backends.winery.settings.Packer, shards_settings: ~swh.objstorage.backends.winery.settings.Shards, shards_pool_settings: ~swh.objstorage.backends.winery.settings.ShardsPool, shared_base: ~swh.objstorage.backends.winery.sharedbase.SharedBase | None = None, abort_packing: ~typing.Callable[[int], bool] = <function never_stop>) → bool[source]#
swh.objstorage.backends.winery.housekeeping.rw_shard_cleaner(database: ~swh.objstorage.backends.winery.settings.Database, min_mapped_hosts: int, stop_cleaning: ~typing.Callable[[int], bool] = <function never_stop>, wait_for_shard: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>) → int[source]#

Clean up RW shards until the stop_cleaning function returns True.

When no shards are available for packing, call the wait_for_shard function.

Parameters:

  • database – database settings (e.g. db connection string)

  • min_mapped_hosts – how many hosts should have mapped the image read-only before cleaning it

  • stop_cleaning – callback to determine whether the cleaner should exit

  • wait_for_shard – sleep function called when no shards are available to be cleaned

swh.objstorage.backends.winery.housekeeping.cleanup_rw_shard(shard, base_dsn) → bool[source]#
swh.objstorage.backends.winery.housekeeping.deleted_objects_cleaner(base: SharedBase, pool: Pool, stop_running: Callable[[], bool])[source]#

Clean up deleted objects from RO shards and the shared database.

This requires the ability to map RBD images in read-write mode. Images will be left mapped by this process as it is meant to be executed in a transient host dedicated to this purpose.

Parameters:

  • base_dsn – PostgreSQL dsn for the shared database

  • pool – Ceph RBD pool for Winery shards

  • stop_running – callback that returns True when the manager should stop running

swh.objstorage.backends.winery.housekeeping.import_ro_shards(base: SharedBase, pool: Pool, shards: Iterable[str] | None = None) → Tuple[int, int][source]#

Import existing shard files in the winery database.