swh.objstorage.backends.winery.objstorage module#
- class swh.objstorage.backends.winery.objstorage.WineryObjStorage(database: Database, shards: Shards, shards_pool: ShardsPool, throttler: Throttler, packer: Packer | None = None, readonly: bool = False, allow_delete: bool = False, name: str = 'winery')[source]#
Bases:
ObjStorage
- PRIMARY_HASH: LiteralPrimaryHash = 'sha256'#
- name: str = 'winery'#
Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor
- get(obj_id: ObjId) bytes [source]#
Retrieve the content of a given object.
- Parameters:
obj_id – object id.
- Returns:
the content of the requested object as bytes.
- Raises:
ObjNotFoundError – if the requested object is missing.
- check_config(*, check_write: bool) bool [source]#
Check whether the object storage is properly configured.
- Parameters:
check_write (bool) – if True, check if writes to the object storage
succeed. (can)
- Returns:
True if the configuration check worked, False if ‘check_write’ is True and the object storage is actually read only, and an exception if the check failed.
- add(content: bytes, obj_id: ObjId, check_presence: bool = True) None [source]#
Add a new object to the object storage.
- Parameters:
content – object’s raw content to add in storage.
obj_id – either dict of checksums, or single checksum of [bytes] using [ID_HASH_ALGO] algorithm. It is trusted to match the bytes.
check_presence (bool) – indicate if the presence of the content should be verified before adding the file.
- Returns:
the id (bytes) of the object into the storage.
- delete(obj_id: ObjId)[source]#
Delete an object.
- Parameters:
obj_id – object identifier.
- Raises:
ObjNotFoundError – if the requested object is missing.
- list_content(last_obj_id: ObjId | None = None, limit: int | None = 10000) Iterator[ObjId] [source]#
Generates known object ids.
- Parameters:
last_obj_id – object id from which to iterate from (excluded).
limit (int) – max number of object ids to generate. If unset (None), generate all objects (behavior might not be guaranteed for all backends).
- Generates:
obj_id: object ids.
- class swh.objstorage.backends.winery.objstorage.WineryReader(throttler: Throttler, pool: Pool, database: Database)[source]#
Bases:
object
- swh.objstorage.backends.winery.objstorage.pack(shard: str, base_dsn: str, packer_settings: Packer, throttler_settings: Throttler | None, shards_settings: Shards, shards_pool_settings: ShardsPool, shared_base: SharedBase | None = None) bool [source]#
- swh.objstorage.backends.winery.objstorage.cleanup_rw_shard(shard, base_dsn=None, shared_base=None) bool [source]#
- class swh.objstorage.backends.winery.objstorage.WineryWriter(packer_settings: Packer, throttler_settings: Throttler | None, shards_settings: Shards, shards_pool_settings: ShardsPool, database_settings: Database)[source]#
Bases:
object
- release_shard(shard: RWShard | None = None, from_idle_handler: bool = False, new_state: ShardState = ShardState.STANDBY)[source]#
Release the currently locked shard
- property shard#
Lock a shard to be able to use it. Release it after
idle_timeout
.
- swh.objstorage.backends.winery.objstorage.stop_after_shards(max_shards_packed: int) Callable[[int], bool] [source]#
- swh.objstorage.backends.winery.objstorage.shard_packer(database: ~swh.objstorage.backends.winery.settings.Database, shards: ~swh.objstorage.backends.winery.settings.Shards, shards_pool: ~swh.objstorage.backends.winery.settings.ShardsPool, throttler: ~swh.objstorage.backends.winery.settings.Throttler, packer: ~swh.objstorage.backends.winery.settings.Packer | None = None, stop_packing: ~typing.Callable[[int], bool] = <function never_stop>, wait_for_shard: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>) int [source]#
Pack shards until the stop_packing function returns True.
When no shards are available for packing, call the wait_for_shard function.
- Parameters:
database – database settings (e.g. db connection string)
shards – shards settings (e.g. max_size)
shards_pool – shards pool settings (e.g. Ceph RBD settings)
throttler – throttler settings
packer – packer settings
stop_packing – callback to determine whether the packer should exit
wait_for_shard – sleep function called when no shards are available to be packed
- swh.objstorage.backends.winery.objstorage.rw_shard_cleaner(database: ~swh.objstorage.backends.winery.settings.Database, min_mapped_hosts: int, stop_cleaning: ~typing.Callable[[int], bool] = <function never_stop>, wait_for_shard: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>) int [source]#
Clean up RW shards until the stop_cleaning function returns True.
When no shards are available for packing, call the wait_for_shard function.
- Parameters:
database – database settings (e.g. db connection string)
min_mapped_hosts – how many hosts should have mapped the image read-only before cleaning it
stop_cleaning – callback to determine whether the cleaner should exit
wait_for_shard – sleep function called when no shards are available to be cleaned
- swh.objstorage.backends.winery.objstorage.deleted_objects_cleaner(base: SharedBase, pool: Pool, stop_running: Callable[[], bool])[source]#
Clean up deleted objects from RO shards and the shared database.
This requires the ability to map RBD images in read-write mode. Images will be left mapped by this process as it is meant to be executed in a transient host dedicated to this purpose.
- Parameters:
base_dsn – PostgreSQL dsn for the shared database
pool – Ceph RBD pool for Winery shards
stop_running – callback that returns True when the manager should stop running