swh.objstorage.backends.winery.objstorage module

class swh.objstorage.backends.winery.objstorage.WineryObjStorage(database: Database, shards: Shards, shards_pool: ShardsPool, throttler: Throttler, packer: Packer | None = None, readonly: bool = False, allow_delete: bool = False, name: str = 'winery')

Bases: ObjStorage

PRIMARY_HASH: LiteralPrimaryHash = 'sha256'

name: str = 'winery'

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor

get(obj_id: ObjId) → bytes

Retrieve the content of a given object.

Parameters:

obj_id – object id.

Returns:

the content of the requested object as bytes.

Raises:

ObjNotFoundError – if the requested object is missing.

check_config(*, check_write: bool) → bool

Check whether the object storage is properly configured.

Parameters:

• check_write (bool) – if True, check if writes to the object storage

• succeed. (can)

Returns:

True if the configuration check worked, False if ‘check_write’ is True and the object storage is actually read only, and an exception if the check failed.

add(content: bytes, obj_id: ObjId, check_presence: bool = True) → None

Add a new object to the object storage.

Parameters:

• content – object’s raw content to add in storage.

• obj_id – either dict of checksums, or single checksum of [bytes] using [ID_HASH_ALGO] algorithm. It is trusted to match the bytes.

• check_presence (bool) – indicate if the presence of the content should be verified before adding the file.

Returns:

the id (bytes) of the object into the storage.

delete(obj_id: ObjId)

Delete an object.

Parameters:

obj_id – object identifier.

Raises:

ObjNotFoundError – if the requested object is missing.

list_content(last_obj_id: ObjId | None = None, limit: int | None = 10000) → Iterator[ObjId]

Generates known object ids.

Parameters:

• last_obj_id – object id from which to iterate from (excluded).

• limit (int) – max number of object ids to generate. If unset (None), generate all objects (behavior might not be guaranteed for all backends).

Generates:

obj_id: object ids.

on_shutdown()

class swh.objstorage.backends.winery.objstorage.WineryReader(throttler: Throttler, pool: Pool, database: Database)

Bases: object

list_signatures(after_id: bytes | None = None, limit: int | None = None) → Iterator[bytes]

roshard(name) → ROShard | None

rwshard(name) → RWShard

get(obj_id: bytes) → bytes

on_shutdown()

swh.objstorage.backends.winery.objstorage.pack(shard: str, base_dsn: str, packer_settings: Packer, throttler_settings: Throttler | None, shards_settings: Shards, shards_pool_settings: ShardsPool, shared_base: SharedBase | None = None) → bool

swh.objstorage.backends.winery.objstorage.cleanup_rw_shard(shard, base_dsn=None, shared_base=None) → bool

class swh.objstorage.backends.winery.objstorage.WineryWriter(packer_settings: Packer, throttler_settings: Throttler | None, shards_settings: Shards, shards_pool_settings: ShardsPool, database_settings: Database)

Bases: object

release_shard(shard: RWShard | None = None, from_idle_handler: bool = False, new_state: ShardState = ShardState.STANDBY)

Release the currently locked shard

property shard

Lock a shard to be able to use it. Release it after idle_timeout.

add(content: bytes, obj_id: bytes) → None

delete(obj_id: bytes)

check(obj_id: ObjId) → None

pack(shard_name: str)

on_shutdown()

swh.objstorage.backends.winery.objstorage.never_stop(_: int) → bool

swh.objstorage.backends.winery.objstorage.stop_after_shards(max_shards_packed: int) → Callable[[int], bool]

swh.objstorage.backends.winery.objstorage.shard_packer(database: ~swh.objstorage.backends.winery.settings.Database, shards: ~swh.objstorage.backends.winery.settings.Shards, shards_pool: ~swh.objstorage.backends.winery.settings.ShardsPool, throttler: ~swh.objstorage.backends.winery.settings.Throttler, packer: ~swh.objstorage.backends.winery.settings.Packer | None = None, stop_packing: ~typing.Callable[[int], bool] = <function never_stop>, wait_for_shard: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>) → int

Pack shards until the stop_packing function returns True.

When no shards are available for packing, call the wait_for_shard function.

Parameters:

• database – database settings (e.g. db connection string)

• shards – shards settings (e.g. max_size)

• shards_pool – shards pool settings (e.g. Ceph RBD settings)

• throttler – throttler settings

• packer – packer settings

• stop_packing – callback to determine whether the packer should exit

• wait_for_shard – sleep function called when no shards are available to be packed

swh.objstorage.backends.winery.objstorage.rw_shard_cleaner(database: ~swh.objstorage.backends.winery.settings.Database, min_mapped_hosts: int, stop_cleaning: ~typing.Callable[[int], bool] = <function never_stop>, wait_for_shard: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>) → int

Clean up RW shards until the stop_cleaning function returns True.

When no shards are available for packing, call the wait_for_shard function.

Parameters:

• database – database settings (e.g. db connection string)

• min_mapped_hosts – how many hosts should have mapped the image read-only before cleaning it

• stop_cleaning – callback to determine whether the cleaner should exit

• wait_for_shard – sleep function called when no shards are available to be cleaned

swh.objstorage.backends.winery.objstorage.deleted_objects_cleaner(base: SharedBase, pool: Pool, stop_running: Callable[[], bool])

Clean up deleted objects from RO shards and the shared database.

This requires the ability to map RBD images in read-write mode. Images will be left mapped by this process as it is meant to be executed in a transient host dedicated to this purpose.

Parameters:

• base_dsn – PostgreSQL dsn for the shared database

• pool – Ceph RBD pool for Winery shards

• stop_running – callback that returns True when the manager should stop running