swh.objstorage.backends.winery.objstorage module#

class swh.objstorage.backends.winery.objstorage.WineryObjStorage(database: Database, shards: Shards, shards_pool: ShardsPool, throttler: Throttler, packer: Packer | None = None, readonly: bool = False, allow_delete: bool = False, name: str = 'winery')[source]#

Bases: ObjStorage

PRIMARY_HASH: LiteralPrimaryHash = 'sha256'#
name: str = 'winery'#

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor

get(obj_id: ObjId) → bytes[source]#

Retrieve the content of a given object.

Parameters:

obj_id – object id.

Returns:

the content of the requested object as bytes.

Raises:

ObjNotFoundError – if the requested object is missing.

check_config(*, check_write: bool) → bool[source]#

Check whether the object storage is properly configured.

Parameters:

  • check_write (bool) – if True, check if writes to the object storage

  • succeed. (can)

Returns:

True if the configuration check worked, False if ‘check_write’ is True and the object storage is actually read only, and an exception if the check failed.

add(content: bytes, obj_id: ObjId, check_presence: bool = True) → None[source]#

Add a new object to the object storage.

Parameters:

  • content – object’s raw content to add in storage.

  • obj_id – either dict of checksums, or single checksum of [bytes] using [ID_HASH_ALGO] algorithm. It is trusted to match the bytes.

  • check_presence (bool) – indicate if the presence of the content should be verified before adding the file.

Returns:

the id (bytes) of the object into the storage.

delete(obj_id: ObjId)[source]#

Delete an object.

Parameters:

obj_id – object identifier.

Raises:

ObjNotFoundError – if the requested object is missing.

list_content(last_obj_id: ObjId | None = None, limit: int | None = 10000) → Iterator[ObjId][source]#

Generates known object ids.

Parameters:

  • last_obj_id – object id from which to iterate from (excluded).

  • limit (int) – max number of object ids to generate. If unset (None), generate all objects (behavior might not be guaranteed for all backends).

Generates:

obj_id: object ids.

on_shutdown()[source]#
class swh.objstorage.backends.winery.objstorage.WineryReader(throttler: Throttler, pool: Pool, database: Database)[source]#

Bases: object

list_signatures(after_id: bytes | None = None, limit: int | None = None) → Iterator[bytes][source]#
roshard(name) → ROShard | None[source]#
rwshard(name) → RWShard[source]#
get(obj_id: bytes) → bytes[source]#
on_shutdown()[source]#
swh.objstorage.backends.winery.objstorage.pack(shard: str, base_dsn: str, packer_settings: Packer, throttler_settings: Throttler | None, shards_settings: Shards, shards_pool_settings: ShardsPool, shared_base: SharedBase | None = None) → bool[source]#
swh.objstorage.backends.winery.objstorage.cleanup_rw_shard(shard, base_dsn=None, shared_base=None) → bool[source]#
class swh.objstorage.backends.winery.objstorage.WineryWriter(packer_settings: Packer, throttler_settings: Throttler | None, shards_settings: Shards, shards_pool_settings: ShardsPool, database_settings: Database)[source]#

Bases: object

release_shard(shard: RWShard | None = None, from_idle_handler: bool = False, new_state: ShardState = ShardState.STANDBY)[source]#

Release the currently locked shard

property shard#

Lock a shard to be able to use it. Release it after idle_timeout.

add(content: bytes, obj_id: bytes) → None[source]#
delete(obj_id: bytes)[source]#
check(obj_id: ObjId) → None[source]#
pack(shard_name: str)[source]#
on_shutdown()[source]#
swh.objstorage.backends.winery.objstorage.never_stop(_: int) → bool[source]#
swh.objstorage.backends.winery.objstorage.stop_after_shards(max_shards_packed: int) → Callable[[int], bool][source]#
swh.objstorage.backends.winery.objstorage.shard_packer(database: ~swh.objstorage.backends.winery.settings.Database, shards: ~swh.objstorage.backends.winery.settings.Shards, shards_pool: ~swh.objstorage.backends.winery.settings.ShardsPool, throttler: ~swh.objstorage.backends.winery.settings.Throttler, packer: ~swh.objstorage.backends.winery.settings.Packer | None = None, stop_packing: ~typing.Callable[[int], bool] = <function never_stop>, wait_for_shard: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>) → int[source]#

Pack shards until the stop_packing function returns True.

When no shards are available for packing, call the wait_for_shard function.

Parameters:

  • database – database settings (e.g. db connection string)

  • shards – shards settings (e.g. max_size)

  • shards_pool – shards pool settings (e.g. Ceph RBD settings)

  • throttler – throttler settings

  • packer – packer settings

  • stop_packing – callback to determine whether the packer should exit

  • wait_for_shard – sleep function called when no shards are available to be packed

swh.objstorage.backends.winery.objstorage.rw_shard_cleaner(database: ~swh.objstorage.backends.winery.settings.Database, min_mapped_hosts: int, stop_cleaning: ~typing.Callable[[int], bool] = <function never_stop>, wait_for_shard: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>) → int[source]#

Clean up RW shards until the stop_cleaning function returns True.

When no shards are available for packing, call the wait_for_shard function.

Parameters:

  • database – database settings (e.g. db connection string)

  • min_mapped_hosts – how many hosts should have mapped the image read-only before cleaning it

  • stop_cleaning – callback to determine whether the cleaner should exit

  • wait_for_shard – sleep function called when no shards are available to be cleaned

swh.objstorage.backends.winery.objstorage.deleted_objects_cleaner(base: SharedBase, pool: Pool, stop_running: Callable[[], bool])[source]#

Clean up deleted objects from RO shards and the shared database.

This requires the ability to map RBD images in read-write mode. Images will be left mapped by this process as it is meant to be executed in a transient host dedicated to this purpose.

Parameters:

  • base_dsn – PostgreSQL dsn for the shared database

  • pool – Ceph RBD pool for Winery shards

  • stop_running – callback that returns True when the manager should stop running