swh.objstorage.backends.winery.roshard module

exception swh.objstorage.backends.winery.roshard.ShardNotMapped

Bases: Exception

class swh.objstorage.backends.winery.roshard.Pool(*args, **kwargs)

Bases: Protocol

image_exists(image: str) → bool

Check whether the named image exists (it does not have to be mapped)

image_mapped(image: str) → Literal['ro', 'rw'] | None

Check whether the image is already mapped, read-only or read-write

image_list() → List[str]

List all known images, mapped or not

image_path(image: str) → str

Return a path to the image, that can be opened with open().

image_create(image: str) → None

Create a new image named image and allocate the right amount of space.

image_map(image: str, options: str) → None

Map an image for use. Options can be “ro” to map the image read-only, or “rw” to map the image read-write.

image_unmap(image: str) → None

Unmap the image. Once this is done, the image is unavailable for use.

image_remap_ro(image: str)

class swh.objstorage.backends.winery.roshard.RBDPool(shard_max_size: int, rbd_use_sudo: bool = True, rbd_pool_name: str = 'shards', rbd_data_pool_name: str | None = None, rbd_image_features_unsupported: Tuple[str, ...] = (), rbd_map_options: str = '')

Bases: Pool

Manage a Ceph RBD pool for Winery shards.

Parameters:

• shard_max_size – max size of shard contents

• rbd_use_sudo – whether to use sudo for rbd commands

• rbd_pool_name – name of the pool used for RBD images (metadata)

• rbd_data_pool_name – name of the pool used for RBD images (data)

• rbd_image_features_unsupported – features not supported by the kernel mounting the rbd images

• rbd_map_options – options to pass to rbd device map, e.g. ms_mode=prefer-secure to connect to a ceph cluster with encryption enabled

POOL_CONFIG: Tuple[str, ...] = ('shard_max_size', 'rbd_use_sudo', 'rbd_pool_name', 'rbd_data_pool_name', 'rbd_image_features_unsupported', 'rbd_map_options')

classmethod from_kwargs(**kwargs) → RBDPool

Create a Pool from a set of arbitrary keyword arguments

run(*cmd: str) → Iterable[str]

Run the given command, and return its output as lines.

Return: the standard output of the run command

Raises: CalledProcessError if the command doesn’t exit with exit code 0.

rbd(*arguments: str) → Iterable[str]

Run rbd with the given arguments

image_exists(image: str)

Check whether the named image exists (it does not have to be mapped)

image_list()

List all known images, mapped or not

image_path(image: str) → str

Return a path to the image, that can be opened with open().

image_create(image: str)

Create a new image named image and allocate the right amount of space.

image_map(image: str, options: str)

Map an image for use. Options can be “ro” to map the image read-only, or “rw” to map the image read-write.

image_unmap(image: str)

Unmap the image. Once this is done, the image is unavailable for use.

class swh.objstorage.backends.winery.roshard.FileBackedPool(base_directory: Path, pool_name: str, shard_max_size: int)

Bases: Pool

File-backed pool for Winery shards mimicking a Ceph RBD pool.

Unmapped images are represented by setting the file permission to 0o000.

image_exists(image: str) → bool

Check whether the named image exists (it does not have to be mapped)

image_list() → List[str]

List all known images, mapped or not

image_path(image: str) → str

Return a path to the image, that can be opened with open().

image_create(image: str) → None

Create a new image named image and allocate the right amount of space.

image_map(image: str, options: str) → None

Map an image for use. Options can be “ro” to map the image read-only, or “rw” to map the image read-write.

image_unmap(image: str) → None

Unmap the image. Once this is done, the image is unavailable for use.

image_unmap_all() → None

swh.objstorage.backends.winery.roshard.pool_from_settings(shards_settings: Shards, shards_pool_settings: ShardsPool) → Pool

Return a Pool from the settings

swh.objstorage.backends.winery.roshard.record_shard_mapped(base: SharedBase, shard_name: str)

Record a shard as mapped, bailing out after a few attempts.

Multiple attempts are used to handle a race condition when two hosts attempt to record the shard as mapped at the same time. In this situation, one of the two hosts will succeed and the other one will fail, the sleep delay can be kept short and linear.

swh.objstorage.backends.winery.roshard.manage_images(pool: Pool, base_dsn: str, manage_rw_images: bool, wait_for_image: Callable[[int], None], stop_running: Callable[[], bool], only_prefix: str | None = None, application_name: str | None = None) → None

Manage RBD image creation and mapping automatically.

Parameters:

• base_dsn – the DSN of the connection to the SharedBase

• manage_rw_images – whether RW images should be created and mapped

• wait_for_image – function which is called at each loop iteration, with an attempt number, if no images had to be mapped recently

• stop_running – callback that returns True when the manager should stop running

• only_prefix – only map images with the given name prefix

• application_name – the application name sent to PostgreSQL

class swh.objstorage.backends.winery.roshard.ROShard(name, throttler, pool)

Bases: object

open()

get(key)

close()

static delete(pool, shard_name, obj_id)

class swh.objstorage.backends.winery.roshard.ROShardCreator(name: str, count: int, throttler: ~swh.objstorage.backends.winery.throttler.Throttler, pool: ~swh.objstorage.backends.winery.roshard.Pool, rbd_create_images: bool = True, rbd_wait_for_image: ~typing.Callable[[int], None] = <function sleep_exponential.<locals>.sleep>, **kwargs)

Bases: object

Helper for Read-Only shard creation.

Parameters:

• name – Name of the shard to be initialized

• count – Number of objects to provision in the shard

• throttler – An instance of a winery throttler

• rbd_create_images – whether the ROShardCreator should create the rbd image, or delegate to the rbd_shard_manager

• rbd_wait_for_image – function called when waiting for a shard to be mapped

• shard_max_size – the size of the shard, passed to Pool

• rbd_* – other RBD-related Pool arguments

zero_image_if_needed()

Check whether the image is empty, and zero it out if it’s not.

We really check only the first 1kB, as we assume that the SWHShard marker will have been written at the beginning of the image under all circumstances if the RO Shard creation has been interrupted.

add(content, obj_id)