swh.objstorage.backends.libcloud module

swh.objstorage.backends.libcloud.patch_libcloud_s3_urlencode()[source]

Patches libcloud’s S3 backend to properly sign queries.

Recent versions of libcloud are not affected (they use signature V4), but 1.5.0 (the one in Debian 9) is.

class swh.objstorage.backends.libcloud.CloudObjStorage(container_name: str, compression: Optional[str] = None, path_prefix: Optional[str] = None, **kwargs)[source]

Bases: swh.objstorage.objstorage.ObjStorage

Abstract ObjStorage that connect to a cloud using Libcloud

Implementations of this class must redefine the _get_provider method to make it return a driver provider (i.e. object that supports get_driver method) which return a LibCloud driver (see https://libcloud.readthedocs.io/en/latest/storage/api.html).

Parameters

  • container_name – Name of the base container

  • path_prefix – prefix to prepend to object paths in the container, separated with a slash

  • compression – compression algorithm to use for objects

  • kwargs – extra arguments are passed through to the LibCloud driver

check_config(*, check_write)[source]

Check the configuration for this object storage

add(content, obj_id=None, check_presence=True)[source]

Add a new object to the object storage.

Parameters

  • content (bytes) – object’s raw content to add in storage.

  • obj_id (bytes) – checksum of [bytes] using [ID_HASH_ALGO] algorithm. When given, obj_id will be trusted to match the bytes. If missing, obj_id will be computed on the fly.

  • check_presence (bool) – indicate if the presence of the content should be verified before adding the file.

Returns

the id (bytes) of the object into the storage.

restore(content, obj_id=None)[source]

Restore a content that have been corrupted.

This function is identical to add but does not check if the object id is already in the file system. The default implementation provided by the current class is suitable for most cases.

Parameters

  • content (bytes) – object’s raw content to add in storage

  • obj_id (bytes) – checksum of bytes as computed by ID_HASH_ALGO. When given, obj_id will be trusted to match bytes. If missing, obj_id will be computed on the fly.

get(obj_id)[source]

Retrieve the content of a given object.

Parameters

obj_id (bytes) – object id.

Returns

the content of the requested object as bytes.

Raises

ObjNotFoundError – if the requested object is missing.

check(obj_id)[source]

Perform an integrity check for a given object.

Verify that the file object is in place and that the content matches the object id.

Parameters

obj_id (bytes) – object identifier.

Raises
delete(obj_id)[source]

Delete an object.

Parameters

obj_id (bytes) – object identifier.

Raises

ObjNotFoundError – if the requested object is missing.

class swh.objstorage.backends.libcloud.AwsCloudObjStorage(container_name: str, compression: Optional[str] = None, path_prefix: Optional[str] = None, **kwargs)[source]

Bases: swh.objstorage.backends.libcloud.CloudObjStorage

Amazon’s S3 Cloud-based object storage

class swh.objstorage.backends.libcloud.OpenStackCloudObjStorage(container_name: str, compression: Optional[str] = None, path_prefix: Optional[str] = None, **kwargs)[source]

Bases: swh.objstorage.backends.libcloud.CloudObjStorage

OpenStack Swift Cloud based object storage