swh.objstorage.backends.libcloud module#

swh.objstorage.backends.libcloud.patch_libcloud_s3_urlencode()[source]#

Patches libcloud’s S3 backend to properly sign queries.

Recent versions of libcloud are not affected (they use signature V4), but 1.5.0 (the one in Debian 9) is.

class swh.objstorage.backends.libcloud.CloudObjStorage(container_name: str, compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] = 'gzip', path_prefix: str | None = None, **kwargs)[source]#

Bases: ObjStorage

Abstract ObjStorage that connect to a cloud using Libcloud

Implementations of this class must redefine the _get_provider method to make it return a driver provider (i.e. object that supports get_driver method) which return a LibCloud driver (see https://libcloud.readthedocs.io/en/latest/storage/api.html).

Parameters:

  • container_name – Name of the base container

  • path_prefix – prefix to prepend to object paths in the container, separated with a slash

  • compression – compression algorithm to use for objects

  • kwargs – extra arguments are passed through to the LibCloud driver

PRIMARY_HASH: Literal['sha1', 'sha256'] = 'sha1'#
name: str = 'cloud'#

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor

check_config(*, check_write)[source]#

Check the configuration for this object storage

add(content: bytes, obj_id: bytes | CompositeObjId, check_presence: bool = True) → None[source]#
restore(content: bytes, obj_id: bytes | CompositeObjId) → None[source]#
get(obj_id: bytes | CompositeObjId) → bytes[source]#
download_url(obj_id: bytes | CompositeObjId, content_disposition: str | None = None, expiry: timedelta | None = None) → str | None[source]#
delete(obj_id: bytes | CompositeObjId)[source]#
class swh.objstorage.backends.libcloud.AwsCloudObjStorage(container_name: str, compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] = 'gzip', path_prefix: str | None = None, **kwargs)[source]#

Bases: CloudObjStorage

Amazon’s S3 Cloud-based object storage

name: str = 's3'#

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor

download_url(obj_id: bytes | CompositeObjId, content_disposition: str | None = None, expiry: timedelta | None = None) → str | None[source]#
class swh.objstorage.backends.libcloud.OpenStackCloudObjStorage(container_name: str, compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] = 'gzip', path_prefix: str | None = None, **kwargs)[source]#

Bases: CloudObjStorage

OpenStack Swift Cloud based object storage

name: str = 'swift'#

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor