swh.objstorage.backends.libcloud module

swh.objstorage.backends.libcloud.patch_libcloud_s3_urlencode()

Patches libcloud’s S3 backend to properly sign queries.

Recent versions of libcloud are not affected (they use signature V4), but 1.5.0 (the one in Debian 9) is.

class swh.objstorage.backends.libcloud.CloudObjStorage(container_name: str, compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] = 'gzip', path_prefix: str | None = None, **kwargs)

Bases: ObjStorage

Abstract ObjStorage that connect to a cloud using Libcloud

Implementations of this class must redefine the _get_provider method to make it return a driver provider (i.e. object that supports get_driver method) which return a LibCloud driver (see https://libcloud.readthedocs.io/en/latest/storage/api.html).

Parameters:

• container_name – Name of the base container

• path_prefix – prefix to prepend to object paths in the container, separated with a slash

• compression – compression algorithm to use for objects

• kwargs – extra arguments are passed through to the LibCloud driver

PRIMARY_HASH: Literal['sha1', 'sha256'] = 'sha1'

name: str = 'cloud'

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor

check_config(*, check_write)

Check the configuration for this object storage

add(content: bytes, obj_id: bytes | CompositeObjId, check_presence: bool = True) → None

restore(content: bytes, obj_id: bytes | CompositeObjId) → None

get(obj_id: bytes | CompositeObjId) → bytes

download_url(obj_id: bytes | CompositeObjId, content_disposition: str | None = None, expiry: timedelta | None = None) → str | None

delete(obj_id: bytes | CompositeObjId)

class swh.objstorage.backends.libcloud.AwsCloudObjStorage(container_name: str, compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] = 'gzip', path_prefix: str | None = None, **kwargs)

Bases: CloudObjStorage

Amazon’s S3 Cloud-based object storage

name: str = 's3'

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor

download_url(obj_id: bytes | CompositeObjId, content_disposition: str | None = None, expiry: timedelta | None = None) → str | None

class swh.objstorage.backends.libcloud.OpenStackCloudObjStorage(container_name: str, compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] = 'gzip', path_prefix: str | None = None, **kwargs)

Bases: CloudObjStorage

OpenStack Swift Cloud based object storage

name: str = 'swift'

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor