swh.objstorage.backends.libcloud module#

swh.objstorage.backends.libcloud.patch_libcloud_s3_urlencode()[source]#

Patches libcloud’s S3 backend to properly sign queries.

Recent versions of libcloud are not affected (they use signature V4), but 1.5.0 (the one in Debian 9) is.

class swh.objstorage.backends.libcloud.CloudObjStorage(container_name: str, compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] = 'gzip', path_prefix: str | None = None, **kwargs)[source]#

Bases: ObjStorage

Abstract ObjStorage that connect to a cloud using Libcloud

Implementations of this class must redefine the _get_provider method to make it return a driver provider (i.e. object that supports get_driver method) which return a LibCloud driver (see https://libcloud.readthedocs.io/en/latest/storage/api.html).

Parameters:

  • container_name – Name of the base container

  • path_prefix – prefix to prepend to object paths in the container, separated with a slash

  • compression – compression algorithm to use for objects

  • kwargs – extra arguments are passed through to the LibCloud driver

PRIMARY_HASH: LiteralPrimaryHash = 'sha1'#
name: str = 'cloud'#

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor

check_config(*, check_write)[source]#

Check the configuration for this object storage

add(content: bytes, obj_id: ObjId, check_presence: bool = True) → None[source]#

Add a new object to the object storage.

Parameters:

  • content – object’s raw content to add in storage.

  • obj_id – either dict of checksums, or single checksum of [bytes] using [ID_HASH_ALGO] algorithm. It is trusted to match the bytes.

  • check_presence (bool) – indicate if the presence of the content should be verified before adding the file.

Returns:

the id (bytes) of the object into the storage.

restore(content: bytes, obj_id: ObjId) → None[source]#

Restore a content that have been corrupted.

This function is identical to add but does not check if the object id is already in the file system. The default implementation provided by the current class is suitable for most cases.

Parameters:

  • content – object’s raw content to add in storage

  • obj_id – dict of hashes of the content (or only the sha1, for legacy clients)

get(obj_id: ObjId) → bytes[source]#

Retrieve the content of a given object.

Parameters:

obj_id – object id.

Returns:

the content of the requested object as bytes.

Raises:

ObjNotFoundError – if the requested object is missing.

download_url(obj_id: ObjId, content_disposition: str | None = None, expiry: timedelta | None = None) → str | None[source]#

Get a direct download link for the object if the obstorage backend supports such feature.

Some objstorage backends, typically cloud based ones like azure or s3, can provide a direct download link for a stored object.

Parameters:

  • obj_id – object identifier

  • content_disposition – set Content-Disposition header for the generated URL response if the objstorage backend supports it

  • expiry – the duration after which the URL expires if the objstorage backend supports it, if not provided the URL expires 24 hours after its creation

Returns:

Direct download URL for the object or None if the objstorage backend does

not support such feature.

delete(obj_id: ObjId)[source]#

Delete an object.

Parameters:

obj_id – object identifier.

Raises:

ObjNotFoundError – if the requested object is missing.

class swh.objstorage.backends.libcloud.AwsCloudObjStorage(container_name: str, compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] = 'gzip', path_prefix: str | None = None, **kwargs)[source]#

Bases: CloudObjStorage

Amazon’s S3 Cloud-based object storage

name: str = 's3'#

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor

download_url(obj_id: ObjId, content_disposition: str | None = None, expiry: timedelta | None = None) → str | None[source]#

Get a direct download link for the object if the obstorage backend supports such feature.

Some objstorage backends, typically cloud based ones like azure or s3, can provide a direct download link for a stored object.

Parameters:

  • obj_id – object identifier

  • content_disposition – set Content-Disposition header for the generated URL response if the objstorage backend supports it

  • expiry – the duration after which the URL expires if the objstorage backend supports it, if not provided the URL expires 24 hours after its creation

Returns:

Direct download URL for the object or None if the objstorage backend does

not support such feature.

class swh.objstorage.backends.libcloud.OpenStackCloudObjStorage(container_name: str, compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] = 'gzip', path_prefix: str | None = None, **kwargs)[source]#

Bases: CloudObjStorage

OpenStack Swift Cloud based object storage

name: str = 'swift'#

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor