swh.objstorage.backends.azure module#

swh.objstorage.backends.azure.get_container_url(account_name: str, account_key: str, container_name: str, access_policy: str = 'read_only', expiry: timedelta = datetime.timedelta(days=365), container_url_template: str = 'https://{account_name}.blob.core.windows.net/{container_name}?{signature}', **kwargs) → str[source]#

Get the full url, for the given container on the given account, with a Shared Access Signature granting the specified access policy.

Parameters:

  • account_name – name of the storage account for which to generate the URL

  • account_key – shared account key of the storage account used to generate the SAS

  • container_name – name of the container for which to grant access in the storage account

  • access_policy – one of read_only, append_only, full

  • expiry – the interval in the future with which the signature will expire

Returns:

the full URL of the container, with the shared access signature.

class swh.objstorage.backends.azure.AzureCloudObjStorage(*, container_url: str | None = None, account_name: str | None = None, api_secret_key: str | None = None, container_name: str | None = None, connection_string: str | None = None, compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] = 'gzip', use_secondary_endpoint_for_downloads=False, **kwargs)[source]#

Bases: ObjStorage

ObjStorage backend for Azure blob storage accounts.

Parameters:

  • container_url – the URL of the container in which the objects are stored.

  • account_name – (deprecated) the name of the storage account under which objects are stored

  • api_secret_key – (deprecated) the shared account key

  • container_name – (deprecated) the name of the container under which objects are stored

  • compression – the compression algorithm used to compress objects in storage

  • use_secondary_endpoint_for_downloads – if True, use the secondary endpoint url to generate download URLs. To configure the secondary endpoint, use the BlobSecondaryEndpoint entry of the connection string.

Notes

The container url should contain the credentials via a “Shared Access Signature”. The get_container_url() helper can be used to generate such a URL from the account’s access keys. The account_name, api_secret_key and container_name arguments are deprecated.

PRIMARY_HASH: LiteralPrimaryHash = 'sha1'#
name: str = 'azure'#

Default objstorage name; can be overloaded at instantiation time giving a ‘name’ argument to the constructor

get_container_client(hex_obj_id)[source]#

Get the container client for the container that contains the object with internal id hex_obj_id

This is used to allow the PrefixedAzureCloudObjStorage to dispatch the client according to the prefix of the object id.

get_async_container_clients()[source]#

Returns a collection of container clients, to be passed to get_async_blob_client.

Each container may not be used in more than one asyncio loop.

get_blob_client(hex_obj_id)[source]#

Get the azure blob client for the given hex obj id

get_async_blob_client(hex_obj_id, container_clients)[source]#

Get the azure blob client for the given hex obj id and a collection yielded by get_async_container_clients.

get_all_container_clients()[source]#

Get all active block_blob_services

check_config(*, check_write)[source]#

Check the configuration for this object storage

add(content: bytes, obj_id: ObjId, check_presence: bool = True) → None[source]#

Add an obj in storage if it’s not there already.

restore(content: bytes, obj_id: ObjId) → None[source]#

Restore a content.

get(obj_id: ObjId) → bytes[source]#

retrieve blob’s content if found.

get_batch(obj_ids: Iterable[ObjId]) → Iterator[bytes | None][source]#

Retrieve objects’ raw content in bulk from storage, concurrently.

delete(obj_id: ObjId)[source]#

Delete an object.

download_url(obj_id: ObjId, content_disposition: str | None = None, expiry: timedelta | None = None) → str | None[source]#

Get a direct download link for the object if the obstorage backend supports such feature.

Some objstorage backends, typically cloud based ones like azure or s3, can provide a direct download link for a stored object.

Parameters:

  • obj_id – object identifier

  • content_disposition – set Content-Disposition header for the generated URL response if the objstorage backend supports it

  • expiry – the duration after which the URL expires if the objstorage backend supports it, if not provided the URL expires 24 hours after its creation

Returns:

Direct download URL for the object or None if the objstorage backend does

not support such feature.

class swh.objstorage.backends.azure.PrefixedAzureCloudObjStorage(accounts: Mapping[str, str | Mapping[str, str]], name: str = 'azure-prefixed', compression: Literal['bz2', 'lzma', 'gzip', 'zlib', 'none'] = 'gzip', **kwargs)[source]#

Bases: AzureCloudObjStorage

ObjStorage with azure capabilities, striped by prefix.

accounts is a dict containing entries of the form:

<prefix>: <container_url_for_prefix>

get_container_client(hex_obj_id)[source]#

Get the block_blob_service and container that contains the object with internal id hex_obj_id

get_async_container_clients()[source]#

Returns a collection of container clients, to be passed to get_async_blob_client.

Each container may not be used in more than one asyncio loop.

get_async_blob_client(hex_obj_id, container_clients)[source]#

Get the azure blob client for the given hex obj id and a collection yielded by get_async_container_clients.

get_all_container_clients()[source]#

Get all active container clients