swh.objstorage.backends.azure module#

swh.objstorage.backends.azure.get_container_url(account_name: str, account_key: str, container_name: str, access_policy: str = 'read_only', expiry: timedelta = datetime.timedelta(days=365), container_url_template: str = 'https://{account_name}.blob.core.windows.net/{container_name}?{signature}', **kwargs) → str[source]#

Get the full url, for the given container on the given account, with a Shared Access Signature granting the specified access policy.

Parameters:

account_name – name of the storage account for which to generate the URL
account_key – shared account key of the storage account used to generate the SAS
container_name – name of the container for which to grant access in the storage account
access_policy – one of read_only, append_only, full
expiry – the interval in the future with which the signature will expire

Returns:

the full URL of the container, with the shared access signature.

class swh.objstorage.backends.azure.AzureCloudObjStorage(container_url: Optional[str] = None, account_name: Optional[str] = None, api_secret_key: Optional[str] = None, container_name: Optional[str] = None, connection_string: Optional[str] = None, compression='gzip', **kwargs)[source]#

Bases: ObjStorage

ObjStorage backend for Azure blob storage accounts.

Parameters:

container_url – the URL of the container in which the objects are stored.
account_name – (deprecated) the name of the storage account under which objects are stored
api_secret_key – (deprecated) the shared account key
container_name – (deprecated) the name of the container under which objects are stored
compression – the compression algorithm used to compress objects in storage

Notes

The container url should contain the credentials via a “Shared Access Signature”. The get_container_url() helper can be used to generate such a URL from the account’s access keys. The account_name, api_secret_key and container_name arguments are deprecated.

PRIMARY_HASH: typing_extensions.Literal[sha1] = 'sha1'#

get_container_client(hex_obj_id)[source]#

Get the container client for the container that contains the object with internal id hex_obj_id

This is used to allow the PrefixedAzureCloudObjStorage to dispatch the client according to the prefix of the object id.

get_async_container_clients()[source]#

Returns a collection of container clients, to be passed to get_async_blob_client.

Each container may not be used in more than one asyncio loop.

get_blob_client(hex_obj_id)[source]#: Get the azure blob client for the given hex obj id

get_async_blob_client(hex_obj_id, container_clients)[source]#: Get the azure blob client for the given hex obj id and a collection yielded by get_async_container_clients.

get_all_container_clients()[source]#: Get all active block_blob_services

check_config(*, check_write)[source]#: Check the configuration for this object storage

add(content: bytes, obj_id: Union[bytes, CompositeObjId], check_presence: bool = True) → None[source]#: Add an obj in storage if it’s not there already.

restore(content: bytes, obj_id: Union[bytes, CompositeObjId]) → None[source]#: Restore a content.

get(obj_id: Union[bytes, CompositeObjId]) → bytes[source]#: retrieve blob’s content if found.

get_batch(obj_ids: Iterable[Union[bytes, CompositeObjId]]) → Iterator[Optional[bytes]][source]#: Retrieve objects’ raw content in bulk from storage, concurrently.

check(obj_id: Union[bytes, CompositeObjId]) → None[source]#: Check the content integrity.

delete(obj_id: Union[bytes, CompositeObjId])[source]#: Delete an object.

class swh.objstorage.backends.azure.PrefixedAzureCloudObjStorage(accounts: Mapping[str, Union[str, Mapping[str, str]]], compression='gzip', **kwargs)[source]#

Bases: AzureCloudObjStorage

ObjStorage with azure capabilities, striped by prefix.

accounts is a dict containing entries of the form:: <prefix>: <container_url_for_prefix>

get_container_client(hex_obj_id)[source]#: Get the block_blob_service and container that contains the object with internal id hex_obj_id

get_async_container_clients()[source]#

Returns a collection of container clients, to be passed to get_async_blob_client.

Each container may not be used in more than one asyncio loop.

get_async_blob_client(hex_obj_id, container_clients)[source]#: Get the azure blob client for the given hex obj id and a collection yielded by get_async_container_clients.

get_all_container_clients()[source]#: Get all active container clients