swh.objstorage.api.client module#

class swh.objstorage.api.client.RemoteObjStorage(url, api_exception=None, timeout=None, chunk_size=4096, reraise_exceptions=None, **kwargs)[source]#

Bases: RPCClient

Proxy to a remote object storage.

This class allows to connect to an object storage server via http protocol.

url#

The url of the server to connect. Must end with a ‘/’

Type:: string

session#: The session to send requests.

api_exception#: alias of ObjStorageAPIError

reraise_exceptions: ClassVar[List[Type[Exception]]] = [<class 'swh.objstorage.exc.ObjNotFoundError'>, <class 'swh.objstorage.exc.Error'>]#: On server errors, if any of the exception classes in this list has the same name as the error name, then the exception will be instantiated and raised instead of a generic RemoteException.

backend_class#: alias of ObjStorageInterface

restore(content: bytes, obj_id: Union[bytes, CompositeObjId]) → None[source]#

list_content(last_obj_id: Optional[Union[bytes, CompositeObjId]] = None, limit: Optional[int] = 10000) → Iterator[CompositeObjId][source]#

add(content: bytes, obj_id: Union[bytes, CompositeObjId], check_presence: bool = True) → None#

Add a new object to the object storage.

Parameters:

content – object’s raw content to add in storage.
obj_id – either dict of checksums, or single checksum of [bytes] using [ID_HASH_ALGO] algorithm. It is trusted to match the bytes.
check_presence (bool) – indicate if the presence of the content should be verified before adding the file.

Returns:

the id (bytes) of the object into the storage.

add_batch(contents: Union[Mapping[bytes, bytes], Iterable[Tuple[Union[bytes, CompositeObjId], bytes]]], check_presence: bool = True) → Dict#

Add a batch of new objects to the object storage.

Parameters:: contents – either mapping from [ID_HASH_ALGO] checksums to object contents, or list of pairs of dict hashes and object contents
Returns:: the summary of objects added to the storage (count of object, count of bytes object)

check(obj_id: Union[bytes, CompositeObjId]) → None#

Perform an integrity check for a given object.

Verify that the file object is in place and that the content matches the object id.

Parameters:

obj_id – object identifier.

Raises:

ObjNotFoundError – if the requested object is missing.
Error – if the request object is corrupted.

check_config(*, check_write)#

Check whether the object storage is properly configured.

Parameters:

check_write (bool) – if True, check if writes to the object storage
succeed. (can) –

Returns:

True if the configuration check worked, an exception if it didn’t.

delete(obj_id: Union[bytes, CompositeObjId])#

Delete an object.

Parameters:: obj_id – object identifier.
Raises:: ObjNotFoundError – if the requested object is missing.

download_url(obj_id: Union[bytes, CompositeObjId], content_disposition: Optional[str] = None, expiry: Optional[timedelta] = None) → Optional[str]#

Get a direct download link for the object if the obstorage backend supports such feature.

Some objstorage backends, typically cloud based ones like azure or s3, can provide a direct download link for a stored object.

Parameters:

obj_id – object identifier
content_disposition – set Content-Disposition header for the generated URL response if the objstorage backend supports it
expiry – the duration after which the URL expires if the objstorage backend supports it, if not provided the URL expires 24 hours after its creation

Returns:

Direct download URL for the object or None if the objstorage backend does: not support such feature.

get(obj_id: Union[bytes, CompositeObjId]) → bytes#

Retrieve the content of a given object.

Parameters:: obj_id – object id.
Returns:: the content of the requested object as bytes.
Raises:: ObjNotFoundError – if the requested object is missing.

get_batch(obj_ids: Iterable[Union[bytes, CompositeObjId]]) → Iterator[Optional[bytes]]#

Retrieve objects’ raw content in bulk from storage.

Note: This function does have a default implementation in ObjStorage that is suitable for most cases.

For object storages that needs to do the minimal number of requests possible (ex: remote object storages), that method can be overridden to perform a more efficient operation.

Parameters:: obj_ids – list of object ids.
Returns:: list of resulting contents, or None if the content could not be retrieved. Do not raise any exception as a fail for one content will not cancel the whole request.