swh.objstorage.interface module

class swh.objstorage.interface.CompositeObjId

Bases: TypedDict

sha1: bytes

sha1_git: bytes

sha256: bytes

blake2s256: bytes

swh.objstorage.interface.objid_from_dict(d: Dict[str, Any]) → CompositeObjId

swh.objstorage.interface.ObjId

Type of object ids, which should be {hash: value for hash in SUPPORTED_HASHES}; but single sha1 hashes are supported for legacy clients

alias of bytes | CompositeObjId

class swh.objstorage.interface.ObjStorageInterface(*args, **kwargs)

Bases: Protocol

High-level API to manipulate the Software Heritage object storage.

Conceptually, the object storage offers the following methods:

• check_config() check if the object storage is properly configured

• __contains__() check if an object is present, by object id

• add() add a new object, returning an object id

• restore() same as add() but erase an already existed content

• get() retrieve the content of an object, by object id

• check() check the integrity of an object, by object id

• delete() remove an object

Each implementation of this interface can have a different behavior and its own way to store the contents.

check_config(*, check_write)

Check whether the object storage is properly configured.

Parameters:

• check_write (bool) – if True, check if writes to the object storage

• succeed. (can)

Returns:

True if the configuration check worked, an exception if it didn’t.

add(content: bytes, obj_id: bytes | CompositeObjId, check_presence: bool = True) → None

Add a new object to the object storage.

Parameters:

• content – object’s raw content to add in storage.

• obj_id – either dict of checksums, or single checksum of [bytes] using [ID_HASH_ALGO] algorithm. It is trusted to match the bytes.

• check_presence (bool) – indicate if the presence of the content should be verified before adding the file.

Returns:

the id (bytes) of the object into the storage.

add_batch(contents: Mapping[bytes, bytes] | Iterable[Tuple[bytes | CompositeObjId, bytes]], check_presence: bool = True) → Dict

Add a batch of new objects to the object storage.

Parameters:

contents – either mapping from [ID_HASH_ALGO] checksums to object contents, or list of pairs of dict hashes and object contents

Returns:

the summary of objects added to the storage (count of object, count of bytes object)

restore(content: bytes, obj_id: bytes | CompositeObjId) → None

Restore a content that have been corrupted.

This function is identical to add but does not check if the object id is already in the file system. The default implementation provided by the current class is suitable for most cases.

Parameters:

• content – object’s raw content to add in storage

• obj_id – dict of hashes of the content (or only the sha1, for legacy clients)

get(obj_id: bytes | CompositeObjId) → bytes

Retrieve the content of a given object.

Parameters:

obj_id – object id.

Returns:

the content of the requested object as bytes.

Raises:

ObjNotFoundError – if the requested object is missing.

get_batch(obj_ids: Iterable[bytes | CompositeObjId]) → Iterator[bytes | None]

Retrieve objects’ raw content in bulk from storage.

Note: This function does have a default implementation in ObjStorage that is suitable for most cases.

For object storages that needs to do the minimal number of requests possible (ex: remote object storages), that method can be overridden to perform a more efficient operation.

Parameters:

obj_ids – list of object ids.

Returns:

list of resulting contents, or None if the content could not be retrieved. Do not raise any exception as a fail for one content will not cancel the whole request.

download_url(obj_id: bytes | CompositeObjId, content_disposition: str | None = None, expiry: timedelta | None = None) → str | None

Get a direct download link for the object if the obstorage backend supports such feature.

Some objstorage backends, typically cloud based ones like azure or s3, can provide a direct download link for a stored object.

Parameters:

• obj_id – object identifier

• content_disposition – set Content-Disposition header for the generated URL response if the objstorage backend supports it

• expiry – the duration after which the URL expires if the objstorage backend supports it, if not provided the URL expires 24 hours after its creation

Returns:

Direct download URL for the object or None if the objstorage backend does

not support such feature.

check(obj_id: bytes | CompositeObjId) → None

Perform an integrity check for a given object.

Verify that the file object is in place and that the content matches the object id.

Parameters:

obj_id – object identifier.

Raises:

• ObjNotFoundError – if the requested object is missing.

• ObjCorruptedError – if the requested object is corrupted.

delete(obj_id: bytes | CompositeObjId)

Delete an object.

Parameters:

obj_id – object identifier.

Raises:

ObjNotFoundError – if the requested object is missing.

list_content(last_obj_id: bytes | CompositeObjId | None = None, limit: int | None = 10000) → Iterator[CompositeObjId]

Generates known object ids.

Parameters:

• last_obj_id – object id from which to iterate from (excluded).

• limit (int) – max number of object ids to generate. If unset (None), generate all objects (behavior might not be guaranteed for all backends).

Generates:

obj_id: object ids.