swh.objstorage.objstorage module
- class swh.objstorage.objstorage.ObjStorage(*, allow_delete=False, **kwargs)[source]
Bases:
objectHigh-level API to manipulate the Software Heritage object storage.
Conceptually, the object storage offers the following methods:
check_config() check if the object storage is properly configured
__contains__() check if an object is present, by object id
add() add a new object, returning an object id
restore() same as add() but erase an already existed content
get() retrieve the content of an object, by object id
check() check the integrity of an object, by object id
delete() remove an object
And some management methods:
- get_random() get random object id of existing contents (used for the
content integrity checker).
Some of the methods have available streaming equivalents:
add_stream() same as add() but with a chunked iterator
restore_stream() same as add_stream() but erase already existing content
get_stream() same as get() but returns a chunked iterator
Each implementation of this interface can have a different behavior and its own way to store the contents.
- abstract check_config(*, check_write)[source]
Check whether the object storage is properly configured.
- Parameters
check_write (bool) – if True, check if writes to the object storage
succeed. (can) –
- Returns
True if the configuration check worked, an exception if it didn’t.
- abstract add(content, obj_id=None, check_presence=True, *args, **kwargs)[source]
Add a new object to the object storage.
- Parameters
content (bytes) – object’s raw content to add in storage.
obj_id (bytes) – checksum of [bytes] using [ID_HASH_ALGO] algorithm. When given, obj_id will be trusted to match the bytes. If missing, obj_id will be computed on the fly.
check_presence (bool) – indicate if the presence of the content should be verified before adding the file.
- Returns
the id (bytes) of the object into the storage.
- add_batch(contents, check_presence=True) Dict[source]
Add a batch of new objects to the object storage.
- Parameters
contents – mapping from obj_id to object contents
- Returns
the summary of objects added to the storage (count of object, count of bytes object)
- restore(content, obj_id=None, *args, **kwargs)[source]
Restore a content that have been corrupted.
This function is identical to add but does not check if the object id is already in the file system. The default implementation provided by the current class is suitable for most cases.
- abstract get(obj_id, *args, **kwargs)[source]
Retrieve the content of a given object.
- Parameters
obj_id (bytes) – object id.
- Returns
the content of the requested object as bytes.
- Raises
ObjNotFoundError – if the requested object is missing.
- get_batch(obj_ids, *args, **kwargs)[source]
Retrieve objects’ raw content in bulk from storage.
Note: This function does have a default implementation in ObjStorage that is suitable for most cases.
For object storages that needs to do the minimal number of requests possible (ex: remote object storages), that method can be overridden to perform a more efficient operation.
- Parameters
([bytes] (obj_ids) – list of object ids.
- Returns
list of resulting contents, or None if the content could not be retrieved. Do not raise any exception as a fail for one content will not cancel the whole request.
- abstract check(obj_id, *args, **kwargs)[source]
Perform an integrity check for a given object.
Verify that the file object is in place and that the content matches the object id.
- Parameters
obj_id (bytes) – object identifier.
- Raises
ObjNotFoundError – if the requested object is missing.
Error – if the request object is corrupted.
- abstract delete(obj_id, *args, **kwargs)[source]
Delete an object.
- Parameters
obj_id (bytes) – object identifier.
- Raises
ObjNotFoundError – if the requested object is missing.
- get_random(batch_size, *args, **kwargs)[source]
Get random ids of existing contents.
This method is used in order to get random ids to perform content integrity verifications on random contents.
- Parameters
batch_size (int) – Number of ids that will be given
- Yields
An iterable of ids (bytes) of contents that are in the current object storage.
- add_stream(content_iter, obj_id, check_presence=True)[source]
Add a new object to the object storage using streaming.
This function is identical to add() except it takes a generator that yields the chunked content instead of the whole content at once.
- restore_stream(content_iter, obj_id=None)[source]
Restore a content that have been corrupted using streaming.
This function is identical to restore() except it takes a generator that yields the chunked content instead of the whole content at once. The default implementation provided by the current class is suitable for most cases.
- get_stream(obj_id, chunk_size=2097152)[source]
Retrieve the content of a given object as a chunked iterator.
- Parameters
obj_id (bytes) – object id.
- Returns
the content of the requested object as bytes.
- Raises
ObjNotFoundError – if the requested object is missing.