swh.objstorage package

Submodules

swh.objstorage.exc module

exception swh.objstorage.exc.Error[source]

Bases: Exception

exception swh.objstorage.exc.ObjNotFoundError[source]

Bases: swh.objstorage.exc.Error

exception swh.objstorage.exc.ObjStorageAPIError[source]

Bases: Exception

Specific internal exception of an object storage (mainly connection).

swh.objstorage.objstorage module

swh.objstorage.objstorage.compute_hash(content)[source]

Compute the content’s hash.

Parameters:
  • content (bytes) – The raw content to hash
  • hash_name (str) – Hash’s name (default to ID_HASH_ALGO)
Returns:

The ID_HASH_ALGO for the content
class swh.objstorage.objstorage.ObjStorage(*, allow_delete=False, **kwargs)[source]

Bases: object

High-level API to manipulate the Software Heritage object storage.

Conceptually, the object storage offers the following methods:

  • check_config() check if the object storage is properly configured
  • __contains__() check if an object is present, by object id
  • add() add a new object, returning an object id
  • restore() same as add() but erase an already existed content
  • get() retrieve the content of an object, by object id
  • check() check the integrity of an object, by object id
  • delete() remove an object

And some management methods:

  • get_random() get random object id of existing contents (used for the
    content integrity checker).

Some of the methods have available streaming equivalents:

  • add_stream() same as add() but with a chunked iterator
  • restore_stream() same as add_stream() but erase already existing content
  • get_stream() same as get() but returns a chunked iterator

Each implementation of this interface can have a different behavior and its own way to store the contents.

check_config(*, check_write)[source]

Check whether the object storage is properly configured.

Parameters:
  • check_write (bool) – if True, check if writes to the object storage
  • succeed. (can) –
Returns:

True if the configuration check worked, an exception if it didn’t.
add(content, obj_id=None, check_presence=True, *args, **kwargs)[source]

Add a new object to the object storage.

Parameters:
  • content (bytes) – object’s raw content to add in storage.
  • obj_id (bytes) – checksum of [bytes] using [ID_HASH_ALGO] algorithm. When given, obj_id will be trusted to match the bytes. If missing, obj_id will be computed on the fly.
  • check_presence (bool) – indicate if the presence of the content should be verified before adding the file.
Returns:

the id (bytes) of the object into the storage.
add_batch(contents, check_presence=True)[source]

Add a batch of new objects to the object storage.

Parameters:contents (dict) – mapping from obj_id to object conetnts
Returns:the number of objects added to the storage
restore(content, obj_id=None, *args, **kwargs)[source]

Restore a content that have been corrupted.

This function is identical to add but does not check if the object id is already in the file system. The default implementation provided by the current class is suitable for most cases.

Parameters:
  • content (bytes) – object’s raw content to add in storage
  • obj_id (bytes) – checksum of bytes as computed by ID_HASH_ALGO. When given, obj_id will be trusted to match bytes. If missing, obj_id will be computed on the fly.
get(obj_id, *args, **kwargs)[source]

Retrieve the content of a given object.

Parameters:obj_id (bytes) – object id.
Returns:the content of the requested object as bytes.
Raises:ObjNotFoundError – if the requested object is missing.
get_batch(obj_ids, *args, **kwargs)[source]

Retrieve objects’ raw content in bulk from storage.

Note: This function does have a default implementation in ObjStorage that is suitable for most cases.

For object storages that needs to do the minimal number of requests possible (ex: remote object storages), that method can be overridden to perform a more efficient operation.

Parameters:([bytes] (obj_ids) – list of object ids.
Returns:list of resulting contents, or None if the content could not be retrieved. Do not raise any exception as a fail for one content will not cancel the whole request.
check(obj_id, *args, **kwargs)[source]

Perform an integrity check for a given object.

Verify that the file object is in place and that the gziped content matches the object id.

Parameters:

obj_id (bytes) – object identifier.
Raises:
  • ObjNotFoundError – if the requested object is missing.
  • Error – if the request object is corrupted.
delete(obj_id, *args, **kwargs)[source]

Delete an object.

Parameters:obj_id (bytes) – object identifier.
Raises:ObjNotFoundError – if the requested object is missing.
get_random(batch_size, *args, **kwargs)[source]

Get random ids of existing contents.

This method is used in order to get random ids to perform content integrity verifications on random contents.

Parameters:batch_size (int) – Number of ids that will be given
Yields:An iterable of ids (bytes) of contents that are in the current object storage.
add_stream(content_iter, obj_id, check_presence=True)[source]

Add a new object to the object storage using streaming.

This function is identical to add() except it takes a generator that yields the chunked content instead of the whole content at once.

Parameters:
  • content (bytes) – chunked generator that yields the object’s raw content to add in storage.
  • obj_id (bytes) – object identifier
  • check_presence (bool) – indicate if the presence of the content should be verified before adding the file.
Returns:

the id (bytes) of the object into the storage.
restore_stream(content_iter, obj_id=None)[source]

Restore a content that have been corrupted using streaming.

This function is identical to restore() except it takes a generator that yields the chunked content instead of the whole content at once. The default implementation provided by the current class is suitable for most cases.

Parameters:
  • content (bytes) – chunked generator that yields the object’s raw content to add in storage.
  • obj_id (bytes) – object identifier
get_stream(obj_id, chunk_size=2097152)[source]

Retrieve the content of a given object as a chunked iterator.

Parameters:obj_id (bytes) – object id.
Returns:the content of the requested object as bytes.
Raises:ObjNotFoundError – if the requested object is missing.

swh.objstorage.objstorage_in_memory module

class swh.objstorage.objstorage_in_memory.InMemoryObjStorage(**args)[source]

Bases: swh.objstorage.objstorage.ObjStorage

In-Memory objstorage.

Intended for test purposes.

check_config(*, check_write)[source]

Check whether the object storage is properly configured.

Parameters:
  • check_write (bool) – if True, check if writes to the object storage
  • succeed. (can) –
Returns:

True if the configuration check worked, an exception if it didn’t.
add(content, obj_id=None, check_presence=True, *args, **kwargs)[source]

Add a new object to the object storage.

Parameters:
  • content (bytes) – object’s raw content to add in storage.
  • obj_id (bytes) – checksum of [bytes] using [ID_HASH_ALGO] algorithm. When given, obj_id will be trusted to match the bytes. If missing, obj_id will be computed on the fly.
  • check_presence (bool) – indicate if the presence of the content should be verified before adding the file.
Returns:

the id (bytes) of the object into the storage.
get(obj_id, *args, **kwargs)[source]

Retrieve the content of a given object.

Parameters:obj_id (bytes) – object id.
Returns:the content of the requested object as bytes.
Raises:ObjNotFoundError – if the requested object is missing.
check(obj_id, *args, **kwargs)[source]

Perform an integrity check for a given object.

Verify that the file object is in place and that the gziped content matches the object id.

Parameters:

obj_id (bytes) – object identifier.
Raises:
  • ObjNotFoundError – if the requested object is missing.
  • Error – if the request object is corrupted.
delete(obj_id, *args, **kwargs)[source]

Delete an object.

Parameters:obj_id (bytes) – object identifier.
Raises:ObjNotFoundError – if the requested object is missing.

swh.objstorage.objstorage_pathslicing module

class swh.objstorage.objstorage_pathslicing.PathSlicingObjStorage(root, slicing, **kwargs)[source]

Bases: swh.objstorage.objstorage.ObjStorage

Implementation of the ObjStorage API based on the hash of the content.

On disk, an object storage is a directory tree containing files named after their object IDs. An object ID is a checksum of its content, depending on the value of the ID_HASH_ALGO constant (see swh.model.hashutil for its meaning).

To avoid directories that contain too many files, the object storage has a given slicing. Each slicing correspond to a directory that is named according to the hash of its content.

So for instance a file with SHA1 34973274ccef6ab4dfaaf86599792fa9c3fe4689 will be stored in the given object storages :

  • 0:2/2:4/4:6 : 34/97/32/34973274ccef6ab4dfaaf86599792fa9c3fe4689
  • 0:1/0:5/ : 3/34973/34973274ccef6ab4dfaaf86599792fa9c3fe4689

The files in the storage are stored in gzipped compressed format.

root

path to the root directory of the storage on the disk.

Type:string
bounds

list of tuples that indicates the beginning and the end of each subdirectory for a content.

check_config(*, check_write)[source]

Check whether this object storage is properly configured

add(content, obj_id=None, check_presence=True)[source]

Add a new object to the object storage.

Parameters:
  • content (bytes) – object’s raw content to add in storage.
  • obj_id (bytes) – checksum of [bytes] using [ID_HASH_ALGO] algorithm. When given, obj_id will be trusted to match the bytes. If missing, obj_id will be computed on the fly.
  • check_presence (bool) – indicate if the presence of the content should be verified before adding the file.
Returns:

the id (bytes) of the object into the storage.
get(obj_id)[source]

Retrieve the content of a given object.

Parameters:obj_id (bytes) – object id.
Returns:the content of the requested object as bytes.
Raises:ObjNotFoundError – if the requested object is missing.
check(obj_id)[source]

Perform an integrity check for a given object.

Verify that the file object is in place and that the gziped content matches the object id.

Parameters:

obj_id (bytes) – object identifier.
Raises:
  • ObjNotFoundError – if the requested object is missing.
  • Error – if the request object is corrupted.
delete(obj_id)[source]

Delete an object.

Parameters:obj_id (bytes) – object identifier.
Raises:ObjNotFoundError – if the requested object is missing.
get_random(batch_size)[source]

Get random ids of existing contents.

This method is used in order to get random ids to perform content integrity verifications on random contents.

Parameters:batch_size (int) – Number of ids that will be given
Yields:An iterable of ids (bytes) of contents that are in the current object storage.
chunk_writer(obj_id)[source]
add_stream(content_iter, obj_id, check_presence=True)[source]

Add a new object to the object storage using streaming.

This function is identical to add() except it takes a generator that yields the chunked content instead of the whole content at once.

Parameters:
  • content (bytes) – chunked generator that yields the object’s raw content to add in storage.
  • obj_id (bytes) – object identifier
  • check_presence (bool) – indicate if the presence of the content should be verified before adding the file.
Returns:

the id (bytes) of the object into the storage.
get_stream(obj_id, chunk_size=2097152)[source]

Retrieve the content of a given object as a chunked iterator.

Parameters:obj_id (bytes) – object id.
Returns:the content of the requested object as bytes.
Raises:ObjNotFoundError – if the requested object is missing.

swh.objstorage.objstorage_rados module

class swh.objstorage.objstorage_rados.RADOSObjStorage(*, rados_id, pool_name, ceph_config, allow_delete=False)[source]

Bases: swh.objstorage.objstorage.ObjStorage

Object storage implemented with RADOS

check_config(*, check_write)[source]

Check whether the object storage is properly configured.

Parameters:
  • check_write (bool) – if True, check if writes to the object storage
  • succeed. (can) –
Returns:

True if the configuration check worked, an exception if it didn’t.
ioctx
add(content, obj_id=None, check_presence=True)[source]

Add a new object to the object storage.

Parameters:
  • content (bytes) – object’s raw content to add in storage.
  • obj_id (bytes) – checksum of [bytes] using [ID_HASH_ALGO] algorithm. When given, obj_id will be trusted to match the bytes. If missing, obj_id will be computed on the fly.
  • check_presence (bool) – indicate if the presence of the content should be verified before adding the file.
Returns:

the id (bytes) of the object into the storage.
get(obj_id)[source]

Retrieve the content of a given object.

Parameters:obj_id (bytes) – object id.
Returns:the content of the requested object as bytes.
Raises:ObjNotFoundError – if the requested object is missing.
check(obj_id)[source]

Perform an integrity check for a given object.

Verify that the file object is in place and that the gziped content matches the object id.

Parameters:

obj_id (bytes) – object identifier.
Raises:
  • ObjNotFoundError – if the requested object is missing.
  • Error – if the request object is corrupted.
delete(obj_id)[source]

Delete an object.

Parameters:obj_id (bytes) – object identifier.
Raises:ObjNotFoundError – if the requested object is missing.

Module contents

swh.objstorage.get_objstorage(cls, args)[source]

Create an ObjStorage using the given implementation class.

Parameters:
  • cls (str) – objstorage class unique key contained in the _STORAGE_CLASSES dict.
  • args (dict) – arguments for the required class of objstorage that must match exactly the one in the __init__ method of the class.
Returns:

subclass of ObjStorage that match the given storage_class argument.
Raises:

ValueError – if the given storage class is not a valid objstorage key.
class swh.objstorage.ObjStorage(*, allow_delete=False, **kwargs)[source]

Bases: object

High-level API to manipulate the Software Heritage object storage.

Conceptually, the object storage offers the following methods:

  • check_config() check if the object storage is properly configured
  • __contains__() check if an object is present, by object id
  • add() add a new object, returning an object id
  • restore() same as add() but erase an already existed content
  • get() retrieve the content of an object, by object id
  • check() check the integrity of an object, by object id
  • delete() remove an object

And some management methods:

  • get_random() get random object id of existing contents (used for the
    content integrity checker).

Some of the methods have available streaming equivalents:

  • add_stream() same as add() but with a chunked iterator
  • restore_stream() same as add_stream() but erase already existing content
  • get_stream() same as get() but returns a chunked iterator

Each implementation of this interface can have a different behavior and its own way to store the contents.

check_config(*, check_write)[source]

Check whether the object storage is properly configured.

Parameters:
  • check_write (bool) – if True, check if writes to the object storage
  • succeed. (can) –
Returns:

True if the configuration check worked, an exception if it didn’t.
add(content, obj_id=None, check_presence=True, *args, **kwargs)[source]

Add a new object to the object storage.

Parameters:
  • content (bytes) – object’s raw content to add in storage.
  • obj_id (bytes) – checksum of [bytes] using [ID_HASH_ALGO] algorithm. When given, obj_id will be trusted to match the bytes. If missing, obj_id will be computed on the fly.
  • check_presence (bool) – indicate if the presence of the content should be verified before adding the file.
Returns:

the id (bytes) of the object into the storage.
add_batch(contents, check_presence=True)[source]

Add a batch of new objects to the object storage.

Parameters:contents (dict) – mapping from obj_id to object conetnts
Returns:the number of objects added to the storage
restore(content, obj_id=None, *args, **kwargs)[source]

Restore a content that have been corrupted.

This function is identical to add but does not check if the object id is already in the file system. The default implementation provided by the current class is suitable for most cases.

Parameters:
  • content (bytes) – object’s raw content to add in storage
  • obj_id (bytes) – checksum of bytes as computed by ID_HASH_ALGO. When given, obj_id will be trusted to match bytes. If missing, obj_id will be computed on the fly.
get(obj_id, *args, **kwargs)[source]

Retrieve the content of a given object.

Parameters:obj_id (bytes) – object id.
Returns:the content of the requested object as bytes.
Raises:ObjNotFoundError – if the requested object is missing.
get_batch(obj_ids, *args, **kwargs)[source]

Retrieve objects’ raw content in bulk from storage.

Note: This function does have a default implementation in ObjStorage that is suitable for most cases.

For object storages that needs to do the minimal number of requests possible (ex: remote object storages), that method can be overridden to perform a more efficient operation.

Parameters:([bytes] (obj_ids) – list of object ids.
Returns:list of resulting contents, or None if the content could not be retrieved. Do not raise any exception as a fail for one content will not cancel the whole request.
check(obj_id, *args, **kwargs)[source]

Perform an integrity check for a given object.

Verify that the file object is in place and that the gziped content matches the object id.

Parameters:

obj_id (bytes) – object identifier.
Raises:
  • ObjNotFoundError – if the requested object is missing.
  • Error – if the request object is corrupted.
delete(obj_id, *args, **kwargs)[source]

Delete an object.

Parameters:obj_id (bytes) – object identifier.
Raises:ObjNotFoundError – if the requested object is missing.
get_random(batch_size, *args, **kwargs)[source]

Get random ids of existing contents.

This method is used in order to get random ids to perform content integrity verifications on random contents.

Parameters:batch_size (int) – Number of ids that will be given
Yields:An iterable of ids (bytes) of contents that are in the current object storage.
add_stream(content_iter, obj_id, check_presence=True)[source]

Add a new object to the object storage using streaming.

This function is identical to add() except it takes a generator that yields the chunked content instead of the whole content at once.

Parameters:
  • content (bytes) – chunked generator that yields the object’s raw content to add in storage.
  • obj_id (bytes) – object identifier
  • check_presence (bool) – indicate if the presence of the content should be verified before adding the file.
Returns:

the id (bytes) of the object into the storage.
restore_stream(content_iter, obj_id=None)[source]

Restore a content that have been corrupted using streaming.

This function is identical to restore() except it takes a generator that yields the chunked content instead of the whole content at once. The default implementation provided by the current class is suitable for most cases.

Parameters:
  • content (bytes) – chunked generator that yields the object’s raw content to add in storage.
  • obj_id (bytes) – object identifier
get_stream(obj_id, chunk_size=2097152)[source]

Retrieve the content of a given object as a chunked iterator.

Parameters:obj_id (bytes) – object id.
Returns:the content of the requested object as bytes.
Raises:ObjNotFoundError – if the requested object is missing.