swh.objstorage.multiplexer.filter.id_filter module

class swh.objstorage.multiplexer.filter.id_filter.IdObjStorageFilter(storage)[source]

Bases: swh.objstorage.multiplexer.filter.filter.ObjStorageFilter

Filter that only allow operations if the object id match a requirement.

Even for read operations, check before if the id match the requirements. This may prevent for unnecessary disk access.

abstract is_valid(obj_id)[source]

Indicates if the given id is valid.

add(content, obj_id=None, check_presence=True, *args, **kwargs)[source]

Add a new object to the object storage.

Parameters
  • content (bytes) – object’s raw content to add in storage.

  • obj_id (bytes) – checksum of [bytes] using [ID_HASH_ALGO] algorithm. When given, obj_id will be trusted to match the bytes. If missing, obj_id will be computed on the fly.

  • check_presence (bool) – indicate if the presence of the content should be verified before adding the file.

Returns

the id (bytes) of the object into the storage.

restore(content, obj_id=None, *args, **kwargs)[source]

Restore a content that have been corrupted.

This function is identical to add but does not check if the object id is already in the file system. The default implementation provided by the current class is suitable for most cases.

Parameters
  • content (bytes) – object’s raw content to add in storage

  • obj_id (bytes) – checksum of bytes as computed by ID_HASH_ALGO. When given, obj_id will be trusted to match bytes. If missing, obj_id will be computed on the fly.

get(obj_id, *args, **kwargs)[source]

Retrieve the content of a given object.

Parameters

obj_id (bytes) – object id.

Returns

the content of the requested object as bytes.

Raises

ObjNotFoundError – if the requested object is missing.

check(obj_id, *args, **kwargs)[source]

Perform an integrity check for a given object.

Verify that the file object is in place and that the content matches the object id.

Parameters

obj_id (bytes) – object identifier.

Raises
get_random(*args, **kwargs)[source]

Get random ids of existing contents.

This method is used in order to get random ids to perform content integrity verifications on random contents.

Parameters

batch_size (int) – Number of ids that will be given

Yields

An iterable of ids (bytes) of contents that are in the current object storage.

class swh.objstorage.multiplexer.filter.id_filter.RegexIdObjStorageFilter(storage, regex)[source]

Bases: swh.objstorage.multiplexer.filter.id_filter.IdObjStorageFilter

Filter that allow operations if the content’s id as hex match a regex.

is_valid(obj_id)[source]

Indicates if the given id is valid.

class swh.objstorage.multiplexer.filter.id_filter.PrefixIdObjStorageFilter(storage, prefix)[source]

Bases: swh.objstorage.multiplexer.filter.id_filter.IdObjStorageFilter

Filter that allow operations if the hexlified id have a given prefix.

is_valid(obj_id)[source]

Indicates if the given id is valid.