swh.mosaic package#

Module contents:

class swh.mosaic.IdxDescription#

Bases: object

This enumerates supported indexes. Each value describes the index key and the mapping technique.

BLAKE2FMPHGO = IdxDescription.BLAKE2FMPHGO#

SHA1FMPHGO = IdxDescription.SHA1FMPHGO#

SHA1GITFMPHGO = IdxDescription.SHA1GITFMPHGO#

SHA256FMPHGO = IdxDescription.SHA256FMPHGO#

class swh.mosaic.MosaicCreator(path, indexes, comments=None, tile_threshold=32000000)#

Bases: object

MOSAIC file creator

Indexes and some important header fields are only written on close(), so do not forget to call it. Objects’ keys must be computed by the caller.

add(keys, object)#

Inserts an object, with keys for configured indexes.

keys[i] corresponds to the i-th index given at construction.

close()#: Finishes a MOSAIC file; once this is done, any call to the MosaicCreator will have an undefined behavior.

max_object_size#: Objects bigger than this cannot be inserted.

objects_counter#: Current number of objects in the container

objects_total_size#: Total size of objects in the container

tile_threshold#: Current Tile’s target size. Tiles can be bigger.

class swh.mosaic.MosaicReader(path)#

Bases: object

Reads a local MOSAIC file

close()#: This no-op is only here for swh-shard compliance

comments#: Returns the content of files’ comment fields, if any.

indexes#

Returns the list of indexes available in that file. No index will be loaded.

May raise ValueError if the MOSAIC is corrupted

load_index(idx_description)#: May raise KeyError if given index does not appear in that file

lookup(key)#

Returns the object associated to given key, if it exists in that file. key semantics are preset by load_index().

May raise ValueError if given key size does not match the key size expected by the loaded index

objects_counter#: Number of objects contained in that file.

objects_total_size#: Total size of objects in that file (uncompressed).

class swh.mosaic.MosaicUpdater(path)#

Bases: object

delete(index)#

Deletes an object.

It is done by replacing it, as well as all of its keys and offsets (across all indexes) by MosaicTag::Void elements. MosaicTag::ObjectsCounter and MosaicTag::ObjectsTotalSize are also updated to reflect the deletion. All impacted MosaicTag::Crc32 elements are also updated.

It will raise MosaicUpdaterError::MissingKey if not all the keys corresponding to the object are given.

swh.mosaic.max_object_size(tile_threshold)#

Computes the maximum object size for a given tile threshold.

Use this to determine the largest object that can be inserted into a MOSAIC file before starting to write it.