swh.mosaic package#

Module contents:

class swh.mosaic.CompressionMethod#

Bases: object

Supported compression methods.

NONE = CompressionMethod.NONE#

ZSTD = CompressionMethod.ZSTD#

ZSTDDICT = CompressionMethod.ZSTDDICT#

class swh.mosaic.IdxDescription#

Bases: object

BLAKE2FMPHGO = IdxDescription.BLAKE2FMPHGO#

SHA1FMPHGO = IdxDescription.SHA1FMPHGO#

SHA1GITFMPHGO = IdxDescription.SHA1GITFMPHGO#

SHA256FMPHGO = IdxDescription.SHA256FMPHGO#

class swh.mosaic.MosaicCreator(path, indexes, comments=None, tile_threshold=32000000, compression_level=None)#

Bases: object

MOSAIC file creator

Indexes and some important header fields are only written on close(), so do not forget to call it. Objects’ keys must be computed by the caller.

Compression is enabled if provided. Depending on your CPU budget we advise to use between 3 and 15.

add(keys, object)#

Inserts an object, with keys for configured indexes.

keys[i] corresponds to the i-th index given at construction.

close()#: Finishes a MOSAIC file; once this is done, any call to the MosaicCreator will have an undefined behavior.

max_object_size#: Objects bigger than this cannot be inserted.

objects_counter#: Current number of objects in the container

objects_total_size#: Total size of objects in the container

class swh.mosaic.MosaicReader(path, idx_description)#

Bases: object

Reads a local MOSAIC file

close()#: This no-op is only here for swh-shard compliance

comments#: Returns the content of files’ comment fields, if any.

compression#: Compression method, for your information - decompression will be handled transparently by the class.

get_batch(keys)#: Parallel version of lookup.

indexes#

Returns the list of indexes available in that file. No index will be loaded.

May raise ValueError if the MOSAIC is corrupted

items()#

Returns an iterator on (key, content) pairs, where key comes from the currently selected index.

Pairs are returned in the same order as the keys iterators, but not as values.

keys()#: Returns an iterator on keys from the currently selected index, according to the index’s order (which can be considered random).

load_index(idx_description)#

Changes the reader’s current index: this sets key semantics for future calls to lookup or key iterators.

May raise KeyError if given index does not appear in that file

lookup(key)#

Returns the object associated to given key, if it exists in that file. key semantics are preset by load_index().

May raise RuntimeError if given key size does not match the key size expected by the loaded index

objects_counter#: Number of objects contained in that file.

objects_total_size#: Total size of objects in that file (uncompressed).

values()#: This iterates contents by reading the file linearly. This is faster, but will not provide contents in the same order as iter or keys.

class swh.mosaic.MosaicUpdater(path)#

Bases: object

close()#

delete(index)#

Deletes an object.

It is done by replacing it, as well as all of its keys and offsets (across all indexes) by MosaicTag::Void elements. MosaicTag::ObjectsCounter and MosaicTag::ObjectsTotalSize are also updated to reflect the deletion. All impacted MosaicTag::Crc32 elements are also updated.

It will raise MosaicUpdaterError::MissingKey if not all the keys corresponding to the object are given.

swh.mosaic.max_object_size(tile_threshold)#

Computes the maximum object size for a given tile threshold.

Use this to determine the largest object that can be inserted into a MOSAIC file before starting to write it.