swh.mosaic#

MOSAIC (MOdular Storage of Archived and Indexed Contents) is a file format designed to efficiently store and randomly read contents archived by Software Heritage. Target content is source code and therefore small objects (median size: 3kb), indexed over one (or more) of the possible objstorage keys.

The motivations and design of this file format are thoroughly explained in SWH Enhancement Proposal #5. Format evolutions are described in CHANGELOG.md in the package’s sources.

The Python module provides high-level classes (implemented in Rust) to read/write MOSAIC files.

Usage example#

from swh.mosaic import EBML_SCHEMA, IdxDescription, MosaicCreator, MosaicReader, MosaicUpdater
from pathlib import Path

creator = MosaicCreator(
    Path("example.mosaic"),
    indexes=[IdxDescription.SHA1FMPHGO, IdxDescription.SHA256FMPHGO],
    comments=["Example MOSAIC file", EBML_SCHEMA],
    compression_level=3, # optional, defaults to no compression
    tile_threshold=32_000, # optional, defaults to 32MB
)

# Add an object. Keys must match what was provided as `indexes` above.
obj1 = b"Hello World"
obj1_sha1 = b"1" * 20  # fake SHA1 hash
obj1_sha256 = b"1" * 32  # fake SHA256 hash
creator.add([obj1_sha1, obj1_sha256], obj1)

# Finalize the file (write its indexes)
creator.close()

# When reading, we must choose which index we'll use for lookups
reader = MosaicReader(Path("example.mosaic"), IdxDescription.SHA1FMPHGO)
retrieved = reader.lookup(obj1_sha1)
print(f"Retrieved: {retrieved}")

# The reader provides some meta-data...
print(f"Objects: {reader.objects_counter}")
print(f"Comments: {reader.comments}")

# ... and dict-like iterators
for (obj1_sha1, obj_content) in reader.items():
    with open(obj1_sha1, 'wb') as f:
        f.write(obj_content)

reader.close()

# Objects can be used as context wrappers, to close() automatically
with MosaicReader(Path("example.mosaic"), IdxDescription.SHA256FMPHGO) as reader:
    # get_batch fetches objects in parallel
    fetched = reader.get_batch([obj1_sha256, b"9" * 32])
    assert fetched == [obj1, None]

# MosaicUpdater can erase individual objects in the file and indexes:
with MosaicUpdater(Path("example.mosaic")) as updater:
    updater.delete(
        [
            (IdxDescription.SHA1FMPHGO, obj1_sha1),
            (IdxDescription.SHA256FMPHGO, obj1_sha256),
        ]
    )

reader = MosaicReader(Path("example.mosaic"), IdxDescription.SHA1FMPHGO)
assert list(reader.items()) == []
assert reader.objects_counter == 0

Note that:

  • MosaicReader is optimized for random accesses. If you need a fast iterator over all contents in a MOSAIC, reader.values() will read them in the file’s order.

  • tile_threshold is the target size of “tiles”, that are checksum’d objects groups. This size affects the result of the max_object_size() function, which is usually an order of magnitude bigger than tile_threshold. Objects bigger than max_object_size() will be rejected.

  • This module is implemented in Rust, it mirrors the Rust crate. This allows some functions to use many CPU cores, for example when compressing in MosaicCreator or in MosaicReader.get_batch. If you need to restrict the number of threads, set the environment variable RAYON_NUM_THREADS.

Available Index Types#

The following indexes are supported through the IdxDescription enum. Currently all indexes rely on an FMPHGO MPH, and differ by their keys’ semantics:

  • SHA1FMPHGO: keys are objects’ SHA1

  • SHA1GITFMPHGO: Git-style SHA1

  • SHA256FMPHGO: SHA256

  • BLAKE2FMPHGO: BLAKE2

Context Manager Support#

MosaicReader, MosaicCreator and MosaicUpdater support the context manager protocol:

# Writing with context manager (automatically closes)
with MosaicCreator(
    Path("example.mosaic"),
    indexes=[IdxDescription.SHA1GITFMPHGO]
) as creator:
    creator.add([b"1"*20], b"data")

In that setting,index(es) are written when exiting context.