Software Heritage - Development Documentation
Data Model and Specifications
Here is brief overview of the most relevant software components in the Software Heritage stack, in alphabetical order. For a better introduction to the architecture, see the Software Architecture Overview, which presents each of them in a didactical order.
Each component name is linked to the development documentation of the corresponding Python module.
low-level library used by modules needing keycloak authentication
low-level utilities and helpers used by almost all other modules in the stack
service providing efficient estimates of the number of objects in the SWH archive, using Redis’s Hyperloglog
public datasets and periodic data dumps of the archive released by Software Heritage
push-based deposit of software artifacts to the archive
developer documentation (used to generate this doc you are reading)
Virtual file system to browse the Software Heritage archive, based on FUSE
Fast, compressed, in-memory representation of the archive, with tooling to generate and query it.
GraphQL API to request archive data offering more precise and flexible queries than the REST API.
tools and workers used to crawl the content of the archive and extract derived information from any artifact stored in it
persistent logger of changes to the archive, with publish-subscribe support
collection of listers for all sorts of source code hosting and distribution places (forges, distributions, package managers, etc.)
low-level loading utilities and helpers used by all other loaders
loader for Git repositories
loader for Mercurial repositories
pseudo-loader, which fetches extrinsic metadata from forges instead of software artifacts
loader for Subversion repositories
loader for CVS repositories
implementation of the Data model to archive source code artifacts
content-addressable object storage
Object storage replication tool
Low level management for read-only content-addressable object storage indexed with a perfect hash table
source code scanner to analyze code bases and compare them with source code artifacts archived by Software Heritage
task manager for asynchronous/delayed tasks, used for recurrent (e.g., listing a forge, loading new stuff from a Git repository) and one-off activities (e.g., loading a specific version of a source package)
Tooling to check integrity of various data stores (swh.journal, swh.objstorage, swh.storage) and fix corrupt objects they contain.
search engine for the archive
abstraction layer over the archive, allowing to access all stored source code artifacts as well as their metadata
implementation of the vault service, allowing to retrieve parts of the archive as self-contained bundles (e.g., individual releases, entire repository snapshots, etc.)
Web application(s) to browse the archive, for both interactive (HTML UI) and mechanized (REST API) use
Python client for swh.web
The dependency relationships among the various modules are depicted below.
Archive ChangeLog: notable changes to the archive over time