Software Heritage - Loader foundations

The Software Heritage Loader Core is a low-level loading utilities and helpers used by loaders.

The main entry points are classes: - swh.loader.core.loader.BaseLoader for VCS loaders (e.g. svn) - swh.loader.core.loader.DVCSLoader for DVCS loaders (e.g. git, …) - swh.loader.core.loader.ContentLoader for Content loader - swh.loader.core.loader.DirectoryLoader for Directory loader - swh.loader.package.loader.PackageLoader for Package loaders (e.g. PyPI, Npm, …) - …

Package loaders

This package also implements many package loaders directly, out of convenience, as they usually are quite similar and each fits in a single file.

They all roughly follow these steps, explained in the swh.loader.package.loader.PackageLoader.load() documentation. See the Package Loader Tutorial for details.

VCS loaders

Unlike package loaders, VCS loaders remain in separate packages, as they often need more advanced conversions and very VCS-specific operations.

This usually involves getting the branches of a repository and recursively loading revisions in the history (and directory trees in these revisions), until a known revision is found

Contents:

• VCS Loader Overview
  • Architecture
  • Base classes
  • Incremental loading
  • Integrity
• Package Loader Tutorial
  • Creating the files hierarchy
  • Writing a minimal loader
  • Making your loader incremental
  • Loading metadata
  • Checklist
  • Final words
• Package loader specifications
  • Release fields

Reference Documentation

• Command-line interface
  • swh loader