Software Heritage - Loader foundations

The Software Heritage Loader Core is a low-level loading utilities and helpers used by loaders.

The main entry points are classes: - swh.loader.core.loader.BaseLoader for loaders (e.g. svn) - swh.loader.core.loader.DVCSLoader for DVCS loaders (e.g. hg, git, …) - swh.loader.package.loader.PackageLoader for Package loaders (e.g. PyPI, Npm, …)

Package loaders

This package also implements many package loaders directly, out of convenience, as they usually are quite similar and each fits in a single file.

They all roughly follow these steps, explained in the swh.loader.package.loader.PackageLoader.load() documentation. See the Package Loader Tutorial for details.

VCS loaders

Unlike package loaders, VCS loaders remain in separate packages, as they often need more advanced conversions and very VCS-specific operations.

This usually involves getting the branches of a repository and recursively loading revisions in the history (and directory trees in these revisions), until a known revision is found

Contents:

• Package Loader Tutorial
  • Creating the files hierarchy
  • Writing a minimal loader
  • Making your loader incremental
  • Loading metadata
  • Final words
• VCS Loader Overview
  • Architecture
  • Base classes
  • Incremental loading
  • Integrity

Reference Documentation

• Command-line interface
  • swh loader
• swh.loader.core package
  • swh.loader.core.converters module
  • swh.loader.core.loader module
  • swh.loader.core.utils module
• swh.loader.package package
  • swh.loader.package.archive package
  • swh.loader.package.cran package
  • swh.loader.package.debian package
  • swh.loader.package.deposit package
  • swh.loader.package.nixguix package
  • swh.loader.package.npm package
  • swh.loader.package.pypi package
  • swh.loader.package.loader module
  • swh.loader.package.utils module