- class swh.loader.core.loader.BaseLoader(storage: swh.storage.interface.StorageInterface, logging_class: Optional[str] = None, save_data_path: Optional[str] = None, max_content_size: Optional[int] = None)
Base class for (D)VCS loaders (e.g Svn, Git, Mercurial, …) or PackageLoader (e.g PyPI, Npm, CRAN, …)
A loader retrieves origin information (git/mercurial/svn repositories, pypi/npm/… package artifacts), ingests the contents/directories/revisions/releases/snapshot read from those artifacts and send them to the archive through the storage backend.
The main entry point for the loader is the
Some class examples:
- classmethod from_config(storage: Dict[str, Any], **config: Any)
Instantiate a loader from a configuration dict.
This is basically a backwards-compatibility shim for the CLI.
storage – instantiation config for the storage
config – the configuration dict for the loader, with the following keys: - credentials (optional): credentials list for the scheduler - any other kwargs passed to the loader.
the instantiated loader
- classmethod from_configfile(**kwargs: Any)
Instantiate a loader from the configuration loaded from the SWH_CONFIG_FILENAME envvar, with potential extra keyword arguments if their value is not None.
kwargs – kwargs passed to the loader instantiation
- prepare_origin_visit() None
First step executed by the loader to prepare origin and visit references. Set/update self.origin, and optionally self.origin_url, self.visit_date.
- prepare() None
- Second step executed by the loader to prepare some state needed by
NotFound exception if the origin to ingest is not found.
- get_origin() swh.model.model.Origin
Get the origin that is currently being loaded. self.origin should be set in
an origin ready to be sent to storage by
- Return type
- fetch_data() bool
- Fetch the data from the source the loader is currently loading
(ex: git/hg/svn/… repository).
a value that is interpreted as a boolean. If True, fetch_data needs to be called again to complete loading.
Store fetched data in the database.
Should call the
maybe_load_xyz()methods, which handle the bundles sent to storage, rather than send directly.
- store_metadata() None
Store fetched metadata in the database.
For more information, see implementation in
- load_status() Dict[str, str]
Detailed loading status.
Defaults to logging an eventful load.
- Returns: a dictionary that is eventually passed back as the task’s
result to the scheduler, allowing tuning of the task recurrence mechanism.
- post_load(success: bool = True) None
Permit the loader to do some additional actions according to status after the loading is done. The flag success indicates the loading’s status.
Defaults to doing nothing.
This is up to the implementer of this method to make sure this does not break.
success (bool) – the success status of the loading
- pre_cleanup() None
As a first step, will try and check for dangling data to cleanup. This should do its best to avoid raising issues.
- load() Dict[str, str]
Loading logic for the loader to follow:
prepare_origin_visit()to prepare the origin and visit we will associate loading data to
Store the actual
prepare()to prepare any eventual state
get_origin()to get the origin we work with and store
- class swh.loader.core.loader.DVCSLoader(storage: swh.storage.interface.StorageInterface, logging_class: Optional[str] = None, save_data_path: Optional[str] = None, max_content_size: Optional[int] = None)
This base class is a pattern for dvcs loaders (e.g. git, mercurial).
Those loaders are able to load all the data in one go. For example, the loader defined in swh-loader-git
For other loaders (stateful one, (e.g
SWHSvnLoader), inherit directly from