- class swh.loader.git.loader.RepoRepresentation(storage, base_snapshots: Optional[List[Snapshot]] = None, incremental: bool = True, statsd: Optional[Statsd] = None)[source]#
Repository representation for a Software Heritage origin.
- class swh.loader.git.loader.FetchPackReturn(remote_refs: Dict[bytes, HexBytes], symbolic_refs: Dict[bytes, HexBytes], pack_buffer: tempfile.SpooledTemporaryFile, pack_size: int)[source]#
- pack_buffer: SpooledTemporaryFile#
- class swh.loader.git.loader.GitLoader(storage: ~swh.storage.interface.StorageInterface, url: str, incremental: bool = True, repo_representation: ~typing.Type[~swh.loader.git.loader.RepoRepresentation] = <class 'swh.loader.git.loader.RepoRepresentation'>, pack_size_bytes: int = 4294967296, temp_file_cutoff: int = 104857600, **kwargs: ~typing.Any)[source]#
A bulk loader for a git repository
Emits the following statsd stats:
swh_loader_git_ignored_refs_percentis the ratio of refs ignored over all refs of the remote repository
swh_loader_git_known_refs_percentis the ratio of (non-ignored) remote heads that are already local over all non-ignored remote heads
All three are tagged with
incremental_modeis one of:
from_same_originwhen the origin was already loaded
from_parent_originwhen the origin was not already loaded, but it was detected as a forge-fork of an origin that was already loaded
no_previous_snapshotwhen the origin was not already loaded, and it was detected as a forge-fork of origins that were not already loaded either
no_parent_originwhen the origin was no already loaded, and it was not detected as a forge-fork of any other origin
disabledwhen incremental loading is disabled by configuration
Initialize the bulk updater.
repo_representation – swh’s repository representation
remote (which is in charge of filtering between known and) –
incremental – If True, the default, this starts from the last known snapshot (if any) references. Otherwise, this loads the full repository.
- fetch_pack_from_origin(origin_url: str, base_repo: RepoRepresentation, do_activity: Callable[[bytes], None]) FetchPackReturn [source]#
Fetch a pack from the origin
- prepare() None [source]#
- Second step executed by the loader to prepare some state needed by
NotFound exception if the origin to ingest is not found.
- fetch_data() bool [source]#
- Fetch the data from the source the loader is currently loading
(ex: git/hg/svn/… repository).
a value that is interpreted as a boolean. If True, fetch_data needs to be called again to complete loading.
- iter_objects(object_type: bytes) Iterator[ShaFile] [source]#
Read all the objects of type object_type from the packfile
- get_contents() Iterable[BaseContent] [source]#
Format the blobs from the git repository as swh contents
- get_snapshot() Snapshot [source]#
Get the snapshot for the current visit.
The main complexity of this function is mapping target objects to their types, as the refs dictionaries returned by the git server only give us the identifiers for the target objects, and not their types.
The loader itself only knows the types of the objects that it has fetched from the server (as it has parsed them while loading them to the archive). As we only fetched an increment between the previous snapshot and the current state of the server, we are missing the type information for the objects that would already have been referenced by the previous snapshot, and that the git server didn’t send us. We infer the type of these objects from the previous snapshot.
- load_status() Dict[str, Any] [source]#
The load was eventful if the current snapshot is different to the one we retrieved at the beginning of the run