swh.lister.nixguix.lister module#
NixGuix lister definition.
This lists artifacts out of manifest for Guix or Nixpkgs manifests.
Artifacts can be of types: - upstream git repository (NixOS/nixpkgs, Guix) - VCS repositories (svn, git, hg, …) - unique file - unique tarball
- exception swh.lister.nixguix.lister.ArtifactNatureUndetected[source]#
Bases:
ValueError
Raised when a remote artifact’s nature (tarball, file) cannot be detected.
- exception swh.lister.nixguix.lister.ArtifactNatureMistyped[source]#
Bases:
ValueError
Raised when a remote artifact is neither a tarball nor a file.
Error of this type are’ probably a misconfiguration in the manifest generation that badly typed a vcs repository.
- exception swh.lister.nixguix.lister.ArtifactWithoutExtension[source]#
Bases:
ValueError
Raised when an artifact nature cannot be determined by its name.
- class swh.lister.nixguix.lister.ChecksumLayout(value)[source]#
Bases:
Enum
The possible artifact types listed out of the manifest.
- STANDARD = 'standard'#
Standard “flat” checksums (e.g. sha1, sha256, …) on the tarball or file.
- NAR = 'nar'#
The checksum(s) are computed over the NAR dump of the output (e.g. uncompressed directory.). That uncompressed directory can come from a tarball or a (d)vcs. It’s also called “recursive” in the “outputHashMode” key in the upstream dataset.
- swh.lister.nixguix.lister.MAPPING_CHECKSUM_LAYOUT = {'flat': ChecksumLayout.STANDARD, 'recursive': ChecksumLayout.NAR}#
Mapping between the outputHashMode from the manifest and how to compute checksums.
- class swh.lister.nixguix.lister.Artifact(origin: str, visit_type: str, fallback_urls: List[str], checksums: Dict[str, str], checksum_layout: ChecksumLayout, ref: Optional[str])[source]#
Bases:
object
Metadata information on Remote Artifact with url (tarball or file).
- fallback_urls: List[str]#
List of urls to retrieve tarball artifact if canonical url no longer works.
- checksum_layout: ChecksumLayout#
Checksum layout mode to provide to loaders (e.g. nar, standard, …)
- class swh.lister.nixguix.lister.VCS(origin: str, type: str)[source]#
Bases:
object
Metadata information on VCS.
- class swh.lister.nixguix.lister.ArtifactType(value)[source]#
Bases:
Enum
The possible artifact types listed out of the manifest.
- ARTIFACT = 'artifact'#
- VCS = 'vcs'#
- swh.lister.nixguix.lister.url_endswith(urlparsed, extensions: List[str], raise_when_no_extension: bool = True) bool [source]#
Determine whether urlparsed ends with one of the extensions passed as parameter.
This also account for the edge case of a filename with only a version as name (so no extension in the end.)
- Raises:
ArtifactWithoutExtension in case no extension is available and –
raise_when_no_extension is True (the default) –
- swh.lister.nixguix.lister.is_tarball(urls: List[str], request: Optional[Any] = None) Tuple[bool, str] [source]#
Determine whether a list of files actually are tarball or simple files.
This iterates over the list of urls provided to detect the artifact’s nature. When this cannot be answered simply out of the url and
request
is provided, this executes a HTTP HEAD query on the url to determine the information. If request is not provided, this raises an ArtifactNatureUndetected exception.If, at the end of the iteration on the urls, no detection could be deduced, this raises an ArtifactNatureUndetected.
- Parameters:
urls – name of the remote files to check for artifact nature.
request – (Optional) Request object allowing http calls. If not provided and naive check cannot detect anything, this raises ArtifactNatureUndetected.
- Raises:
ArtifactNatureUndetected when the artifact's nature cannot be detected out – of its urls
ArtifactNatureMistyped when the artifact is not a tarball nor a file. It's up to – the caller to do what’s right with it.
- Returns: A tuple (bool, url). The boolean represents whether the url is an archive
or not. The second parameter is the actual url once the head request is issued as a fallback of not finding out whether the urls are tarballs or not.
- swh.lister.nixguix.lister.VCS_ARTIFACT_TYPE_TO_VISIT_TYPE = {'git': 'git-checkout', 'hg': 'hg-checkout', 'svn': 'svn-export'}#
Mapping between the vcs artifact type to the loader’s visit type.
- class swh.lister.nixguix.lister.NixGuixLister(scheduler, url: str, origin_upstream: str, instance: Optional[str] = None, credentials: Optional[Dict[str, Dict[str, List[Dict[str, str]]]]] = None, max_origins_per_page: Optional[int] = None, max_pages: Optional[int] = None, enable_origins: bool = True, canonicalize: bool = True, extensions_to_ignore: List[str] = [], **kwargs: Any)[source]#
Bases:
StatelessLister
[Tuple
[ArtifactType
,Union
[Artifact
,VCS
]]]List Guix or Nix sources out of a public json manifest.
This lister can output: - unique tarball (.tar.gz, .tbz2, …) - vcs repositories (e.g. git, hg, svn) - unique file (.lisp, .py, …)
In the case of vcs repositories, if a reference is provided (
git_ref
,svn_revision
orhg_changeset
with a specificoutputHashMode
recursive
), this provides one more origin to ingest as ‘directory’. The DirectoryLoader will then be in charge to ingest the origin (checking the associatedintegrity
field first).Note that, no last_update is available in either manifest (
guix
ornixpkgs
), so listed_origins do not have it set.For url types artifacts, this tries to determine the artifact’s nature, tarball or file. It first tries to compute out of the “url” extension. In case of no extension, it fallbacks to (HEAD) query the url to retrieve the origin out of the Location response header, and then checks the extension again.
Optionally, when the extension_to_ignore parameter is provided, it extends the default extensions to ignore (DEFAULT_EXTENSIONS_TO_IGNORE) with those passed. This can be optionally used to filter some more binary files detected in the wild.
- build_artifact(artifact_url: str, artifact_type: str) Optional[Tuple[ArtifactType, VCS]] [source]#
Build a canonicalized vcs artifact when possible.
- convert_integrity_to_checksums(integrity: str, failure_log: str) Optional[Dict[str, str]] [source]#
Determine the content checksum stored in the integrity field and convert into a dict of checksums. This only parses the hash-expression (hash-<b64-encoded-checksum>) as defined in https://w3c.github.io/webappsec-subresource-integrity/#the-integrity-attribute
- get_pages() Iterator[Tuple[ArtifactType, Union[Artifact, VCS]]] [source]#
Yield one page per “typed” origin referenced in manifest.
- vcs_to_listed_origin(artifact: VCS) Iterator[ListedOrigin] [source]#
Given a vcs repository, yield a ListedOrigin.
- artifact_to_listed_origin(artifact: Artifact) Iterator[ListedOrigin] [source]#
Given an artifact (tarball, file), yield one ListedOrigin.
- get_origins_from_page(artifact_tuple: Tuple[ArtifactType, Union[Artifact, VCS]]) Iterator[ListedOrigin] [source]#
Given an artifact tuple (type, artifact), yield a ListedOrigin.