swh.lister.nixguix.lister module

NixGuix lister definition.

This lists artifacts out of manifest for Guix or Nixpkgs manifests.

Artifacts can be of types: - upstream git repository (NixOS/nixpkgs, Guix) - VCS repositories (svn, git, hg, …) - unique file - unique tarball

exception swh.lister.nixguix.lister.ArtifactNatureUndetected

Bases: ValueError

Raised when a remote artifact’s nature (tarball, file) cannot be detected.

exception swh.lister.nixguix.lister.ArtifactNatureMistyped

Bases: ValueError

Raised when a remote artifact is neither a tarball nor a file.

Error of this type are’ probably a misconfiguration in the manifest generation that badly typed a vcs repository.

exception swh.lister.nixguix.lister.ArtifactWithoutExtension

Bases: ValueError

Raised when an artifact nature cannot be determined by its name.

class swh.lister.nixguix.lister.ChecksumLayout(value)

Bases: Enum

The possible artifact types listed out of the manifest.

STANDARD = 'standard'

Standard “flat” checksums (e.g. sha1, sha256, …) on the tarball or file.

NAR = 'nar'

The checksum(s) are computed over the NAR dump of the output (e.g. uncompressed directory.). That uncompressed directory can come from a tarball or a (d)vcs. It’s also called “recursive” in the “outputHashMode” key in the upstream dataset.

swh.lister.nixguix.lister.MAPPING_CHECKSUM_LAYOUT = {'flat': ChecksumLayout.STANDARD, 'recursive': ChecksumLayout.NAR}

Mapping between the outputHashMode from the manifest and how to compute checksums.

class swh.lister.nixguix.lister.Artifact(origin: str, visit_type: str, fallback_urls: List[str], checksums: Dict[str, str], checksum_layout: ChecksumLayout, ref: Optional[str])

Bases: object

Metadata information on Remote Artifact with url (tarball or file).

origin: str

Canonical url retrieve the tarball artifact.

visit_type: str

Either ‘tar’ or ‘file’

fallback_urls: List[str]

List of urls to retrieve tarball artifact if canonical url no longer works.

checksums: Dict[str, str]

Integrity hash converted into a checksum dict.

checksum_layout: ChecksumLayout

Checksum layout mode to provide to loaders (e.g. nar, standard, …)

ref: Optional[str]

Optional reference on the artifact (git commit, branch, svn commit, tag, …)

class swh.lister.nixguix.lister.VCS(origin: str, type: str)

Bases: object

Metadata information on VCS.

origin: str

Origin url of the vcs

type: str

Type of (d)vcs, e.g. svn, git, hg, …

class swh.lister.nixguix.lister.ArtifactType(value)

Bases: Enum

The possible artifact types listed out of the manifest.

ARTIFACT = 'artifact'

VCS = 'vcs'

swh.lister.nixguix.lister.url_endswith(urlparsed, extensions: List[str], raise_when_no_extension: bool = True) → bool

Determine whether urlparsed ends with one of the extensions passed as parameter.

This also account for the edge case of a filename with only a version as name (so no extension in the end.)

Raises:

• ArtifactWithoutExtension in case no extension is available and –

• raise_when_no_extension is True (the default) –

swh.lister.nixguix.lister.is_tarball(urls: List[str], request: Optional[Any] = None) → Tuple[bool, str]

Determine whether a list of files actually are tarball or simple files.

This iterates over the list of urls provided to detect the artifact’s nature. When this cannot be answered simply out of the url and request is provided, this executes a HTTP HEAD query on the url to determine the information. If request is not provided, this raises an ArtifactNatureUndetected exception.

If, at the end of the iteration on the urls, no detection could be deduced, this raises an ArtifactNatureUndetected.

Parameters:

• urls – name of the remote files to check for artifact nature.

• request – (Optional) Request object allowing http calls. If not provided and naive check cannot detect anything, this raises ArtifactNatureUndetected.

Raises:

• ArtifactNatureUndetected when the artifact's nature cannot be detected out – of its urls

• ArtifactNatureMistyped when the artifact is not a tarball nor a file. It's up to – the caller to do what’s right with it.

Returns: A tuple (bool, url). The boolean represents whether the url is an archive

or not. The second parameter is the actual url once the head request is issued as a fallback of not finding out whether the urls are tarballs or not.

swh.lister.nixguix.lister.VCS_ARTIFACT_TYPE_TO_VISIT_TYPE = {'git': 'git-checkout', 'hg': 'hg-checkout', 'svn': 'svn-export'}

Mapping between the vcs artifact type to the loader’s visit type.

class swh.lister.nixguix.lister.NixGuixLister(scheduler, url: str, origin_upstream: str, instance: Optional[str] = None, credentials: Optional[Dict[str, Dict[str, List[Dict[str, str]]]]] = None, max_origins_per_page: Optional[int] = None, max_pages: Optional[int] = None, enable_origins: bool = True, canonicalize: bool = True, extensions_to_ignore: List[str] = [], **kwargs: Any)

Bases: StatelessLister[Tuple[ArtifactType, Union[Artifact, VCS]]]

List Guix or Nix sources out of a public json manifest.

This lister can output: - unique tarball (.tar.gz, .tbz2, …) - vcs repositories (e.g. git, hg, svn) - unique file (.lisp, .py, …)

In the case of vcs repositories, if a reference is provided (git_ref, svn_revision or hg_changeset with a specific outputHashMode recursive), this provides one more origin to ingest as ‘directory’. The DirectoryLoader will then be in charge to ingest the origin (checking the associated integrity field first).

Note that, no last_update is available in either manifest (guix or nixpkgs), so listed_origins do not have it set.

For url types artifacts, this tries to determine the artifact’s nature, tarball or file. It first tries to compute out of the “url” extension. In case of no extension, it fallbacks to (HEAD) query the url to retrieve the origin out of the Location response header, and then checks the extension again.

Optionally, when the extension_to_ignore parameter is provided, it extends the default extensions to ignore (DEFAULT_EXTENSIONS_TO_IGNORE) with those passed. This can be optionally used to filter some more binary files detected in the wild.

LISTER_NAME: str = 'nixguix'

build_artifact(artifact_url: str, artifact_type: str) → Optional[Tuple[ArtifactType, VCS]]

Build a canonicalized vcs artifact when possible.

convert_integrity_to_checksums(integrity: str, failure_log: str) → Optional[Dict[str, str]]

Determine the content checksum stored in the integrity field and convert into a dict of checksums. This only parses the hash-expression (hash-<b64-encoded-checksum>) as defined in https://w3c.github.io/webappsec-subresource-integrity/#the-integrity-attribute

get_pages() → Iterator[Tuple[ArtifactType, Union[Artifact, VCS]]]

Yield one page per “typed” origin referenced in manifest.

vcs_to_listed_origin(artifact: VCS) → Iterator[ListedOrigin]

Given a vcs repository, yield a ListedOrigin.

artifact_to_listed_origin(artifact: Artifact) → Iterator[ListedOrigin]

Given an artifact (tarball, file), yield one ListedOrigin.

get_origins_from_page(artifact_tuple: Tuple[ArtifactType, Union[Artifact, VCS]]) → Iterator[ListedOrigin]

Given an artifact tuple (type, artifact), yield a ListedOrigin.