swh.lister.arch package#

Submodules#

Module contents#

Arch Linux lister#

The Arch lister list origins from archlinux.org, the official Arch Linux packages, and from archlinuxarm.org, the Arch Linux ARM packages, an unofficial port for arm.

Packages are put in three different repositories, core, extra and community.

To manage listing those origins, this lister must be instantiated with a flavours dict.

flavours default values:

"official": {
    "archs": ["x86_64"],
    "repos": ["core", "extra", "community"],
    "base_info_url": "https://archlinux.org",
    "base_archive_url": "https://archive.archlinux.org",
    "base_mirror_url": "",
    "base_api_url": "https://archlinux.org",
},
"arm": {
    "archs": ["armv7h", "aarch64"],
    "repos": ["core", "extra", "community"],
    "base_info_url": "https://archlinuxarm.org",
    "base_archive_url": "",
    "base_mirror_url": "https://uk.mirror.archlinuxarm.org",
    "base_api_url": "",
}

From official Arch Linux repositories we can list all packages and all released versions. They provide an api and archives.

From Arch Linux ARM repositories we can list all packages at their latest versions, they do not provide api or archives.

As of August 2022 archlinux.org list 12592 packages and archlinuxarm.org 24044 packages. Please note that those amounts are the total of regular and split packages.

Origins retrieving strategy#

Download repositories archives as tar.gz files from https://archive.archlinux.org/repos/last/, extract to a temp directory and then walks through each ‘desc’ files. Repository archive index url example for Arch Linux core repository and Arch Linux ARM extra repository.

Each ‘desc’ file describe the latest released version of a package and helps to build an origin url and package versions url from where scrapping artifacts metadata and get a list of versions.

For Arch Linux ARM it follow the same discovery process parsing ‘desc’ files. The main difference is that we can’t get existing versions of an arm package because https://archlinuxarm.org does not have an ‘archive’ website or api.

Page listing#

Each page is a list of package belonging to a flavour (‘official’, ‘arm’), and a repo (‘core’, ‘extra’, ‘community’).

Each line of a page represents an origin url for a package name with related metadata and versions.

Origin url examples:

Arch Linux: https://archlinux.org/packages/extra/x86_64/mercurial
Arch Linux ARM: https://archlinuxarm.org/packages/armv7h/mercurial

The data schema for each line is:

name: Package name
version: Last released package version
last_modified: Iso8601 last modified date from timestamp
url: Origin url
data: Package metadata dict
versions: A list of dict with artifacts metadata for each versions

The data schema for versions within a line:

name: Package name
version: Package version
repo: One of core, extra, community
arch: Processor architecture targeted
filename: Filename of the archive to download
url: Package download url
last_modified: Iso8601 last modified date from timestamp, used as publication date for this version
length: Length of the archive to download

Origins from page#

The origin url corresponds to:

Arch Linux: https://archlinux.org/packages/{repo}/{arch}/{name}
Arch Linux ARM: https://archlinuxarm.org/packages/{arch}/{name}

Additionally we add some data set to “extra_loader_arguments”:

artifacts: Represent data about the Arch Linux package archive to download, following original-artifacts-json specification
arch_metadata: To store all other interesting attributes that do not belongs to artifacts.

Origin data example Arch Linux official:

{
    "url": "https://archlinux.org/packages/extra/x86_64/mercurial",
    "visit_type": "arch",
    "extra_loader_arguments": {
        "artifacts": [
            {
                "url": "https://archive.archlinux.org/packages/m/mercurial/mercurial-4.8.2-1-x86_64.pkg.tar.xz",  # noqa: B950
                "version": "4.8.2-1",
                "length": 4000000,
                "filename": "mercurial-4.8.2-1-x86_64.pkg.tar.xz",
            },
            {
                "url": "https://archive.archlinux.org/packages/m/mercurial/mercurial-4.9-1-x86_64.pkg.tar.xz",  # noqa: B950
                "version": "4.9-1",
                "length": 4000000,
                "filename": "mercurial-4.9-1-x86_64.pkg.tar.xz",
            },
            {
                "url": "https://archive.archlinux.org/packages/m/mercurial/mercurial-4.9.1-1-x86_64.pkg.tar.xz",  # noqa: B950
                "version": "4.9.1-1",
                "length": 4000000,
                "filename": "mercurial-4.9.1-1-x86_64.pkg.tar.xz",
            },
            ...
        ],
        "arch_metadata": [
            {
                "arch": "x86_64",
                "repo": "extra",
                "name": "mercurial",
                "version": "4.8.2-1",
                "last_modified": "2019-01-15T20:31:00",
            },
            {
                "arch": "x86_64",
                "repo": "extra",
                "name": "mercurial",
                "version": "4.9-1",
                "last_modified": "2019-02-12T06:15:00",
            },
            {
                "arch": "x86_64",
                "repo": "extra",
                "name": "mercurial",
                "version": "4.9.1-1",
                "last_modified": "2019-03-30T17:40:00",
            },
        ],
    },
},

Origin data example Arch Linux ARM:

{
    "url": "https://archlinuxarm.org/packages/armv7h/mercurial",
    "visit_type": "arch",
    "extra_loader_arguments": {
        "artifacts": [
            {
                "url": "https://uk.mirror.archlinuxarm.org/armv7h/extra/mercurial-6.1.3-1-armv7h.pkg.tar.xz",  # noqa: B950
                "length": 4897816,
                "version": "6.1.3-1",
                "filename": "mercurial-6.1.3-1-armv7h.pkg.tar.xz",
            }
        ],
        "arch_metadata": [
            {
                "arch": "armv7h",
                "name": "mercurial",
                "repo": "extra",
                "version": "6.1.3-1",
                "last_modified": "2022-06-02T22:13:08",
            }
        ],
    },
},

Running tests#

Activate the virtualenv and run from within swh-lister directory:

pytest -s -vv --log-cli-level=DEBUG swh/lister/arch/tests

Testing with Docker#

Change directory to swh/docker then launch the docker environment:

docker compose up -d

Then schedule an arch listing task:

docker compose exec swh-scheduler swh scheduler task add -p oneshot list-arch

You can follow lister execution by displaying logs of swh-lister service:

docker compose logs -f swh-lister

swh.lister.arch.register()[source]#