Add-forge-now automation#

Intended audience

sysadm staff members

When a request status is updated to Accepted, a Gitlab pipeline is triggered to process the request.

Pipeline#

To overview the pipelines, see Add-forge-now-requests Pipelines.
To see only the running pipelines, see Add-forge-now-requests Running Pipelines.
To view the pipeline details, click on the pipeline status.
To view the job details, click on the stage and choose a job.

Concurrent jobs

Only ten concurrent pipelines can be processed at the same time.
It can be changed on runner0:
root@runner0:~# awk '/concurrent/' config.toml
concurrent = 10

afnr_pipeline

Pre stage#

The first job create a Gitlab issue with the forge URL in the title.
To overview the issues, see AFN issues,.
The second job check the SWH components ports, the forge URL (https) availability and the webapp token validity (recreate it if needed).
The last job of the pre stage add a comment on the webapp request with the Gitlab issue URL.

afnr_pipeline_pre_stage

Staging stages#

The second, third and fourth stages process the request in staging environment.
If all the jobs are successfully completed, the request status is updated to Scheduled and a comment is added to the Gitlab issue.

afnr_pipeline_staging_stages

Production stages#

The fifth, sixth and seventh stages process the request in production environment.
If all the jobs are successfully completed, the request status is updated to First Origins Loaded, a comment is added to the Gitlab issue and this issue is closed.

pipeline_production_stages

Request status#

  • pre stage adds a comment on request;

  • if the staging stages are successfully completed, the request status is updated to Scheduled;

  • if the production stages are successfully completed, the request status is updated to First Origin Loaded.

afnr_request_status

All comments contain the url of the issue where to find the associated pipeline.

afnr_request_comment

Common errors#

afnr_errors

A non-exhaustive list of common pipeline errors is:

  • Stage pre, job 02_check_ports_and_token:

    • error: the network ports checks failed;

    • cause: one of the SWH components are unreachable or the forge URL (https) is unreachable;

    • relaunch the job.

  • Stage *_list_origins, job 2*_check_listed_origins:

    • error: Forge <forge> (<type>) isn't registered in the scheduler database.;

    • cause: the lister pod creation took more time than the listing delay (600s);

    • relaunch the job.

  • Stage *_list_origins, job 2*_check_listed_origins:

    • error: Forge <forge> (<type>) has 0 listed origin in the scheduler database.;

    • cause: the lister failed to find some origin;

    • process request manually (most of the time, it needs the optional base_git_url parameter). See How to process add-forge-now requests

  • Stage *_ingest_origins, job 4*_check_ingested_origins:

    • error: There are too many ingestion failures.;

    • cause: ingestion failures > 70%;

    • check ingestion (e.g. lots of empty repositories,…) and/or swh platform (e.g. objstorage unavailable,…); process request manually (combination of url and base_git_url).

  • Any stage, any job (generally in job 4p_check-ingested-origins):

    • error: ERROR: Job failed: execution took longer than 168h0m0s seconds;

    • cause: job timeout (7d);

    • relaunch the job.

Gitlab-runner#

The Gitlab-runner is dedicated to the add-forge-now-requests project.
The CI/CD settings page contains:
- the Runners section to check the runner’s status and update the runner configuration;

afnr_gitlab-runner

- the Variables section to check, retrieve or update the variables that need to be protected and masked;

Webapp Token

If the webapp token is expired and not renewed (normally it should), the pipeline will fail in third job of pre stage. You can update the variable WEBAPP_TOKEN in this section.

- the Pipeline trigger tokens section to check or change the pipeline token.

It runs on runner0 virtual machine with a docker executor.

docker-compose.yml

version: "3"
services:
  gitlab-runner:
    image: gitlab/gitlab-runner:alpine
    container_name: gitlab-runner
    restart: always
    volumes:
      - "/run/docker.sock:/var/run/docker.sock"
      - "gitlab_gitlab-runner:/etc/gitlab-runner"
      - "./config.toml:/etc/gitlab-runner/config.toml"

volumes:
  gitlab_gitlab-runner:

Runner configuration file config.toml

concurrent = 10
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "add-forge-now-runner"
  url = "https://gitlab.softwareheritage.org"
  id = 4
  token = "xxx"
  token_obtained_at = 2023-10-12T14:12:16Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "ruby:2.7"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache",
        "/etc/swh/scheduler-staging.yml:/etc/swh/scheduler-staging.yml",
        "/etc/swh/scheduler-production.yml:/etc/swh/scheduler-production.yml"]
    shm_size = 0

Check the runner version and status:

root@runner0:~# alias dck
alias dck='docker compose'
root@runner0:~# dck exec -ti gitlab-runner gitlab-runner --version
Version:      16.10.0
Git revision: 81ab07f6
Git branch:   16-10-stable
GO version:   go1.21.7
Built:        2024-03-21T19:43:25+0000
OS/Arch:      linux/amd64
root@runner0:~# dck exec -ti gitlab-runner gitlab-runner verify
Runtime platform                                    arch=amd64 os=linux pid=177 revision=81ab07f6 version=16.10.0
Running in system-mode.

Verifying runner... is valid                        runner=6aU8maWXg