How to deploy a new lister#

Intended audience

sysadm staff members

This page describes step by step how to deploy and initialize a new lister on the staging infractucture.

Upgrade the packages#

The actions can be launched on all the workers at the same time with clush.

There is more information about clush on the Upgrades page.

With clush on pergamon:

# check current version
clush -b -w @staging-loader-workers dpkg -l python3-swh.lister
# Upgrade
clush -b -w @staging-loader-workers 'apt-get update; apt-get install -y python3-swh.lister'
# check the new version is well deployed
clush -b -w @staging-loader-workers dpkg -l python3-swh.lister

Example of execution:

# check current version
root@pergamon:~# clush -b -w @staging-loader-workers 'dpkg -l python3-swh.lister | grep ii'
ii  python3-swh.lister 0.1.4-0~swh1~bpo10+1 all          Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...)# Upgrade

# Upgrade
root@pergamon:~# clush -b -w @staging-loader-workers 'apt-get update; apt-get install -y python3-swh.lister'
...

# check the new version is well deployed
root@pergamon:~# clush -b -w @staging-loader-workers "dpkg -l python3-swh.lister | grep ii"
ii  python3-swh.lister 0.1.4-1~swh1~bpo10+1 all          Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...)# Upgrade

Upgrade the puppet configuration#

Each type of task is associated to a rabbitmq queue. To have the listers watching to the new(s) queue(s), the configuration deployed by puppet must be updated to reference them.

The exact name of the task to add is the package name and the name declared on the tasks themselves.

The puppet master must be refreshed and the configuration deployed.

Upgrade the puppet master#

On pergamon:

root@pergamon:~# /usr/local/bin/deploy.sh -v

Apply the configuration on workers#

root@pergamon:~# clush -b -w @staging-loader-workers puppet agent -t

Restart the listers#

On pergamon, with clush

clush -b -w @staging-loader-workers 'systemctl restart swh-worker@lister'

Create the model and update the scheduler#

At this stage, the listers are up to date but not yet ready to accept new tasks. The scheduler must be updated with the new lister task type.

Upgrade the scheduler server packages#

On the scheduler server (scheduler0.internal.staging.swh.network on staging) with the *root* user:

# Update the lister package to refresh the cli tool
apt-get update
apt-get install -y python3-swh.lister
# check the version is the same as the workers
dpkg -l python3-swh.lister

Configure the scheduler#

The lister task type must be registered in the scheduler, trigger the swh scheduler task-type register command, on the scheduler server with swhscheduler user:

swh scheduler --config-file /etc/softwareheritage/scheduler.yml \
  task-type register

Example:

swhscheduler@scheduler0:~$ swh scheduler \
  --config-file /etc/softwareheritage/scheduler.yml \
  task-type register
...
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad
INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-incremental in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-full in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-new in scheduler

Register a new task#

The listers and the scheduler are now ready to work together. Use the swh scheduler task add command. Check ref:register-task-type for more details.

Check the logs#

On pergamon with root (or as a sudo user):

clush -b -w @staging-loader-workers 'systemctl status swh-worker@lister'

It will output the status of the listers and the last lines of the lister’s logs on each worker server.