swh-scheduler#

Job scheduler for the Software Heritage project.

Task manager for asynchronous/delayed tasks, used for both recurrent (e.g., listing a forge, loading new stuff from a Git repository) and one-off activities (e.g., loading a specific version of a source package).

Description#

This module provides two independent scheduler services for the Software Heritage platform.

The first one is a generic asynchronous task management system allowing to define tasks with a small number of scheduling properties. In this documentation, we will call these swh-tasks to prevent confusion. These swh-tasks are stored in the scheduler database, and a HTTP-based RPC service is provided to create or find existing swh-task declarations, or select swh-tasks ready for immediate scheduling.

The second one is specific to the scheduling of origin visits (i.e. loading tasks). These visits used to be managed by the generic task system presented above, but this later proved to be less and less suitable to handle the billions of recurring tasks the origin visits require.

The execution model for all the tasks (generic swh-tasks as well as origin visits) is using Celery. Thus, each swh-task type defined in the database must have a (series of) celery worker capable of executing such a swh-task, as well as there must exist celery workers for origin visits.

As mentioned above, in addition to the original and generic task management system, swh-scheduler is now also responsible for keeping track of forge listing and origin loading statistics. These statistics are used to organize the scheduling of future loading tasks according to configurable scheduling policies.

For each of these 2 scheduling management systems, several services are provided to orchestrate the scheduling of these swh-tasks as Celery tasks on the one hand, and origin visits as Celery tasks on the other hand.

Generic task scheduler#

The generic task scheduler consists in a database and its access API, along with a couple of services.

First, the scheduler-runner service is a daemon that regularly looks for swh-tasks in the database that should be scheduled. For each of the selected swh-task, a Celery task is instantiated.

There is also a scheduler-runner-priority service running; this is a scheduler-runner dedicated to schedule tasks with high priority (e.g. tasks resulting from the save code now feature).

Second, the scheduler-listener service is a daemon that listen to the Celery event bus to maintain scheduled swh-tasks workflow status in the database.

Origin visits scheduler#

The scheduler system dedicated to origin visits also consists in a database (it is actually the same database as the generic task scheduler one, but using dedicated tables) and its access API.

The scheduler-schedule-recurrent service is a daemon for choosing which origins are to be visited according to scheduling policies and visit statistics. It serves the same purpose as the scheduler-runner from the generic task scheduler, but it uses different data model and scheduling algorithms.

Last, there is a scheduler-journal-client service which listen to the Kafka journal of the Storage to maintain the loading tasks status and statistics. Once again, the purpose is roughly similar to the scheduler-listener from the generic task scheduler, using Kafka instead of the Celery bus as feedback loop.

Generic SWH Task Model#

Each swh-task-type is the declaration of a type of swh-task. Each swh-task-type have the following fields:

type: Name of the swh-task type; can be anything but must be unique,
description: Human-readable task description
backend_name: Name of the task in the job-running backend,
default_interval: Default interval for newly scheduled tasks,
min_interval: Minimum interval between two runs of a task,
max_interval: Maximum interval between two runs of a task,
backoff_factor: Adjustment factor for the backoff between two task runs,
max_queue_length: Maximum length of the queue for this type of tasks,
num_retries: Default number of retries on transient failures,
retry_delay: Retry delay for the task,

Existing swh-task-types can be listed using the swh scheduler command line tool:

$ swh scheduler task-type list
Known task types:
check-deposit:
  Pre-checking deposit step before loading into swh archive
index-fossology-license:
  Fossology license indexer task
load-git:
  Update an origin of type git
load-hg:
  Update an origin of type mercurial

You can see the details of a swh-task-type:

$ swh scheduler task-type list -v -t load-git
Known task types:
load-git: swh.loader.git.tasks.UpdateGitRepository
  Update an origin of type git
  interval: 64 days, 0:00:00 [12:00:00, 64 days, 0:00:00]
  backoff_factor: 2.0
  max_queue_length: 5000
  num_retries: None
  retry_delay: None

An swh-task is an ‘instance’ of such a swh-task-type, and consists in:

arguments: Arguments passed to the underlying job scheduler,
next_run: Next run of this task should be run on or after that time,
current_interval: Interval between two runs of this task, taking into
account the backoff factor,
policy: Whether the task is “one-shot” or “recurring”,
retries_left: Number of “short delay” retries of the task in case of
transient failure,
priority: Priority of the task,
id: Internal task identifier,
type: References task_type table,
status: Task status ( among “next_run_not_scheduled”, “next_run_scheduled”,
“completed”, “disabled”).

So a swh-task consist basically in:

a set of parameters defining how the scheduling of the swh-task is handled,
a set of parameters to specify the retry policy in case of transient failure upon execution,
a set of parameters that defines the job to be done (bakend_name + arguments).

You can list pending swh-tasks (tasks that are to be scheduled ASAP):

$ swh scheduler task list-pending load-git --limit 2
Found 1 load-git tasks

Task 9052257
  Next run: 15 days ago (2019-06-25 10:35:10+00:00)
  Interval: 2 days, 0:00:00
  Type: load-git
  Policy: recurring
  Args:
    'https://github.com/turtl/mobile'
  Keyword args:

Looking for existing swh-task can be done via the command line tool:

$ swh scheduler task list -t load-hg --limit 2
Found 2 tasks

Task 168802702
  Next run: in 4 hours (2019-07-10 17:56:48+00:00)
  Interval: 1 day, 0:00:00
  Type: load-hg
  Policy: recurring
  Status: next_run_not_scheduled
  Priority:
  Args:
    'https://bitbucket.org/kepung/pypy'
  Keyword args:

Task 169800445
  Next run: in a month (2019-08-10 17:54:24+00:00)
  Interval: 32 days, 0:00:00
  Type: load-hg
  Policy: recurring
  Status: next_run_not_scheduled
  Priority:
  Args:
    'https://bitbucket.org/lunixbochs/pypy-1'
  Keyword args:

Writing a new worker for a new swh-task-type#

When you want to add a new swh-task-type, you need a celery worker backend capable of executing this new task-type instances.

Celery workers for swh-scheduler based tasks should be started using the Celery app in swh.scheduler.celery_config. This later, among other things, provides a loading mechanism for task types based on pkg_resources declared plugins under the [swh.workers] entry point.

TODO: add a fully working example of a dumb task.