Software Heritage - Job scheduler
Task manager for asynchronous/delayed tasks, used for both recurrent (e.g., listing a forge, loading new stuff from a Git repository) and one-off activities (e.g., loading a specific version of a source package).
This module provides two independent scheduler services for the Software Heritage platform.
The first one is a generic asynchronous task management system allowing to define tasks with a small number of scheduling properties. In this documentation, we will call these swh-tasks to prevent confusion. These swh-tasks are stored in the scheduler database, and a HTTP-based RPC service is provided to create or find existing swh-task declarations, or select swh-tasks ready for immediate scheduling.
The second one is specific to the scheduling of origin visits (i.e. loading tasks). These visits used to be managed by the generic task system presented above, but this later proved to be less and less suitable to handle the billions of recurring tasks the origin visits require.
The execution model for all the tasks (generic swh-tasks as well as origin visits) is using Celery. Thus, each swh-task type defined in the database must have a (series of) celery worker capable of executing such a swh-task, as well as there must exist celery workers for origin visits.
As mentioned above, in addition to the original and generic task management
swh-scheduler is now also responsible for keeping track of forge
listing and origin loading statistics. These statistics are used to organize
the scheduling of future loading tasks according to configurable scheduling
For each of these 2 scheduling management systems, several services are provided to orchestrate the scheduling of these swh-tasks as Celery tasks on the one hand, and origin visits as Celery tasks on the other hand.
Generic task scheduler
The generic task scheduler consists in a database and its access API, along with a couple of services.
scheduler-runner service is a daemon that regularly looks for
swh-tasks in the database that should be scheduled. For each of the selected
swh-task, a Celery task is instantiated.
There is also a
scheduler-runner-priority service running; this
scheduler-runner dedicated to schedule tasks with high priority (e.g.
tasks resulting from the
save code now feature).
scheduler-listener service is a daemon that listen to the
Celery event bus to maintain scheduled swh-tasks workflow status in the
Origin visits scheduler
The scheduler system dedicated to origin visits also consists in a database (it is actually the same database as the generic task scheduler one, but using dedicated tables) and its access API.
scheduler-schedule-recurrent service is a daemon for choosing which
origins are to be visited according to scheduling policies and visit
statistics. It serves the same purpose as the
scheduler-runner from the
generic task scheduler, but it uses different data model and scheduling
Last, there is a
scheduler-journal-client service which listen to the Kafka
journal of the Storage to maintain the loading tasks status and statistics.
Once again, the purpose is roughly similar to the
the generic task scheduler, using Kafka instead of the Celery bus as feedback
Generic SWH Task Model
Each swh-task-type is the declaration of a type of swh-task. Each swh-task-type have the following fields:
type: Name of the swh-task type; can be anything but must be unique,
description: Human-readable task description
backend_name: Name of the task in the job-running backend,
default_interval: Default interval for newly scheduled tasks,
min_interval: Minimum interval between two runs of a task,
max_interval: Maximum interval between two runs of a task,
backoff_factor: Adjustment factor for the backoff between two task runs,
max_queue_length: Maximum length of the queue for this type of tasks,
num_retries: Default number of retries on transient failures,
retry_delay: Retry delay for the task,
Existing swh-task-types can be listed using the
swh scheduler command line
$ swh scheduler task-type list Known task types: check-deposit: Pre-checking deposit step before loading into swh archive index-fossology-license: Fossology license indexer task load-git: Update an origin of type git load-hg: Update an origin of type mercurial
You can see the details of a swh-task-type:
$ swh scheduler task-type list -v -t load-git Known task types: load-git: swh.loader.git.tasks.UpdateGitRepository Update an origin of type git interval: 64 days, 0:00:00 [12:00:00, 64 days, 0:00:00] backoff_factor: 2.0 max_queue_length: 5000 num_retries: None retry_delay: None
An swh-task is an ‘instance’ of such a swh-task-type, and consists in:
arguments: Arguments passed to the underlying job scheduler,
next_run: Next run of this task should be run on or after that time,
current_interval: Interval between two runs of this task, taking into
account the backoff factor,
policy: Whether the task is “one-shot” or “recurring”,
retries_left: Number of “short delay” retries of the task in case of
priority: Priority of the task,
id: Internal task identifier,
type: References task_type table,
status: Task status ( among “next_run_not_scheduled”, “next_run_scheduled”,
So a swh-task consist basically in:
a set of parameters defining how the scheduling of the swh-task is handled,
a set of parameters to specify the retry policy in case of transient failure upon execution,
a set of parameters that defines the job to be done (
You can list pending swh-tasks (tasks that are to be scheduled ASAP):
$ swh scheduler task list-pending load-git --limit 2 Found 1 load-git tasks Task 9052257 Next run: 15 days ago (2019-06-25 10:35:10+00:00) Interval: 2 days, 0:00:00 Type: load-git Policy: recurring Args: 'https://github.com/turtl/mobile' Keyword args:
Looking for existing swh-task can be done via the command line tool:
$ swh scheduler task list -t load-hg --limit 2 Found 2 tasks Task 168802702 Next run: in 4 hours (2019-07-10 17:56:48+00:00) Interval: 1 day, 0:00:00 Type: load-hg Policy: recurring Status: next_run_not_scheduled Priority: Args: 'https://bitbucket.org/kepung/pypy' Keyword args: Task 169800445 Next run: in a month (2019-08-10 17:54:24+00:00) Interval: 32 days, 0:00:00 Type: load-hg Policy: recurring Status: next_run_not_scheduled Priority: Args: 'https://bitbucket.org/lunixbochs/pypy-1' Keyword args:
Writing a new worker for a new swh-task-type
When you want to add a new swh-task-type, you need a celery worker backend capable of executing this new task-type instances.
Celery workers for swh-scheduler based tasks should be started using the Celery
swh.scheduler.celery_config. This later, among other things, provides
a loading mechanism for task types based on pkg_resources declared plugins under
[swh.workers] entry point.
TODO: add a fully working example of a dumb task.
- Command-line interface
- Software Heritage Scheduler Simulator
- swh.scheduler package
- swh.scheduler.api package
- swh.scheduler.celery_backend package
- swh.scheduler.cli package
- swh.scheduler.simulator package
- swh.scheduler.backend module
- swh.scheduler.cli_utils module
- swh.scheduler.exc module
- swh.scheduler.interface module
- swh.scheduler.journal_client module
- swh.scheduler.model module
- swh.scheduler.pytest_plugin module
- swh.scheduler.task module
- swh.scheduler.utils module