swh.scheduler.cli.origin_utils module#

Defines the <swh scheduler origin send-origins-from-file-to-celery> cli utility functions. This uses a list of origins read from the standard input or file, massage them into scheduler tasks to send directly to celery to a queue (according to a task type specified).

The list of origins has been extracted by other means (e.g. sentry extract, combination of various shell scripts, …). Then, a human operator provides the list to the cli so it’s consumed by standard swh queues (understand scheduler configured backend).

swh.scheduler.cli.origin_utils.get_scheduler_task_info(scheduler: SchedulerInterface, task_type: str) Dict[source]#

Retrieve information on task_type from the scheduler.

  • scheduler – Scheduler instance to lookup data from

  • task_type – The task type to lookup


ValueError when task_type and its fallback is not found.


Dict of information for the task type

swh.scheduler.cli.origin_utils.lines_to_task_args(lines: Iterable[str], columns: List[str] = ['url'], postprocess: Callable[[Dict[str, Any]], Dict[str, Any]] | None = None, **kwargs) Iterator[Dict[str, Any]][source]#

Iterate over the lines and convert them into celery tasks ready to be sent.

  • lines – Line read from a file or stdin

  • columns – structure of the lines to be read (usually only the url column)

  • postprocess – An optional callable to enrich the task with

  • **kwargs – extra static arguments to enrich the task with


task ready to be sent to celery