swh.lister.core.simple_lister module

class swh.lister.core.simple_lister.SimpleLister(override_config=None)[source]

Bases: swh.lister.core.lister_base.ListerBase

Lister* intermediate class for any service that follows the simple,

‘list in oneshot information’ pattern.

  • Client sends a request to list repositories in oneshot

  • Client receives structured (json/xml/etc) response with information and stores those in db

flush_packet_db = 2

Number of iterations in-between write flushes of lister repositories to db (see fn:ingest_data).

list_packages(response: Any) → List[Any][source]

Listing packages method.

ingest_data(identifier, checks=False)[source]
Rework the base ingest_data.

Request server endpoint which gives all in one go.

Simplify and filter response list of repositories. Inject repo information into local db. Queue loader tasks for linked repositories.

  • identifier – Resource identifier (unused)

  • checks (bool) – Additional checks required (unused)


Transform response to list for model manipulation

Query the server which answers in one query. Stores the

information, dropping actual redundant information we already have.