swh.lister.debian.lister module

class swh.lister.debian.lister.DebianLister(distribution: str = 'Debian', date: Optional[datetime.datetime] = None, override_config: Mapping = {})[source]

Bases: swh.lister.core.lister_transports.ListerHttpTransport, swh.lister.core.lister_base.ListerBase

MODEL

alias of swh.lister.debian.models.Package

PATH_TEMPLATE = None
LISTER_NAME = 'debian'
instance = 'debian'
transport_request(identifier) → requests.models.Response[source]

Subvert ListerHttpTransport.transport_request, to try several index URIs in turn.

The Debian repository format supports several compression algorithms across the ages, so we try several URIs.

Once we have found a working URI, we break and set self.decompressor to the one that matched.

Returns

a requests Response object.

Raises

FetchError – when all the URIs failed to be retrieved.

request_uri(identifier)[source]

Get the full request URI given the transport_request identifier.

MAY BE OVERRIDDEN if something more complex than the PATH_TEMPLATE is required.

request_params(identifier) → Dict[str, Any][source]

Get the full parameters passed to requests given the transport_request identifier.

This uses credentials if any are provided (see request_instance_credentials).

MAY BE OVERRIDDEN if something more complex than the request headers is needed.

transport_response_simplified(response)[source]

Decompress and parse the package index fetched in transport_request.

For each package, we “pivot” the file list entries (Files, Checksums-Sha1, Checksums-Sha256), to return a files dict mapping filenames to their checksums.

inject_repo_data_into_db(models_list)[source]

Generate the Package entries that didn’t previously exist.

Contrary to ListerBase, we don’t actually insert the data in database. schedule_missing_tasks does it once we have the origin and task identifiers.

schedule_missing_tasks(models_list, added_packages)[source]

We create tasks at the end of the full snapshot processing

create_tasks_for_snapshot(snapshot)[source]
run()[source]

Run the lister for a given (distribution, area) tuple.