swh.lister.core.lister_transports module

class swh.lister.core.lister_transports.ListerHttpTransport(url=None)[source]

Bases: abc.ABC

Use the Requests library for making Lister endpoint requests.

To be used in conjunction with ListerBase or a subclass of it.

DEFAULT_URL = None
PATH_TEMPLATE = <swh.lister.core.abstractattribute.AbstractAttribute object>
EXPECTED_STATUS_CODES = (200, 429, 403, 404)
request_headers() → Dict[str, Any][source]

Returns dictionary of any request headers needed by the server.

MAY BE OVERRIDDEN if request headers are needed.

request_instance_credentials() → List[Dict[str, Any]][source]

Returns dictionary of any credentials configuration needed by the forge instance to list.

The ‘credentials’ configuration is expected to be a dict of multiple levels. The first level is the lister’s name, the second is the lister’s instance name, which value is expected to be a list of credential structures (typically a couple username/password).

For example:

credentials:
  github:  # github lister
    github:  # has only one instance (so far)
      - username: some
        password: somekey
      - username: one
        password: onekey
      - ...
    gitlab:  # gitlab lister
      riseup:  # has many instances
        - username: someone
          password: ...
        - ...
      gitlab:
        - username: someone
          password: ...
        - ...
Returns

list of credential dicts for the current lister.

request_uri(identifier: str) → str[source]

Get the full request URI given the transport_request identifier.

MAY BE OVERRIDDEN if something more complex than the PATH_TEMPLATE is required.

request_params(identifier: str) → Dict[str, Any][source]

Get the full parameters passed to requests given the transport_request identifier.

This uses credentials if any are provided (see request_instance_credentials).

MAY BE OVERRIDDEN if something more complex than the request headers is needed.

transport_quota_check(response)[source]
Implements ListerBase.transport_quota_check with standard 429

code check for HTTP with Requests library.

MAY BE OVERRIDDEN if the server notifies about rate limits in a

non-standard way that doesn’t use HTTP 429 and the Retry-After response header. ( https://tools.ietf.org/html/rfc6585#section-4 )

transport_head(identifier: str) → requests.models.Response[source]

Retrieve head information on api.

transport_request(identifier: str) → requests.models.Response[source]

Implements ListerBase.transport_request for HTTP using Requests.

Retrieve get information on api.

transport_response_to_string(response: requests.models.Response) → str[source]

Implements ListerBase.transport_response_to_string for HTTP given Requests responses.

class swh.lister.core.lister_transports.ListerOnePageApiTransport(url=None)[source]

Bases: swh.lister.core.lister_transports.ListerHttpTransport

Leverage requests library to retrieve basic html page and parse result.

To be used in conjunction with ListerBase or a subclass of it.

PAGE = <swh.lister.core.abstractattribute.AbstractAttribute object>
PATH_TEMPLATE = None
request_uri(_)[source]

Get the full request URI given the transport_request identifier.