swh.lister.github.lister module

class swh.lister.github.lister.GitHubLister(url=None, override_config=None)[source]

Bases: swh.lister.core.indexing_lister.IndexingHttpLister

PATH_TEMPLATE = '/repositories?since=%d'
MODEL

alias of swh.lister.github.models.GitHubModel

DEFAULT_URL = 'https://api.github.com'
API_URL_INDEX_RE = re.compile('^.*/repositories\\?since=(\\d+)')
LISTER_NAME = 'github'
instance = 'github'
default_min_bound = 0
get_model_from_repo(repo)[source]
transport_quota_check(response)[source]
Implements ListerBase.transport_quota_check with standard 429

code check for HTTP with Requests library.

MAY BE OVERRIDDEN if the server notifies about rate limits in a

non-standard way that doesn’t use HTTP 429 and the Retry-After response header. ( https://tools.ietf.org/html/rfc6585#section-4 )

get_next_target_from_response(response)[source]

Find the next server endpoint identifier given the entire response.

Implementation of this method depends on the server API spec and the shape of the network response object returned by the transport_request method.

Parameters

response (transport response) – response page from the server

Returns

index of next page, possibly extracted from a next href url

transport_response_simplified(response)[source]
Convert the server response into list of a dict for each repo in the

response, mapping columns in the lister’s MODEL class to repo data.

Implementation of this method depends on the server API spec and the shape of the network response object returned by the transport_request method.

Parameters

response – response object from the server.

Returns

list of repo MODEL dicts

( eg. [{‘uid’: r[‘id’], etc.} for r in response.json()] )

request_headers()[source]

(Override) Set requests headers to send when querying the GitHub API

disable_deleted_repo_tasks(index, next_index, keep_these)[source]

(Overrides) Fix provided index value to avoid erroneously disabling some scheduler tasks