swh.lister.core.page_by_page_lister module

class swh.lister.core.page_by_page_lister.PageByPageLister(override_config=None)[source]

Bases: swh.lister.core.lister_base.ListerBase

Lister* intermediate class for any service that follows the simple

pagination page pattern.

  • Client sends a request to list repositories starting from a given page identifier.

  • Client receives structured (json/xml/etc) response with information about a sequential series of repositories (per page) starting from a given index. And, if available, some indication of the next page index for fetching the remaining repository data.

See swh.lister.core.lister_base.ListerBase for more details.

This class cannot be instantiated. To create a new Lister for a source code listing service that follows the model described above, you must subclass this class. Then provide the required overrides in addition to any unmet implementation/override requirements of this class’s base (see parent class and member docstrings for details).

Required Overrides:

def get_next_target_from_response
abstract get_next_target_from_response(response)[source]

Find the next server endpoint page given the entire response.

Implementation of this method depends on the server API spec and the shape of the network response object returned by the transport_request method.

For example, some api can use the headers links to provide the next page.

Parameters

response (transport response) – response page from the server

Returns

index of next page, possibly extracted from a next href url

abstract get_pages_information()[source]

Find the total number of pages.

Implementation of this method depends on the server API spec and the shape of the network response object returned by the transport_request method.

For example, some api can use dedicated headers: - x-total-pages to provide the total number of pages - x-total to provide the total number of repositories - x-per-page to provide the number of elements per page

Returns

tuple (total number of repositories, total number of pages, per_page)

do_additional_checks(models_list)[source]

Potentially check for existence of repositories in models_list.

This will be called only if check_existence is flipped on in the run method below.

run(min_bound=None, max_bound=None, check_existence=False)[source]
Main entry function. Sequentially fetches repository data from the

service according to the basic outline in the class docstring. Continually fetching sublists until either there is no next page reference given or the given next page is greater than the desired max_page.

Parameters
  • min_bound – optional page to start from

  • max_bound – optional page to stop at

  • check_existence (bool) – optional existence check (for incremental lister whose sort order is inverted)

Returns

nothing

class swh.lister.core.page_by_page_lister.PageByPageHttpLister(url=None, override_config=None)[source]

Bases: swh.lister.core.lister_transports.ListerHttpTransport, swh.lister.core.page_by_page_lister.PageByPageLister

Convenience class for ensuring right lookup and init order when combining PageByPageLister and ListerHttpTransport.