swh.lister.gitweb.lister module#

class swh.lister.gitweb.lister.GitwebLister(scheduler: SchedulerInterface, url: Optional[str] = None, instance: Optional[str] = None, credentials: Optional[Dict[str, Dict[str, List[Dict[str, str]]]]] = None, max_origins_per_page: Optional[int] = None, max_pages: Optional[int] = None, enable_origins: bool = True)[source]#

Bases: StatelessLister[List[Dict[str, Any]]]

Lister class for Gitweb repositories.

This lister will retrieve the list of published git repositories by parsing the HTML page(s) of the index retrieved at url.

Lister class for Gitweb repositories.

Parameters:

  • url – (Optional) Root URL of the Gitweb instance, i.e. url of the index of published git repositories on this instance. Defaults to https://instance if unset.

  • instance – Name of gitweb instance. Defaults to url’s network location if unset.

LISTER_NAME: str = 'gitweb'#
get_pages() → Iterator[List[Dict[str, Any]]][source]#

Generate git ‘project’ URLs found on the current Gitweb server.

get_origins_from_page(repositories: List[Dict[str, Any]]) → Iterator[ListedOrigin][source]#

Convert a page of gitweb repositories into a list of ListedOrigins.

swh.lister.gitweb.lister.try_to_determine_git_repository(repository_url: str) → Optional[str][source]#

Some gitweb instances does not advertise the git urls.

This heuristic works on instances demonstrating this behavior.

swh.lister.gitweb.lister.parse_last_update(last_update_interval: Optional[str]) → Optional[datetime][source]#

Parse the last update string into a datetime.