swh.lister.rubygems package#


Module contents#

RubyGems lister#

The RubyGems lister list origins from RubyGems.org, the Ruby community’s gem hosting service.

As of September 2022 RubyGems.org list 173384 package names.

Origins retrieving strategy#

To get a list of all package names we call an http endpoint which returns a list of gems as text.

Page listing#

Each page returns an origin url based on the following pattern:


Origins from page#

The lister yields one origin url per page.

Running tests#

Activate the virtualenv and run from within swh-lister directory:

pytest -s -vv --log-cli-level=DEBUG swh/lister/rubygems/tests

Testing with Docker#

Change directory to swh/docker then launch the docker environment:

docker compose up -d

Then schedule a RubyGems listing task:

docker compose exec swh-scheduler swh scheduler task add -p oneshot list-rubygems

You can follow lister execution by displaying logs of swh-lister service:

docker compose logs -f swh-lister