Developer setup

In this guide we describe how to set up a developer environment in which one can easily navigate the source code, make modifications, write and execute unit tests.

For this, we will use a virtualenv in which all the Software Heritage packages will be installed in ‘develop’ mode, this will allow you to navigate the source code, hack it, and run locally the unit tests.

If you want to test the effect of your modifications in a running Software Heritage instance, you should check the documentation of the swh-docker-dev project.

Install required dependencies

Software Heritage requires some dependencies that are usually packaged by your package manager. On Debian/Ubuntu-based distributions:

sudo wget https://www.postgresql.org/media/keys/ACCC4CF8.asc -O /etc/apt/trusted.gpg.d/postgresql.asc
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
sudo apt update
sudo apt install python3 python3-venv libsvn-dev postgresql-11 \
                 libsystemd-dev libpython3-dev graphviz postgresql-autodoc \
                 postgresql-server-dev-all virtualenvwrapper git build-essential

Checkout the source code

Clone the Software Heritage environment repository:

~$ git clone https://forge.softwareheritage.org/source/swh-environment.git
[...]
~$ cd swh-environment
~/swh-environment$

Checkout all the swh packages source repositories:

~/swh-environment$ ./bin/update

Create a virtualenv:

~/swh-environment$ mkvirtualenv -p /usr/bin/python3 -a $PWD swh
[...]
(swh) ~/swh-environment$

Note

using virtualenvwrapper is not mandatory here. You can use plain virtualenvs, or any other venv management tool (pipenv or poetry for example). Using a tool such as virtualenvwrapper just makes life easier…

Install all the swh packages (in develop mode):

(swh) ~/swh-environment$ pip install $(./bin/pip-swh-packages --with-testing) \
                             tox pifpaf flake8
[...]

Executing unit tests

Unit tests are using the pytest framework, and can be executed directly or via tox. The main difference between these 2 test execution environments is:

  • When executed via tox, all the dependencies (including swh ones) are installed from pypi: you test yout modifications against the latest published version of every swh package but the current one.
  • When you execute pytest directly, swh dependencies are used from your current virtualenv, installed from the git repositories: you test your modification against the HEAD of every swh package.

For example, running unit tests for the swh-loader-git package:

(swh) ~/swh-environment$ cd swh-loader-git
(swh) ~/swh-environment/swh-loader-git$ pytest
    =========================== test session starts ============================
platform linux -- Python 3.5.3, pytest-3.8.2, py-1.6.0, pluggy-0.7.1
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/ddouard/src/swh-environment/swh-loader-git/.hypothesis/examples')
rootdir: /home/ddouard/src/swh-environment/swh-loader-git, inifile: pytest.ini
plugins: requests-mock-1.5.2, postgresql-1.3.4, env-0.6.2, django-3.4.7, cov-2.6.0, pylama-7.6.5, hypothesis-3.76.0, celery-4.2.1
collected 25 items

swh/loader/git/tests/test_converters.py ........                     [ 32%]
swh/loader/git/tests/test_from_disk.py .....                         [ 52%]
swh/loader/git/tests/test_loader.py ......                           [ 76%]
swh/loader/git/tests/test_tasks.py ...                               [ 88%]
swh/loader/git/tests/test_utils.py ...                               [100%]
============================= warnings summary =============================
    [...]
    ================== 25 passed, 12 warnings in 6.66 seconds ==================

Running the same test, plus flake8 checks, using tox:

(swh) ~/swh-environment/swh-loader-git$ tox
GLOB sdist-make: ~/swh-environment/swh-loader-git/setup.py
flake8 create: ~/swh-environment/swh-loader-git/.tox/flake8
flake8 installdeps: flake8
flake8 installed: entrypoints==0.3,flake8==3.7.7,mccabe==0.6.1,pycodestyle==2.5.0,pyflakes==2.1.1,swh.loader.git==0.0.48.post3
flake8 run-test-pre: PYTHONHASHSEED='2028963506'
flake8 runtests: commands[0] | ~/swh-environment/swh-loader-git/.tox/flake8/bin/python -m flake8
py3 create: ~/swh-environment/swh-loader-git/.tox/py3
py3 installdeps: .[testing], pytest-cov
py3 inst: ~/swh-environment/swh-loader-git/.tox/.tmp/package/1/swh.loader.git-0.0.48.post3.zip
py3 installed: aiohttp==3.5.4,amqp==2.4.2,arrow==0.13.1,async-timeout==3.0.1,atomicwrites==1.3.0,attrs==19.1.0,billiard==3.5.0.5,celery==4.2.1,certifi==2018.11.29,chardet==3.0.4,Click==7.0,coverage==4.5.2,decorator==4.3.2,dulwich==0.19.11,elasticsearch==6.3.1,Flask==1.0.2,idna==2.8,idna-ssl==1.1.0,itsdangerous==1.1.0,Jinja2==2.10,kombu==4.4.0,MarkupSafe==1.1.1,more-itertools==6.0.0,msgpack-python==0.5.6,multidict==4.5.2,pathlib2==2.3.3,pluggy==0.9.0,psutil==5.6.0,psycopg2==2.7.7,py==1.8.0,pytest==3.10.1,pytest-cov==2.6.1,python-dateutil==2.8.0,pytz==2018.9,PyYAML==3.13,requests==2.21.0,retrying==1.3.3,six==1.12.0,swh.core==0.0.55,swh.loader.core==0.0.39,swh.loader.git==0.0.48.post3,swh.model==0.0.30,swh.objstorage==0.0.30,swh.scheduler==0.0.49,swh.storage==0.0.129,systemd-python==234,typing-extensions==3.7.2,urllib3==1.24.1,vcversioner==2.16.0.0,vine==1.2.0,Werkzeug==0.14.1,yarl==1.3.0
py3 run-test-pre: PYTHONHASHSEED='2028963506'
py3 runtests: commands[0] | pytest --cov=swh --cov-branch
=========================== test session starts ============================
platform linux -- Python 3.5.3, pytest-3.10.1, py-1.8.0, pluggy-0.9.0
rootdir: ~/swh-environment/swh-loader-git, inifile: pytest.ini
plugins: cov-2.6.1, celery-4.2.1
collected 25 items

swh/loader/git/tests/test_converters.py ........                     [ 32%]
swh/loader/git/tests/test_from_disk.py .....                         [ 52%]
swh/loader/git/tests/test_loader.py ......                           [ 76%]
swh/loader/git/tests/test_tasks.py ...                               [ 88%]
swh/loader/git/tests/test_utils.py ...                               [100%]

----------- coverage: platform linux, python 3.5.3-final-0 -----------
Name                                      Stmts   Miss Branch BrPart  Cover
---------------------------------------------------------------------------
swh/__init__.py                               1      0      0      0   100%
swh/loader/__init__.py                        1      0      0      0   100%
swh/loader/git/__init__.py                    0      0      0      0   100%
swh/loader/git/converters.py                102     10     44      7    86%
swh/loader/git/from_disk.py                 157     44     50      6    67%
swh/loader/git/loader.py                    271     59    114     17    75%
swh/loader/git/tasks.py                      14      0      0      0   100%
swh/loader/git/tests/__init__.py              1      0      0      0   100%
swh/loader/git/tests/conftest.py              4      0      0      0   100%
swh/loader/git/tests/test_converters.py      94      0      6      0   100%
swh/loader/git/tests/test_from_disk.py      100      4      0      0    96%
swh/loader/git/tests/test_loader.py          12      0      0      0   100%
swh/loader/git/tests/test_tasks.py           26      0      0      0   100%
swh/loader/git/tests/test_utils.py           14      0      2      0   100%
swh/loader/git/utils.py                      25      8      8      1    61%
---------------------------------------------------------------------------
TOTAL                                       822    125    224     31    80%


============================= warnings summary =============================
.tox/py3/lib/python3.5/site-packages/psycopg2/__init__.py:144
  ~/swh-environment/swh-loader-git/.tox/py3/lib/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
    """)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================== 25 passed, 1 warnings in 7.34 seconds ===================
_________________________________ summary __________________________________
  flake8: commands succeeded
  py3: commands succeeded
  congratulations :)

Beware that some swh packages require a postgresql server properly configured to execute the tests. In this case, you will want to use pifpaf, which will spawn a temporary instance of postgresql, to encapsulate the call to pytest. For example, running pytest in the swh-core package:

(swh) ~/swh-environment$ cd swh-core
    (swh) ~/swh-environment/swh-core$ pifpaf run postgresql -- pytest
=========================== test session starts ============================
platform linux -- Python 3.5.3, pytest-3.8.2, py-1.6.0, pluggy-0.7.1
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/ddouard/src/swh-environment/swh-core/.hypothesis/examples')
rootdir: /home/ddouard/src/swh-environment/swh-core, inifile: pytest.ini
plugins: requests-mock-1.5.2, postgresql-1.3.4, env-0.6.2, django-3.4.7, cov-2.6.0, pylama-7.6.5, hypothesis-3.76.0, celery-4.2.1
collected 79 items

swh/core/tests/test_api.py ..                                        [  2%]
swh/core/tests/test_config.py ..............                         [ 20%]
swh/core/tests/test_db.py ....                                       [ 25%]
swh/core/tests/test_logger.py .                                      [ 26%]
swh/core/tests/test_serializers.py .....                             [ 32%]
swh/core/tests/test_statsd.py ...................................... [ 81%]
........                                                             [ 91%]
swh/core/tests/test_utils.py .......                                 [100%]

======================== 79 passed in 6.59 seconds =========================