Software testing guide
The testing framework we use is pytest. It provides many facilities to write tests efficiently.
It is complemented by hypothesis, a library for property-based testing in some of our test suites. Its usage is a more advanced topic.
We also use tox, the automation framework, to run the tests along with other quality checks in isolated environments.
The main quality checking tools in use are:
mypy, a static type checker. We gradually type-annotate all additions or refactorings to the codebase;
flake8, a simple code style checker (aka linter);
black, an uncompromising code formatter.
They are run automatically through
tox or as
pre-commit hooks in our Git repositories.
The SWH testing framework
This sections shows specifics about our usage of pytest and custom helpers.
The pytest fixture system makes easy to write, share and plug setup and teardown code.
Fixtures are automatically loaded from the project
into any test function by giving its name as argument.
We make of various mocking helpers:
mockerfixture from the
mockplugin: adaptation of
unittest.mockto the fixture system, with a bonus
spyfunction to audit without modifying objects;
monkeypatchingbuiltin fixture: modify object attributes or environment, with automatic teardown.
Other notable helpers include:
datadir: to compute the path to the current test’s
datadirectory. Available in the
requests_mock_datadir: to load network responses from the datadir. Available in the
swh_rpc_client: for testing SWH RPC client and servers without incurring IO. Available in the
postgresql_fact: for testing database-backends interactions. Available in the
core.dbplugin, adapted for performance from the
click.testing.CliRunner: to simplify testing of Click command-line interfaces. It allows to test commands with some level of isolation from the execution environment. https://click.palletsprojects.com/en/7.x/api/#click.testing.CliRunner
We mostly do functional tests, and unit-testing when more granularity is needed. By this, we mean that we test each functionality and invariants of a component, without isolating it from its dependencies systematically. The goal is to strike a balance between test effectiveness and test maintenance. However, the most critical parts, like the storage service, get more extensive unit-testing.
In order to test a component (module, class), one must start by identifying its sets of functionalities and invariants (or properties).
One test may check multiples properties or commonly combined functionalities, if it can fit in a short descriptive name.
Organize tests in multiple modules, one for each aspect or subcomponent tested. e.g.: initialization/configuration, db/backend, service API, utils, CLI, etc.
Each repository has its own
tests directory, some such as listers even have one for
each lister type.
Put any non-trivial test data, used for setup or mocking, in (potentially compressed) files in a
datadirectory under the local testing directory.
datadirfixtures to load them.
Make use of temporary directories for testing code relying on filesystem paths.
Mock only already tested and expensive operations, typically IO with external services.
monkeypatchfixture when updating environment or when mocking is overkill.
Mock HTTP requests with
If testing is difficult, the tested design may need reconsideration.