Benchmarks
==========

The tests (tox) run the benchmark code with very small files. The benchmarks is about running the same
code with Read Shards that have the expected size in production (100GB) and verify:

* The time required to build a Read Shard is less than 5 times the
  time to copy a file of the same size.  It guards against regressions
  that would have a significant performance impact.

* The time required to build and write the perfect hash table is
  always a fraction of the time required to copy the content of the
  objects.

* It is possible to perform at least 10,000 object lookups per second.


Build performances
------------------

In both cases the distribution of the object sizes is uniform.

With a small number of large (<100MB) objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The goal is to verify that manipulating up to the largest supported
object size actually works and does not trigger problems in the
extreme case that all objects have the maximum size.

* time tox run -e py3 -- --basetemp=/mnt/pytest -s --shard-path /mnt/payload --shard-size $((100 * 1024)) --object-max-size $((100 * 1024 * 1024)) -k test_build_speed
  number of objects = 2057, total size = 107374116576
  baseline 165.85, write_duration 327.19, build_duration 0.0062, total_duration 327.19

With a large number of small (<4KB) objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

It approximates the maximum number of objects a Read Shard can
contain. The goal is to verify that creating a perfect hash with this
many items does not require more than a fraction of the time required
to copy the objects.

* time tox run -e py3 -- --basetemp=/mnt/pytest -s --shard-path /mnt/payload --shard-size $((100 * 1024)) --object-max-size $((4 * 1024)) -k test_build_speed
  number of objects = 45973694, total size = 105903024192
  baseline 165.74, write_duration 495.07, build_duration 24.21, total_duration 519.28


Object lookup performances
--------------------------

The machine has 200GB of RAM and can therefore approximately cache the
content of two Read Shards which can have a significant impact on
performances. To minimize that effect, four Read Shard are created
totaling 400GB. All objects are looked up in all shards to verify
the lookup speed is greater than 5,000 objects per second.

* time tox run -e py3 -- --basetemp=/mnt/pytest -s -k test_lookup_speed --lookups $((100 * 1024 * 1024)) --shard-size $((100 * 1024)) --object-max-size $((4 * 1024)) swh/perfecthash/tests/test_hash.py  --shard-path /mnt/payload --shard-count 4
  number of objects = 45974390, total size = 105903001920
  key lookups speed = 9769.68/s


Setup via Fed4Fire
------------------

* https://www.fed4fire.eu/
* /opt/jFed/jFed-Experimenter
* Create an experiment with one machine
* Click on Topology Viewer and run the experiment (name test)
* Once the provisionning is complete (Testing connectivity to resources on Grid5000) click Export As
* Choose Export Configuration Management Settings
* Save under test.zip and unzip
* ssh -i id_rsa -F ssh-config node0
* alias sudo=sudo-g5k

Setup via Grid5000
------------------

* https://www.grid5000.fr/
* oarsub -I -l "{cluster='dahu'}/host=1,walltime=1" -t deploy
* kadeploy3 -f $OAR_NODE_FILE -e debian11-x64-base -k
* ssh root@$(tail -1 $OAR_NODE_FILE)

Common setup
------------

* mkfs.ext4 /dev/sdc
* mount /dev/sdc /mnt
* apt-get install -y python3-venv python3-dev libcmph-dev gcc git emacs-nox
* git clone -b wip-benchmark https://git.easter-eggs.org/biceps/swh-perfecthash/
* python3 -m venv bench
* source bench/bin/activate
* cd swh-perfecthash
* pip install -r requirements.txt -r requirements-test.txt tox wheel
* tox run -e py3