Read Shard format
=================

The Read Shard has the following structure:

* bytes \[0, ``SHARD_OFFSET_MAGIC``\[: The shard magic
* bytes \[``SHARD_OFFSET_MAGIC``, ``objects_position``\[: The header (``shard_header_t``)
* bytes \[``objects_position``, ``index_position``\[: ``objects_count`` times the size of the object (``u_int64_t``) followed by the content of the object
* bytes \[``index_position``, ``hash_position``\[: An array of index entries. The size of the array is provided by ``cmph_size`` after building the hash function. An index entry is made of the key (of ``SHARD_KEY_LEN`` bytes) and the object position (``u_int64_t``) in the range \[``objects_position``, ``index_position``\[. If the object position is ``UINT64_MAX``, this means the object has been deleted.
* bytes \[``hash_position``, ...\[: The hash function, as written by ``cmph_dump``

In more details:

+--------------------------+------+----------------------------+
| Section                  | pos  | description (length)       |
+==========================+======+============================+
| **SHARD_MAGIC**          | 0    | SHARD_OFFSET_MAGIC (32)    |
+--------------------------+------+----------------------------+
| **header**               | 32   | Header (56)                |
+--------------------------+------+----------------------------+
| ``version``              |      | uint64_t (8)               |
+--------------------------+------+----------------------------+
| ``objects_count``        |      | uint64_t (8)               |
+--------------------------+------+----------------------------+
| ``objects_position`` <op>|      | uint64_t (8)               |
+--------------------------+------+----------------------------+
| ``objects_size``         |      | uint64_t (8)               |
+--------------------------+------+----------------------------+
| ``index_position`` <ip>  |      | uint64_t (8)               |
+--------------------------+------+----------------------------+
| ``index_size``           |      | uint64_t (8)               |
+--------------------------+------+----------------------------+
| ``hash_position`` <hp>   |      | uint64_t (8)               |
+--------------------------+------+----------------------------+
| **Objects**              | <op> |                            |
+--------------------------+------+----------------------------+
| ``object0 size``         |      | uint64_t (8)               |
+--------------------------+------+----------------------------+
| ``object0 data``         |      | bytes (<object0 size>)     |
+--------------------------+------+----------------------------+
| ``object1 size``         |      | uint64_t (8)               |
+--------------------------+------+----------------------------+
| ``object1 data``         |      | bytes (<object1 size>      |
+--------------------------+------+----------------------------+
|   ...                    |      |                            |
+--------------------------+------+----------------------------+
| **Index**                | <ip> |                            |
+--------------------------+------+----------------------------+
| ``object0 key``          |      | SHARD_KEY_LEN (32)         |
+--------------------------+------+----------------------------+
| ``object0 offset``       |      | uint64_t (8)               |
+--------------------------+------+----------------------------+
|   ...                    |      |                            |
+--------------------------+------+----------------------------+
| **Hash map**             | <hp> |                            |
+--------------------------+------+----------------------------+
| ``hash function``        |      | <as written by cmph_dump>  |
+--------------------------+------+----------------------------+


``SHARD_MAGIC`` is the constant ``SWHShard`` (with ``\x00`` padding to 32
characters).

Index entries for deleted content are using the special value
``{key=\x00...\x00, offset=2**64-1}``.