Data used to generate the website

The data used to build the website are all coming from a single source of truth, which is swh/dataportal/data/data.yml.

Each entry in this file is the name of the dataset it relates to. Additional fields can be added, such as statistics, specific title or comments.

As such, it is this file that is to be modified when adding a new dataset, see How to update datasets.

Specifications

dataset-unique-id (YYYY-MM-DD-{optional slug}-type):
  title: foo
  size: 2 TiB
  ...

The only required element is the ID of the dataset.

dataset-unique-id:

an identifier composed of the export date, an optional slug (for example popular-10-shell) an the dataset type (MUST BE one of orc, parquet, csv, compressed, contents, popular_contents, license, citation, provenance/all, provenance/heads, topology, path_counts, descendent_counts, vulnerabilities, digestmap), as defined in DatasetType

title:

overwrite the default dataset title (see :py:module:swh.dataportal.models)

description:

overwrite the default dataset description (see :py:module:swh.dataportal.models)

comments:

markdown comment about the dataset

teaser_of:

The ID of the dataset this dataset is a teaser of, if applicable.

deprecated:

True when the dataset is deprecated (the comments field should explain why).

derived_of:

The ID of the dataset this dataset is derived from, if applicable.

size:

Dataset size (unit included).