Command-line interface

swh scrubber

main command group of the datastore scrubber

Expected config format:

scrubber:
    cls: postgresql
    db: "service=..."    # libpq DSN

# for storage checkers + origin locator only:
storage:
    cls: postgresql     # cannot be remote for checkers, as they need direct
                        # access to the pg DB
    db": "service=..."  # libpq DSN
    objstorage:
        cls: memory

# for journal checkers only:
journal:
    # see https://docs.softwareheritage.org/devel/apidoc/swh.journal.client.html
    # for the full list of options
    sasl.mechanism: SCRAM-SHA-512
    security.protocol: SASL_SSL
    sasl.username: ...
    sasl.password: ...
    group_id: ...
    privileged: True
    message.max.bytes: 524288000
    brokers:
      - "broker1.journal.softwareheritage.org:9093
      - "broker2.journal.softwareheritage.org:9093
      - "broker3.journal.softwareheritage.org:9093
      - "broker4.journal.softwareheritage.org:9093
      - "broker5.journal.softwareheritage.org:9093
    object_types: [directory, revision, snapshot, release]
    auto_offset_reset: earliest

swh scrubber [OPTIONS] COMMAND [ARGS]...

Options

-C, --config-file <config_file>

Configuration file.

check

group of commands which read from data stores and report errors.

swh scrubber check [OPTIONS] COMMAND [ARGS]...

init

Initialise a scrubber check configuration for the datastore defined in the configuration file and given object_type.

A checker configuration configuration consists simply in a set of:

• backend: the datastore type being scrubbed (storage, objstorage or journal),

• object-type: the type of object being checked,

• nb-partitions: the number of partitions the hash space is divided in; must be a power of 2,

• name: an unique name for easier reference,

• check-hashes: flag (default to True) to select the hash validation step for this scrubbing configuration,

• check-references: flag (default to True for storage and False for the journal backend) to select the reference validation step for this scrubbing configuration.

swh scrubber check init [OPTIONS] {storage|journal|objstorage}

Options

--object-type <object_type>

Options:

snapshot | revision | release | directory | content

--nb-partitions <nb_partitions>

--name <name>

--check-hashes, --no-check-hashes

--check-references, --no-check-references

Arguments

BACKEND

Required argument

list

List the know configurations

swh scrubber check list [OPTIONS]

run

Run the scrubber checker configured as name and reports corrupt objects to the scrubber DB.

This runs a single thread; parallelism is achieved by running this command multiple times.

This command references an existing scrubbing configuration (either by name or by id); the configuration holds the object type, number of partitions and the storage configuration this scrubbing session will check on.

swh scrubber check run [OPTIONS] [NAME]

Options

--config-id <config_id>

Config ID (is config name is not given as argument)

--use-journal

Flag only relevant for running an object storage scrubber, if set content ids are consumed from a kafka topic of SWH journal instead of getting them from a storage

--limit <limit>

Arguments

NAME

Optional argument

running

List partitions being checked for the check session <name>

swh scrubber check running [OPTIONS] [NAME]

Options

--config-id <config_id>

Arguments

NAME

Optional argument

stalled

List the stuck partitions for a given config

swh scrubber check stalled [OPTIONS] [NAME]

Options

--config-id <config_id>

--for <delay>

Delay for a partition to be considered as stuck; in seconds or ‘auto’

--reset

Reset the stalled partition so it can be grabbed by a scrubber worker

Arguments

NAME

Optional argument

stats

Display statistics for the check session <name>

swh scrubber check stats [OPTIONS] [NAME]

Options

--config-id <config_id>

-j, --json

Arguments

NAME

Optional argument

fix

For each known corrupt object reported in the scrubber DB, looks up origins that may contain this object, and records them; so they can be used later for recovery.

swh scrubber fix [OPTIONS]

Options

--start-object <start_object>

--end-object <end_object>

locate

For each known corrupt object reported in the scrubber DB, looks up origins that may contain this object, and records them; so they can be used later for recovery.

swh scrubber locate [OPTIONS]

Options

--start-object <start_object>

--end-object <end_object>