Software Heritage - Object storage replayer

This Python module provides a command line tool to replicate content objects from a source Object storage to a destination one by listening the content topic of a Software Heritage - Journal kafka stream.

It is meant to be used as the brick of a mirror setup dedicated to replicating content objects.

Quick start

Once installed (using pip or debian packages), the command swh objstorage replay should be available:

It needs a configuration file with 4 sections:

  • objstorage: the source objstorage to retrieve objects from,

  • objstorage_dst: the destination objstorage to put objects into,

  • journal_client: the journal client (kafka configuration where the object hashes are consumed from),

  • replayer (optional): some replayer specific configurations options.

For example with a configuration file like:

  cls: multiplexer
    - cls: http
      compression: gzip
    - cls: remote

  cls: remote
    url: http://objstorage:5003

  cls: kafka
  group_id: kafka-username-content-replayer-003
  sasl.username: kafka-username
  sasl.password: kafka-password
  security.protocol: sasl_ssl
  sasl.mechanism: SCRAM-SHA-512 600000 3600000
  message.max.bytes: 1000000000
  privileged: true
  batch_size: 2000

    host: redis
    port: 6379
    db: 0

you can start the content replayer with:

$ swh objstorage -C replayer-config.yml replay

You would typically run this tool on several machines, using the same group_id, to increase replication parallelism.

Also note that you may increase the default concurrency within one replayer using the --concurrency command line option. This will use as many replication threads as given in argument, distributing the replication of objects within the same kafka consumer among these threads. This is typically useful when the replication of one object comes with non negligeable minimal latency (e.g. consuming from public cloud-based objstorages).

Reference Documentation