Software Heritage - Object storage replayer
This Python module provides a command line tool to replicate content objects from a
source Object storage to a destination one by listening the
content topic of a
Software Heritage - Journal kafka stream.
It is meant to be used as the brick of a mirror setup dedicated to replicating content objects.
Once installed (using pip or debian packages), the command
replay should be available:
It needs a configuration file with 4 sections:
objstorage: the source objstorage to retrieve objects from,
objstorage_dst: the destination objstorage to put objects into,
journal_client: the journal client (kafka configuration where the object hashes are consumed from),
replayer(optional): some replayer specific configurations options.
For example with a configuration file like:
objstorage: cls: multiplexer objstorages: - cls: http url: https://softwareheritage.s3.amazonaws.com/content/ compression: gzip - cls: remote url: https://login:firstname.lastname@example.org objstorage_dst: cls: remote args: url: http://objstorage:5003 journal_client: cls: kafka brokers: - broker1.journal.staging.swh.network:9093 group_id: kafka-username-content-replayer-003 sasl.username: kafka-username sasl.password: kafka-password security.protocol: sasl_ssl sasl.mechanism: SCRAM-SHA-512 session.timeout.ms: 600000 max.poll.interval.ms: 3600000 message.max.bytes: 1000000000 privileged: true batch_size: 2000 replayer: error_reporter: host: redis port: 6379 db: 0
you can start the content replayer with:
$ swh objstorage -C replayer-config.yml replay
You would typically run this tool on several machines, using the same
group_id, to increase replication parallelism.
Also note that you may increase the default concurrency within one replayer
--concurrency command line option. This will use as many
replication threads as given in argument, distributing the replication of
objects within the same kafka consumer among these threads. This is
typically useful when the replication of one object comes with non negligeable
minimal latency (e.g. consuming from public cloud-based objstorages).