How to onboard a mirror#
Intended audience
sysadm staff members
A mirror needs credentials to access our journal and to retrieve the contents.
They are manually created by a software Heritage System Administrator. Different credentials must be provided for staging and production.
The URLs to communicate to the mirror operator are defined in Service urls
in the ‘Public URLs’ sections, Journal TLS
entries and swh-objstorage read-only
.
User naming rules#
The convention in place for the mirroring credentials are the following:
The names are composed of three parts: <short>-<environment>-<id>
A short meaningful name relative to the mirror company/owner. For example
acme
. For a Software Heritage internal account, the name must beswh-<account name>
. For example:swh-jdoe
The environment:
stg
for staging,prod
for productionA counter if several accounts are needed for the same environment
Examples of correct names:
acme-stg-01
acme-stg-02
acme-prod-01
swh-jdoe-stg-01
swh-jdoe-prod-01
How to create the journal credentials#
Connect on the journal orchestrator server (getty)
A script per existing kafka cluster is created by puppet in the /usr/local/sbin
directory.
The file name follow this pattern: create_kafka_user_<cluster name>.sh
The different clusters are:
rocquencourt
: production journalrocquencourt_staging
: staging journal
For example, to add a user new-mirror-stg
in the staging’s journal:
ssh getty.internal.softwareheritage.org
sudo /usr/local/sbin/create_kafka_user_rocquencourt_staging.sh new-mirror-stg
root@getty:/usr/local/sbin# ./create_kafka_users_rocquencourt_staging.sh mirror-test
Creating user mirror-test-stg, with unprivileged access to consumer group prefix mirror-test-stg-
Password for user mirror-test-stg: <---- Enter the user password here
Setting user credentials
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Completed updating config for entity: user-principal 'mirror-test-stg'.
Granting access to topics swh.journal.objects. to mirror-test-stg
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`:
(principal=User:mirror-test-stg, host=*, operation=READ, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`:
(principal=User:mirror-test, host=*, operation=READ, permissionType=ALLOW)
(principal=User:mirror-test, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:mirror-test-stg, host=*, operation=READ, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`:
(principal=User:mirror-test-stg, host=*, operation=DESCRIBE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`:
(principal=User:mirror-test, host=*, operation=READ, permissionType=ALLOW)
(principal=User:mirror-test, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:mirror-test-stg, host=*, operation=READ, permissionType=ALLOW)
(principal=User:mirror-test-stg, host=*, operation=DESCRIBE, permissionType=ALLOW)
Granting access to topics swh.journal.indexed. to mirror-test-stg
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.indexed., patternType=PREFIXED)`:
(principal=User:mirror-test-stg, host=*, operation=READ, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.indexed., patternType=PREFIXED)`:
(principal=User:mirror-test, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:mirror-test, host=*, operation=READ, permissionType=ALLOW)
(principal=User:mirror-test-stg, host=*, operation=READ, permissionType=ALLOW)
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.indexed., patternType=PREFIXED)`:
(principal=User:mirror-test-stg, host=*, operation=DESCRIBE, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.indexed., patternType=PREFIXED)`:
(principal=User:mirror-test, host=*, operation=DESCRIBE, permissionType=ALLOW)
(principal=User:mirror-test, host=*, operation=READ, permissionType=ALLOW)
(principal=User:mirror-test-stg, host=*, operation=READ, permissionType=ALLOW)
(principal=User:mirror-test-stg, host=*, operation=DESCRIBE, permissionType=ALLOW)
Granting access to consumer group prefix mirror-test-stg- to mirror-test-stg
Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=mirror-test-stg-, patternType=PREFIXED)`:
(principal=User:mirror-test-stg, host=*, operation=READ, permissionType=ALLOW)
Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=mirror-test-stg-, patternType=PREFIXED)`:
(principal=User:mirror-test-stg, host=*, operation=READ, permissionType=ALLOW)
How to use the credentials#
Refer to the journal client authentication configuration.
How to create the objstorage credentials#
The read-only public storages are protected by an basic authentication mechanism. To allow a mirror to retrieve the content files, they need to have valid credentials.
These credentials are managed and deployed by puppet.
To add a credential in the puppet configuration:
for staging:
locate the
swh::deploy::objstorage::reverse_proxy::basic_auth::users
property in the data/deployment/staging/common.yaml fileadd the username in the list
- for production
locate the
swh::deploy::objstorage::reverse_proxy::basic_auth::users
property in the data/common/common.yaml fileadd the username in the list
Add an entry
swh::deploy::objstorage::reverse_proxy::basic_auth::<<username>>
in theprivate/swh-private-data/common.yaml
in the
private
directory of your puppet sources, execute the following command to refresh the censored credentials (used by octocatalog-diff and vagrant):
private_data/generate-public-data swh-private-data swh-private-data-censored
Deploy the changes to the puppet master