Webhooks Svix#

Intended audience

sysadm staff members

Scope of commands

For the following page, unless specified otherwise, the commands are executed from the sysadm computer.

Components#

swh-webhooks:
- role: journal client
- repository: swh-charts (swh)
- namespace: swh (production), swh-cassandra (staging)
svix-server:
- role: webhook server
- repository: swh-charts (cluster-components)
- namespace: svix-server
redis:
- role: cache, message queue
- repository: k8s-clusters-config (argocd-configuration/applications)
- namespace: svix-server
postgresql:
- role: database
- server: albertina:5434
- db name: swh-svix

Architecture#

webhooks_architecture

Generate a svix authentication token

ᐅ kubectl --context archive-production-rke2 exec -ti \
$(kubectl --context archive-production-rke2 get po -n svix-server -l app=svix-server -o name) \
-n svix-server -- svix-server jwt generate
Token (Bearer): XXX

It will be necessary to register swh-webhooks in svix-server.

Register default webhook event-types#

swh@webhooks-origin-visit-status:~$ swh webhooks event-type register-defaults
swh@webhooks-origin-visit-status:~$ swh webhooks event-type list

INFO:httpx:HTTP Request: GET https://svix.internal.softwareheritage.org/api/v1/event-type/?include_archived=false&with_content=true "HTTP/1.1 200 OK"
Registered event types:

origin.create:
  This event is triggered when a new software origin is added to the archive
origin.visit:
  This event is triggered when a new visit of a software origin was performed

Create webhook event-type endpoint#

Get the web-archive secret

ᐅ kubectl --context archive-production-rke2 exec -ti -n swh -c web-archive \
$(kubectl --context archive-production-rke2 get pods -n swh -l app=web-archive -o name | awk 'NR == 1') \
-- awk '/save_code_now_webhook_secret/{print $2}' /etc/swh/config.yml

whsec_XXX

Webhook secret must match the following regexp ^whsec_[a-zA-Z0-9+/=]{32,100}$.

swh@webhooks-origin-visit-status:~$ swh webhooks -C $SWH_CONFIG_FILENAME endpoint \
create origin.visit "$URL" --secret "${TOKEN}"

INFO:httpx:HTTP Request: GET https://svix.internal.softwareheritage.org/api/v1/event-type/origin.visit/ "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://svix.internal.softwareheritage.org/api/v1/app/21dacaea-7e38-5e52-80af-4d926f0c43d1/endpoint/ "HTTP/1.1 201 Created"
INFO:httpx:HTTP Request: PUT https://svix.internal.softwareheritage.org/api/v1/app/21dacaea-7e38-5e52-80af-4d926f0c43d1/endpoint/62d4603b-aec8-5ba9-8837-49be0f077bf3/headers/ "HTTP/1.1 204 No Content"

Delete webhook event-type endpoint#

From swh-webhooks:

swh@webhooks-origin-visit-status:~$ URL=$(swh webhooks endpoint list origin.visit \
2> /dev/null)
swh@webhooks-origin-visit-status:~$ swh webhooks endpoint delete origin.visit "$URL"

INFO:httpx:HTTP Request: GET https://svix.internal.staging.swh.network/api/v1/event-type/origin.visit/ "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: DELETE https://svix.internal.staging.swh.network/api/v1/app/21dacaea-7e38-5e52-80af-4d926f0c43d1/endpoint/c82f7b48-8216-54cd-8930-9dadced85f87/ "HTTP/1.1 204 No Content"

From toolbox:

swh@swh-toolbox:~$ APP_ID=$(./svix app list | jq -r '.data[] | \
select(.name=="origin.visit") | .id')
swh@swh-toolbox:~$ ENDPOINT_ID=$(./svix endpoint list "$APP_ID" | \
jq -r '.data[] | select(.filterTypes==["origin.visit"]) | .id')
swh@swh-toolbox-cbdf54845-5ckqt:~$ ./svix endpoint delete "$APP_ID" "$ENDPOINT_ID"

Are you sure you want to delete the the endpoint with id: ep_2fajH3V05o7VLMAxMnaUgyBlQfT? [y/N] y█
Endpoint "ep_2fajH3V05o7VLMAxMnaUgyBlQfT" Deleted!

Install svix-cli on toolbox#

swh@swh-toolbox:~$ wget -q https://github.com/svix/svix-cli/releases/download/v0.21.1/svix_Linux_x86_64.tar.gz
swh@swh-toolbox:~$ tar xvzf svix_Linux_x86_64.tar.gz svix
svix
swh@swh-toolbox:~$ chmod +x svix
swh@swh-toolbox:~$ awk '/auth_token|server_url/{print $2}' /etc/swh/config-webhooks.yml
<XXX1>
<XXX2>
swh@swh-toolbox:~$ ./svix login

Welcome to the Svix CLI, enter your auth token to get started!

✔ Svix Server URL: <XXX2>
✔ Svix Auth Token: <XXX1>

swh@swh-toolbox:~$ cat .config/svix/config.toml
auth_token = "XXX"
color = "auto"
server_url = "XXX"

Check webhooks applications#

`swh-webhooks`#

swh@webhooks-origin-visit-status:~$ swh webhooks -C "$SWH_CONFIG_FILENAME" event-type list

INFO:httpx:HTTP Request: GET https://svix.internal.softwareheritage.org/api/v1/event-type/?include_archived=false&with_content=true "HTTP/1.1 200 OK"
Registered event types:

origin.create:
  This event is triggered when a new software origin is added to the archive
origin.visit:
  This event is triggered when a new visit of a software origin was performed

swh@webhooks-origin-visit-status:~$ swh webhooks -C "$SWH_CONFIG_FILENAME" endpoint list origin.visit

INFO:httpx:HTTP Request: GET https://svix.internal.softwareheritage.org/api/v1/event-type/origin.visit/ "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://svix.internal.softwareheritage.org/api/v1/app/21dacaea-7e38-5e52-80af-4d926f0c43d1/endpoint/?order=descending "HTTP/1.1 200 OK"
https://archive.softwareheritage.org/save/origin/visit/webhook/

`svix-cli`#

swh@swh-toolbox:~$ APPLICATION=$(./svix app list | jq -r '.data[] | select (.name=="origin.visit") | .id')
swh@swh-toolbox:~$ echo "$APPLICATION"
app_2euZUZcVz4Hy5kyctOvLIl0Uy8J
swh@swh-toolbox:~$ ./svix endpoint list "$APPLICATION" | \
   jq -r '.data[] | select(.filterTypes==["origin.visit"])'
{
  "createdAt": "2024-04-23T13:37:43.389652Z",
  "description": "",
  "disabled": false,
  "filterTypes": [
    "origin.visit"
  ],
  "id": "ep_2fVHjwUWJ7TRJC2rnqoFO5cjuHm",
  "metadata": {},
  "rateLimit": null,
  "uid": "a0f2206d-2d21-580e-b1dc-4f4f1f8fd1af",
  "updatedAt": "2024-04-23T13:37:43.414656Z",
  "url": "https://archive.softwareheritage.org/save/origin/visit/webhook/",
  "version": 1
}

Check service logs#

`swh-webhooks`#

ᐅ kail --context archive-production-rke2 -n swh -c webhooks

`svix-server`#

ᐅ kail --context archive-production-rke2 -n svix-server -c svix-server

`postgresql`#

swh-svix=# select * from messageattempt where response_status_code = '403' limit 1;
-[ RECORD 1 ]--------+----------------------------------------------------------------
id                   | atmpt_2euhVJY9NbUimKxVPY4Z6Z16LNW
created_at           | 2024-04-10 14:46:32.39531+00
msg_id               | msg_2euhVJRF8bm1qRvNHkmgWbQADFf
msg_dest_id          | msgep_2euhVJVfsO7HB9sUD23W54f1KeA
endp_id              | ep_2euhUYHc78b3h5vapXbWCwzWBn9
url                  | https://archive.softwareheritage.org/save/origin/visit/webhook/
status               | 2
response_status_code | 403
response             | <html>\r                                                       +
                     | <head><title>403 Forbidden</title></head>\r                    +
                     | <body>\r                                                       +
                     | <center><h1>403 Forbidden</h1></center>\r                      +
                     | <hr><center>nginx</center>\r                                   +
                     | </body>\r                                                      +
                     | </html>\r                                                      +
                     |
ended_at             | 2024-04-10 14:46:32.545165+00
trigger_type         | 0

`swh-web`#

ᐅ kail --context archive-production-rke2 -n swh -c web-archive -l app=web-archive

Webhooks Svix#

Components#

Architecture#

Register default webhook event-types#

Create webhook event-type endpoint#

Delete webhook event-type endpoint#

Install svix-cli on toolbox#

Check webhooks applications#

swh-webhooks#

svix-cli#

Check service logs#

swh-webhooks#

svix-server#

postgresql#

swh-web#

`swh-webhooks`#

`svix-cli`#

`swh-webhooks`#

`svix-server`#

`postgresql`#

`swh-web`#