Upgrade Procedure for Debian Nodes in an Elasticsearch Cluster#
Intended audience
sysadm staff members
Purpose#
This page documents the steps to upgrade Debian nodes running in an Elasticsearch cluster. The upgrade process involves various commands and checks before and after rebooting the node.
Prerequisites#
Familiarity with SSH and CLI-based command execution
Out-of-band Access to the node (IDRAC/ILO) for reboot
Access to the node through SSH (requires the vpn)
Step 0: Initial Steps#
The elasticsearch nodes are only running on bare metal machines. So, ensuring the out of band access to the machine is ok. This definitely helps when something goes wrong during a reboot (disk order or names change, network, …).
Step 1: Migrate to the next debian suite#
Update the Debian version of the node (e.g. bullseye to bookworm) using the following command:
root@node:~# /usr/local/bin/migrate-to-${NEXT_CODENAME}.sh
Note: The script should be present on the machine (installed through puppet).
Step 2: Run Puppet Agent#
Once the upgrade procedure happened, run the puppet agent to apply any necessary configuration changes (e.g. /etc/apt/sources.list change, etc…)
root@node:~# puppet agent -t
Step 3: Stop Puppet Agent#
As we will stop the service, we don’t want the agent to start it back again.
root@node:~# puppet agent --disable "Ongoing debian upgrade"
Step 4: Autoremove and Purge#
Perform autoremove to remove unnecessary packages left-over from the migration.
root@node:~# apt autoremove
Step 5: Stop the elasticsearch service#
The cluster can support one non-responding node so it’s ok to stop the service.
We can check the cluster’s status which should stay green after the elasticsearch service is stopped.
root@node:~# systemctl stop elasticsearch
root@node:~# curl -s $server/_cluster/health | jq .status
"green"
Note: $server if of the form hostname:9200 (with hostname another cluster node
than the one we are currently upgrading)
Step 6: Reboot the node#
We are ready to reboot the node:
root@node:~# reboot
You can connect to the serial console of the machine to follow through the reboot.
Step 7: Clean up some more#
Once the machine is restarted, some cleanup might be necessary.
root@node:~# apt autopurge
Step 8: Activate puppet agent#
Activate back the puppet agent and make it run. This will start back the elasticsearch service again.
root@node:~# puppet agent --enable && puppet agent --test
Step 8: Join back the cluster#
After the service restarted, check the node joined back the cluster.
root@node:~# curl -s $server/_cat/allocation?v\&s=node;
root@node:~# curl -s $server/_cluster/health | jq .number_of_nodes
For example:
root@esnode1:~# server=http://esnode1.internal.softwareheritage.org:9200; date; \
curl -s $server/_cat/allocation?v\&s=node; echo; \
curl -s $server/_cluster/health | jq
Wed Jan 29 09:57:01 UTC 2025
shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent host ip node node.role
638 0 0.0 5.6tb 5.6tb 5.6tb 1.1tb 6.8tb 82 192.168.100.61 192.168.100.61 esnode1 cdfhilmrstw
634 0 0.0 5.7tb 5.7tb 5.7tb 1tb 6.8tb 84 192.168.100.62 192.168.100.62 esnode2 cdfhilmrstw
639 0 0.0 5.6tb 5.6tb 5.6tb 1.1tb 6.8tb 82 192.168.100.63 192.168.100.63 esnode3 cdfhilmrstw
644 0 0.0 5.6tb 5.6tb 5.6tb 8.2tb 13.8tb 40 192.168.100.64 192.168.100.64 esnode7 cdfhilmrstw
645 0 0.0 5.5tb 5.5tb 5.5tb 5.9tb 11.4tb 48 192.168.100.65 192.168.100.65 esnode8 cdfhilmrstw
666 0 0.0 5.1tb 5.1tb 5.1tb 6.3tb 11.4tb 44 192.168.100.66 192.168.100.66 esnode9 cdfhilmrstw
{
"cluster_name": "swh-logging-prod",
"status": "green",
"timed_out": false,
"number_of_nodes": 6,
"number_of_data_nodes": 6,
"active_primary_shards": 1933,
"active_shards": 3866,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100
}
Post cluster migration#
As the cluster should stay green all along the migration, there is nothing more to check (we just did that after each node).