Upgrade Procedure for Debian Nodes in Kafka Cluster#
Intended audience
sysadm staff members
Purpose#
This page documents the steps to upgrade Debian nodes running in a Kafka cluster. The upgrade process involves various commands and checks before and after rebooting the node.
Prerequisites#
Familiarity with SSH and CLI-based command execution
Out-of-band Access to the node (IDRAC/ILO) for reboot
Access to the node through SSH (requires the vpn)
Step 0: Initial Steps#
The kafka nodes are mostly running on bare metal machines. So, ensuring the out of band access to the machine is ok. This definitely helps when something goes wrong during a reboot (disk order or names change, network, …).
Note: For (kubernetes) pods running kafka, this kind is dealt with automatically for kafka operator so this documentation does not cover this part.
Step 1: Migrate to the next debian suite#
Update the Debian version of the node (e.g. bullseye to bookworm) using the following command:
root@node:~# /usr/local/bin/migrate-to-${NEXT_CODENAME}.sh
Note: The script should be present on the machine (installed through puppet).
Step 2: Run Puppet Agent#
Once the upgrade procedure happened, run the puppet agent to apply any necessary configuration changes (e.g. /etc/apt/sources.list change, etc…)
root@node:~# puppet agent -t
Step 2: Stop Puppet Agent#
As we will stop the service, we don’t want the agent to start it back again.
root@node:~# puppet agent --disable "Ongoing debian upgrade"
Step 4: Autoremove and Purge#
Perform autoremove to remove unnecessary packages left-over from the migration.
root@node:~# apt autoremove
Step 5: Stop the kafka service#
The cluster can support one non-responding node so it’s ok to stop the service (and let the node leave the cluster for maintenance operation).
We can check the cluster’s status which should stay green after the elasticsearch service is stopped.
root@node:~# systemctl stop kafka
root@node:~# kcat -b $server
Note: $server
if of the form hostname:9092
(with the hostname another cluster
node than the one we are currently upgrading). kcat
is a program from the kafkacat
debian package.
Step 6: Reboot the node#
We are ready to reboot the node:
root@node:~# reboot
You can connect to the serial console of the machine to follow through the reboot (and unstuck it if any problem arises).
Step 7: Clean up some more#
Once the machine is restarted, some cleanup might be necessary.
root@node:~# apt autopurge
Step 8: Activate puppet agent#
Activate back the puppet agent and make it run. This will start back the disabled service again.
root@node:~# puppet agent --enable && puppet agent --test
Step 8: Join back the cluster#
After the service restarted, check the node joined back the cluster.
root@node:~# kcat -b $server
Example:
root@kafka1:~# kcat -L -b $server | head
Metadata for all topics (from broker -1: kafka1:9092/bootstrap):
2 brokers:
broker 3 at kafka3.internal.staging.swh.network:9092
broker 1 at journal2.internal.staging.swh.network:9092 (controller)