Rossum R&D: Moving RabbitMQ from Bitnami’s Catalog

It is sad to see an excellent open-source project close its doors and unfortunately that is what is happening with Bitnami’s (now owned by Broadcom) open container & chart catalog. We too were using several of their charts and images within Rossum’s infrastructure. As the price for their new commercial offering is a bit too steep for a startup like us, we have been busy migrating away. In this post I would like to detail the process we took to switch our RabbitMQ clusters away from Bitnami charts and images and onto the images published by the Rabbitmq opensource project.

Step 1: What do we migrate?

We are deploying RabbitMQ clusters via the RabbitMQ Kubernetes Cluster Operator and configuring it via RabbitMQ Kubernetes Topology Operator. Each deployed cluster also uses a specific container image for it’s nodes. That means we need to migrate:

1. Cluster Operator & Topology Operator Helm chart
2. Cluster Operator container image
3. Topology Operator container image
4. Container images of each RMQ cluster we run

Step 2: Remove the helm chart

We can quite easily remove the Helm chart by rendering it down to a simple kubernetes yaml file containing all the managed resources that you can deploy to your cluster directly without helm (e.g., using Flux CD, like we do).

helm template rabbitmq-cluster-operator bitnami/rabbitmq-cluster-operator --version 4.4.25 --namespace rabbitmq-system --set commonLabels.rossum\\.ai/owner=sre --set clusterOperator.replicaCount=2 --set useCertManager=true --set msgTopologyOperator.replicaCount=2 --include-crds > upstream-resources.yaml

Step 3: Replace the helm-generated manifests with open-source release files

The problem with keeping the file like this is maintenance. Manually replacing parts of it when time for update comes is not feasible for us. That is why we replaced the upstream-resources.yaml file with the resource files published by the RabbitMQ project. Those are regularly updated and maintained and we can easily update them as a whole. Unfortunately they do not solve all our issues.

By carefully comparing (and breaking the develop environment on our first try 🙈) the two manifests, we found out several crucial differences. Namely the manifests we are switching to don’t support some features we need out of the box:

1. Running operators with multiple replicas
2. Ensuring the operators stay running with PDBs
3. Allowing apps and users with admin/edit/view roles to access RabbitMQ custom resources

(The list of missing features might differ for you because everyone’s needs are different and everyone uses different chart values.)

Fortunately we can add what is missing with the power of Kustomize and avoid having to edit the manifest file we will be replacing during each upgrade. You can find the result in this repository.

Step 4: Applying the operator changes

Now is the time to carefully deploy the new manifests. You will likely hit an error similar to this one:

Deployment/rabbitmq-system/rabbitmq-cluster-operator dry-run failed (Invalid): Deployment.apps "rabbitmq-cluster-operator" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/component":"rabbitmq-operator", "app.kubernetes.io/instance":"rabbitmq-cluster-operator", "app.kubernetes.io/name":"rabbitmq-cluster-operator"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

The new manifests have a different labeling convention, and Kubernetes doesn’t support changing labels in Deployment selectors. The easiest way to fix the error is by removing the old deployment before reapplying the new manifests:

kubectl delete deployment -n rabbitmq-system rabbitmq-cluster-operator

This will cause a short operator (not RMQ cluster) outage but that shouldn’t matter much unless you are creating / destroying whole RMQ clusters often. After the operators restart they will be running with the images published by the RabbitMQ project instead of the ones managed by Bitnami.

Step 5: Replacing images of running RMQ clusters

We are now at a stage where both operators are completely migrated and new RabbitMQ clusters you create will use RabbitMQ project container images by default. What remains to be done is switching your existing clusters to the new images too.

This operation is quite simple but requires rolling cluster restart. All you have to do is modify the spec.image field of each RabbitmqCluster resource you have. You should make sure to use the same version number you ran before.

kubectl patch rabbitmqcluster cluster \
  --type='merge' -p '{"spec":{"image":"rabbitmq:4.1.3-management"}}'

Done!

Now you can endure the chaos that came on August 28th in peace.