VMware Cloud Director (vCD) is deployment, automation, and management software for virtual infrastructure resources in multi-tenant cloud environments. VMware vCloud Director provides role-based access to a Web console that allows the members of an organization to interact with the organization’s resources to create and work with vApps and virtual machines.
RabbitMQ is an open-source message-queuing software which helps in facilitating message exchange between 2 or more applications. The Advanced Message Queuing Protocol (AMQP) is an open standard for message queuing that supports flexible messaging for enterprise systems. VMware Cloud Director uses the RabbitMQ AMQP broker to provide the message bus used by extension services, object extensions, and notifications between the vCloud Director cells and between other applications.
Note: – If you want to use blocking tasks, notifications, or VMware Cloud Director API extensions, like Container Service Extension (CSE) and VMware Cloud Director App Launchpad, you must install and configure a RabbitMQ AMQP Broker for VMware Cloud Director versions older than 10.2. But from 10.2 version onwards VMware Cloud Director uses MQTT (MQ Telemetry Transport) protocol to exchange messages between cells and same is supported by Container Service Extension (CSE) and VMware Cloud Director App Launchpad.
In this blog we will upgrade RabbitMQ cluster configured for a VMware vCloud Director. To know how to install and configure a RabbitMQ cluster please visit my previous blog.
It Is Important to make sure the Integrated products are also compatible with RabbitMQ version you are upgrading to. Below are the VMware products which is connect to VMware Cloud Director in a provider infrastructure.
VMware Cloud Director
VMware vRealize Operations Tenant App
Container Service Extension
Object Storage Extension
VMware App Launchpad
Note: – I am running VMware Cloud Director 10.3 as well as 10.4 in different environments and both supports 3.8.x version of RabbitMQ, and subsequent product also supports the same.
Back up or take a snapshot of the source RabbitMQ VMs that you want to upgrade.
As root, enable and start SSH on the source RabbitMQ VMs by running the commands.
#systemctl enable sshd #systemctl start sshd
You must have a password to log in to the RabbitMQ as root.
Investigate if the current and target versions have an in-place upgrade path with supported Erlang version
Recommend that you upgrade to supported Erlang together with RabbitMQ
Operating system support should be available from the customer.
In-place upgrade where each node is upgraded with its existing on disk data
Blue-green deployment where a new cluster is created, and existing data is migrated to it
Note: – The Blue/Green deployment strategy offers the benefit of making the upgrade process safer at the cost of temporary increasing infrastructure footprint. The safety aspect comes from the fact that the operator can abort an upgrade by switching applications back to the existing cluster. In this blog we will see how we can upgrade to supported version with in-place upgrade.
In-Place Rolling Upgrade
In current environment I am running RabbitMQ 3.8.2 with Erlang version 22.214.171.124. This version supports my VMware Cloud Director 10.3 and 10.4 but still I need to upgrade to latest version of 3.8.x supported by same Erlang version.
The current version of Erlang 126.96.36.199 which supports only till RabbitMQ 3.8.15. So, I will upgrade to 3.8.15 by stopping only one node in cluster.
Note: – RabbitMQ 3.8.0 comes with a feature flag subsystem which is responsible for determining if two versions of RabbitMQ are compatible. If they are, then two nodes with different versions can live in the same cluster: this allows a rolling upgrade of cluster members without shutting down the cluster entirely. Rolling upgrades are possible only between compatible RabbitMQ and Erlang versions.
Log in to the RabbitMQ VMs using root credentials and check that the node or cluster is in a good state to be upgraded
#sudo rabbitmqctl cluster_status
Upgrade binaries are available in below links for the specific versions. Download this and move to respective servers.
When current and target version of RabbitMQ is compatible with the same version of Erlang, then an upgrade is very straight forward. RabbitMQ will simply update the existing RabbitMQ install and carry forward existing settings, such as feature flags and cluster information, when performing an upgrade
Stop the node
#systemctl stop rabbitmq-server
Upgrade RabbitMQ and, if applicable, Erlang
#yum install rabbitmq-server 3.8.8.x
#rpm -Uvh rabbitmq-server 3.8.8.x
Start the node
#systemctl start rabbitmq-server
Repeat the same step in second node as well to complete the cluster upgrade
Check the cluster health status again and check the version of both nodes in output.
#sudo rabbitmqctl cluster_status
Also in the Web UI you can see the updated RabbitMQ version.
Also from VMware Cloud Director Administration – Settings – Extensibility initiate a AMQP Brocker Test to make sure connection is working fine.
Here the AMQP host name is the FQDN whichis load balanced and is explained in Install and Configure Blog
Now We have upgraded the RabbitMQ version to 3.8.15 with same Erlang version 188.8.131.52. If you want further upgrade to latest version where Erlang version also require upgrade, you can go with Full stop upgrade method but this will require more downtime and configuration. I would recommend to go with Blue-Green deployment(create another cluster in parrelel) and replace the production configuration with new RabbitMQ cluster.
If you have any comments, please drop me a line