vExpert

Deep Dive in to Virtualization & Cloud


VMware Cloud Director – Upgrade RabbitMQ Cluster

VMware Cloud Director (vCD) is deployment, automation, and management software for virtual infrastructure resources in multi-tenant cloud environments. VMware vCloud Director provides role-based access to a Web console that allows the members of an organization to interact with the organization’s resources to create and work with vApps and virtual machines.

RabbitMQ is an open-source message-queuing software which helps in facilitating message exchange between 2 or more applications. The Advanced Message Queuing Protocol (AMQP) is an open standard for message queuing that supports flexible messaging for enterprise systems. VMware Cloud Director uses the RabbitMQ AMQP broker to provide the message bus used by extension services, object extensions, and notifications between the vCloud Director cells and between other applications.

Note: – If you want to use blocking tasks, notifications, or VMware Cloud Director API extensions, like Container Service Extension (CSE) and VMware Cloud Director App Launchpad, you must install and configure a RabbitMQ AMQP Broker for VMware Cloud Director versions older than 10.2. But from 10.2 version onwards VMware Cloud Director uses MQTT (MQ Telemetry Transport) protocol to exchange messages between cells and same is supported by Container Service Extension (CSE) and VMware Cloud Director App Launchpad.

In this blog we will upgrade RabbitMQ cluster configured for a VMware vCloud Director. To know how to install and configure a RabbitMQ cluster please visit my previous blog.

Prerequisites

It Is Important to make sure the Integrated products are also compatible with RabbitMQ version you are upgrading to. Below are the VMware products which is connect to VMware Cloud Director in a provider infrastructure.

VMware Cloud Director
VMware vRealize Operations Tenant App
Container Service Extension
Object Storage Extension
VMware App Launchpad

Note: – I am running VMware Cloud Director 10.3 as well as 10.4 in different environments and both supports 3.8.x version of RabbitMQ, and subsequent product also supports the same.

Back up or take a snapshot of the source RabbitMQ VMs that you want to upgrade.

As root, enable and start SSH on the source RabbitMQ VMs by running the commands.

#systemctl enable sshd
#systemctl start sshd

You must have a password to log in to the RabbitMQ as root.

Investigate if the current and target versions have an in-place upgrade path with supported Erlang version

RabbitMQ version upgradability

RabbitMQ Erlang version requirements

Recommend that you upgrade to supported Erlang together with RabbitMQ

Operating system support should be available from the customer.

Depending on cluster configuration, you can use either single node upgraderolling upgrade or full-stop upgrade strategy.

Detailed Upgradation Procedure

There are two major upgrade scenarios that are covered in this guide: a single node and a cluster, and two most commonly used strategies:

In-place upgrade where each node is upgraded with its existing on disk data

Blue-green deployment where a new cluster is created, and existing data is migrated to it

Note: – The Blue/Green deployment strategy offers the benefit of making the upgrade process safer at the cost of temporary increasing infrastructure footprint. The safety aspect comes from the fact that the operator can abort an upgrade by switching applications back to the existing cluster. In this blog we will see how we can upgrade to supported version with in-place upgrade.

In-Place Rolling Upgrade

In current environment I am running RabbitMQ 3.8.2 with Erlang version 22.3.4.11. This version supports my VMware Cloud Director 10.3 and 10.4 but still I need to upgrade to latest version of 3.8.x supported by same Erlang version.

The current version of Erlang 22.3.4.11 which supports only till RabbitMQ 3.8.15. So, I will upgrade to 3.8.15 by stopping only one node in cluster.

Note: – RabbitMQ 3.8.0 comes with a feature flag subsystem which is responsible for determining if two versions of RabbitMQ are compatible. If they are, then two nodes with different versions can live in the same cluster: this allows a rolling upgrade of cluster members without shutting down the cluster entirely. Rolling upgrades are possible only between compatible RabbitMQ and Erlang versions.

Log in to the RabbitMQ VMs using root credentials and check that the node or cluster is in a good state to be upgraded

#sudo rabbitmqctl cluster_status

Upgrade binaries are available in below links for the specific versions. Download this and move to respective servers.

RabbitMQ Binaries

Erlang Binaries

When current and target version of RabbitMQ is compatible with the same version of Erlang, then an upgrade is very straight forward. RabbitMQ will simply update the existing RabbitMQ install and carry forward existing settings, such as feature flags and cluster information, when performing an upgrade

Stop the node

#systemctl stop rabbitmq-server

Upgrade RabbitMQ and, if applicable, Erlang

#yum install rabbitmq-server 3.8.8.x

OR

#rpm -Uvh rabbitmq-server 3.8.8.x

Start the node

#systemctl start rabbitmq-server

Repeat the same step in second node as well to complete the cluster upgrade

Check the cluster health status again and check the version of both nodes in output.

#sudo rabbitmqctl cluster_status

Also in the Web UI you can see the updated RabbitMQ version.

Also from VMware Cloud Director Administration SettingsExtensibility initiate a AMQP Brocker Test to make sure connection is working fine.

Here the AMQP host name is the FQDN whichis load balanced and is explained in Install and Configure Blog

Now We have upgraded the RabbitMQ version to 3.8.15 with same Erlang version 22.3.4.11. If you want further upgrade to latest version where Erlang version also require upgrade, you can go with Full stop upgrade method but this will require more downtime and configuration. I would recommend to go with Blue-Green deployment(create another cluster in parrelel) and replace the production configuration with new RabbitMQ cluster.

Thanks,

If you have any comments, please drop me a line.
I hope this article was informative, and don’t forget to buy me a coffee if you found this worth reading.



Leave a Reply

Your email address will not be published. Required fields are marked *


This site uses Akismet to reduce spam. Learn how your comment data is processed.