vExpert

Deep Dive in to Virtualization & Cloud


VMware vCloud Director – Install and Configure RabbitMQ Cluster

VMware vCloud Director (vCD) is deployment, automation, and management software for virtual infrastructure resources in multi-tenant cloud environments. VMware vCloud Director provides role-based access to a Web console that allows the members of an organization to interact with the organization’s resources to create and work with vApps and virtual machines.

RabbitMQ is an open source message-queuing software which helps in facilitating message exchange between 2 or more applications. The Advanced Message Queuing Protocol (AMQP) is an open standard for message queuing that supports flexible messaging for enterprise systems. VMware Cloud Director uses the RabbitMQ AMQP broker to provide the message bus used by extension services, object extensions, and notifications between the vCloud Director cells and between other applications.

In this blog we will do learn how to install and configure RabbitMQ cluster and configure AMPQ in VMware vCloud Director.

Install RabbitMQ Standalone Node

In this configuration I am using CentOS 8 version to install RabbitMQ, also make sure required DNS entries and NTP configurations are done on the servers. Before installing RabbitMQ, you must install a supported version of Erlang/OTP. Please make sure you check the compatibility matrix for Erlang, RabbitMQ and vCloud Director.

As I am using vCloud Director 10.1, The supported RabbitMQ versions are 3.7.9 or 3.8.2. For RabbitMQ 3.8.2 minimum supported Erlang/OTP version is 21.3 and maximum is 22.x

Run below command to download Erlang supported version from GitHub or you can manually download a copy to server

#wget https://github.com/rabbitmq/erlang-rpm/releases/download/v22.3.4.11/erlang-22.3.4.11-1.el8.x86_64.rpm

Use below command to install the erlang version downloaded

#rpm -ivh erlang-22.3.4.11-1.el8.x86_64.rpm

Now install socat RPM which is a dependency of RabbitMQ

Download and Import RabbitMQ public key

#rpm --import https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc

Now time to install RabbitMQ server. In some cases, it may be easier to download the package and install it manually. The package can be downloaded from GitHub.

Create SSL Certificate for RabbitMQ Node

Now create a CA signed certificate for the RabbitMQ node you deployed using openssl

Edit your open ssl configuration file as below to generate your CSR and key

[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:dxbvxpertrmq01, IP:172.16.6.5, DNS:dxbvxpertrmq01.vxpert.local

[ req_distinguished_name ]
countryName = AE
stateOrProvinceName = Dubai
localityName = Dubai Silicon Oasis
0.organizationName = Vxpert
organizationalUnitName = Information Technology
commonName = dxbvxpertrmq01.vxpert.local

Open a command prompt and navigate to the OpenSSL directory as previously configured in the Configuring OpenSSL article. By default, this is C:\OpenSSL-Win32\bin. Run the below command

./openssl.exe req -new -nodes -out rui.csr -keyout rui-orig.key -config openssl.cfg

There are no prompts because all information was provided in the openssl.cfg file as configured before and this creates the certificate request rui.csr and rui.orig.key files in same directory.

Convert the Key to be in RSA format by running these command

./openssl.exe rsa -in rui-orig.key -out rui.key

After the certificate request is created, the certificate must be given to the certificate authority for generation of the actual certificate. The authority presents a certificate back, as well as a copy of their root certificate, if necessary. For the certificate chain to be trusted, the root certificate must be installed on the server.

Now you have your signed certificate and Root certificate in CER format, convert them to PEM

./openssl x509 -in CER_CERT -outform PEM -out PEM_CERT

Now create server client and ca directories inside /etc/rabbitmq

Now copy your server certificate, key, and Root certificate to RabbitMQ server and further to respective RabbitMQ directory as per your configuration file

Now change the ownership of these certificate directories to rabbitmq user

Create rabbitmq.confg file in /etc/rabbitmq directory with below contents. This will configure RabbitMQ to listen on SSL port 5671 and change management URL to HTTPS with port 15671

[
  {ssl, [{versions, ['tlsv1.2']}]},
  {rabbit, [
      {tcp_listeners, []},
      {ssl_listeners, [5671]},
      {ssl_options, [
        {cacertfile, "/etc/rabbitmq/ca/RootCA.pem"},
        {certfile,   "/etc/rabbitmq/server/dxbvxpertrmq01.pem"},
        {keyfile,    "/etc/rabbitmq/server/key.pem"},
        {fail_if_no_peer_cert,true},
{ciphers, ["ECDHE-ECDSA-AES256-GCM-SHA384","ECDHE-RSA-AES256-GCM-SHA384",
"ECDHE-ECDSA-AES256-SHA384","ECDHE-RSA-AES256-SHA384",
"ECDH-ECDSA-AES256-GCM-SHA384","ECDH-RSA-AES256-GCM-SHA384",
"ECDH-ECDSA-AES256-SHA384","ECDH-RSA-AES256-SHA384",
"DHE-RSA-AES256-GCM-SHA384","DHE-DSS-AES256-GCM-SHA384",
"DHE-RSA-AES256-SHA256","DHE-DSS-AES256-SHA256",
"AES256-GCM-SHA384","AES256-SHA256","ECDHE-ECDSA-AES128-GCM-SHA256",
"ECDHE-RSA-AES128-GCM-SHA256","ECDHE-ECDSA-AES128-SHA256",
"ECDHE-RSA-AES128-SHA256","ECDH-ECDSA-AES128-GCM-SHA256",
"ECDH-RSA-AES128-GCM-SHA256","ECDH-ECDSA-AES128-SHA256",
"ECDH-RSA-AES128-SHA256","DHE-RSA-AES128-GCM-SHA256",
"DHE-DSS-AES128-GCM-SHA256","DHE-RSA-AES128-SHA256",
"DHE-DSS-AES128-SHA256","AES128-GCM-SHA256","AES128-SHA256",
"ECDHE-ECDSA-AES256-SHA","ECDHE-RSA-AES256-SHA",
"DHE-RSA-AES256-SHA","DHE-DSS-AES256-SHA","ECDH-ECDSA-AES256-SHA",
"ECDH-RSA-AES256-SHA","AES256-SHA","ECDHE-ECDSA-DES-CBC3-SHA",
"ECDHE-RSA-DES-CBC3-SHA","EDH-RSA-DES-CBC3-SHA","EDH-DSS-DES-CBC3-SHA",
"ECDH-ECDSA-DES-CBC3-SHA","ECDH-RSA-DES-CBC3-SHA","DES-CBC3-SHA",
"ECDHE-ECDSA-AES128-SHA","ECDHE-RSA-AES128-SHA","DHE-RSA-AES128-SHA",
"DHE-DSS-AES128-SHA","ECDH-ECDSA-AES128-SHA","ECDH-RSA-AES128-SHA",
"AES128-SHA","EDH-RSA-DES-CBC-SHA","DES-CBC-SHA"]},
        {versions, ['tlsv1.2']}
      ]}
    ]
  },
  {rabbitmq_management, [
    {listener, [
      {port, 15671},
      {ssl,  true},
      {ssl_opts, [
        {cacertfile, "/etc/rabbitmq/ca/RootCA.pem"},
        {certfile,   "/etc/rabbitmq/server/dxbvxpertrmq01.pem"},
        {keyfile,    "/etc/rabbitmq/server/key.pem"},
        {fail_if_no_peer_cert,true},
{ciphers, ["ECDHE-ECDSA-AES256-GCM-SHA384","ECDHE-RSA-AES256-GCM-SHA384",
"ECDHE-ECDSA-AES256-SHA384","ECDHE-RSA-AES256-SHA384",
"ECDH-ECDSA-AES256-GCM-SHA384","ECDH-RSA-AES256-GCM-SHA384",
"ECDH-ECDSA-AES256-SHA384","ECDH-RSA-AES256-SHA384",
"DHE-RSA-AES256-GCM-SHA384","DHE-DSS-AES256-GCM-SHA384",
"DHE-RSA-AES256-SHA256","DHE-DSS-AES256-SHA256","AES256-GCM-SHA384",
"AES256-SHA256","ECDH-RSA-AES128-SHA","ECDHE-ECDSA-AES128-GCM-SHA256",
"ECDHE-RSA-AES128-GCM-SHA256","ECDHE-ECDSA-AES128-SHA256",
"ECDHE-RSA-AES128-SHA256","ECDH-ECDSA-AES128-GCM-SHA256",
"ECDH-RSA-AES128-GCM-SHA256","ECDH-ECDSA-AES128-SHA256",
"ECDH-RSA-AES128-SHA256","DHE-RSA-AES128-GCM-SHA256",
"DHE-DSS-AES128-GCM-SHA256","DHE-RSA-AES128-SHA256",
"DHE-DSS-AES128-SHA256","AES128-GCM-SHA256","AES128-SHA256",
"ECDHE-ECDSA-AES256-SHA","ECDHE-RSA-AES256-SHA","DHE-RSA-AES256-SHA",
"DHE-DSS-AES256-SHA","ECDH-ECDSA-AES256-SHA","ECDH-RSA-AES256-SHA",
"AES256-SHA","ECDHE-ECDSA-DES-CBC3-SHA","ECDHE-RSA-DES-CBC3-SHA",
"EDH-RSA-DES-CBC3-SHA","EDH-DSS-DES-CBC3-SHA","ECDH-ECDSA-DES-CBC3-SHA",
"ECDH-RSA-DES-CBC3-SHA","DES-CBC3-SHA","ECDHE-ECDSA-AES128-SHA",
"ECDHE-RSA-AES128-SHA","DHE-RSA-AES128-SHA","DHE-DSS-AES128-SHA",
"ECDH-ECDSA-AES128-SHA","AES128-SHA","EDH-RSA-DES-CBC-SHA","DES-CBC-SHA"]},
        {versions, ['tlsv1.2']}
      ]}
    ]}
]}
].

Now start the RabbitMQ service and enable service to run in startup

#systemctl start rabbitmq-server
#systemctl enable rabbitmq-server

Run the below command to enable RabbitMQ UI on https://FQDN or IP Address:15671/

#rabbitmq-plugins enable rabbitmq_management

Also review the ports your RabbitMQ server is listening

Now Create new user for RMQ administration, same you can do in GUI as well

#rabbitmqctl add_user rmqadmin

Set permissions for the rmqadmin user created

#rabbitmqctl set_permissions -p / rmqadmin ".*" ".*" ".*"

Set tag for the admin user created

#rabbitmqctl set_user_tags rmqadmin administrator

Now login with the admin account to the Web UI

Configure RabbitMQ Clustering

A RabbitMQ cluster is a logical grouping of one or several nodes, each sharing users, virtual hosts, queues, exchanges, bindings, runtime parameters and other distributed state. All data/state required for the operation of a RabbitMQ broker is replicated across all nodes. An exception to this is message queues, which by default reside on one node, though they are visible and reachable from all nodes. To replicate queues across nodes in a cluster, use a queue type that supports replication. This topic is covered in the Quorum Queues and Classic Mirrored Queues guides. There are multiple ways to configure RabbitMQ cluster and we will use rabbitmqctl to configure the same.

Before proceeding with the configuration make sure the cluster name and individual FQDN and hostnames are getting resolved from all nodes. In this blog I have installed one more RabbitMQ cluster with same above configuration.

RabbitMQ nodes and CLI tools (e.g. rabbitmqctl) use a cookie to determine whether they can communicate with each other. For two nodes to be able to communicate they must have the same shared secret called the Erlang cookie. The cookie is just a string of alphanumeric characters. Every cluster node must have the same cookie. Copy the Erlang cookie as mentioned below

#cd /var/lib/rabbitmq/
#scp -v .erlang.cookie root@server02:/var/lib/rabbitmq/

Once copied the cookie value is same on both RabbitMQ server

Now Stop second RabbitMQ app node and reset it.

#rabbitmqctl stop_app
#rabbitmqctl reset

Before adding second node to cluster you can check the current cluster status on both nodes

#rabbitmqctl cluster_status

Run the below command to add the second node in cluster with first node.

#rabbitmqctl join_cluster rabbit@dxbvxpertrmq01

Now start the RabbitMQ app in second node

#rabbitmqctl start_app

Check the cluster status by running below command

#rabbitmqctl cluster_status

Also, you can see the cluster nodes and status details in Web UI

By default, contents of a queue within a RabbitMQ cluster are located on a single node (the node on which the queue was declared). This contrasts with exchanges and bindings, which can always be on all nodes. Queues can optionally be made mirrored across multiple nodes. Each mirrored queue consists of one master and one or more mirrors. The master is hosted on one node commonly referred as the master node. Each queue has its own master node. All operations for a given queue are first applied on the queue’s master node and then propagated to mirrors.

To sync all the queues across all nodes run the below command

#rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

Review the applied policy in Web UI also in Admin – Policies tab

Load Balancing RabbitMQ Cluster

Building a RabbitMQ cluster to ensure availability and performance is only half the battle of ensuring a resilient messaging infrastructure. The other half is writing applications that expect node failure and knowing how to reconnect to the cluster when it happens. There are several strategies for handling reconnection to the cluster, but the one we will focus on is using a load balancer to handle node selection. By using a load balancer, you not only reduce the complexity of the failure handling code in your apps, but you also ensure even connection distribution across your cluster.

In my lab I have configured RabbitMQ behind KEMP Load Balancer for Management URL as well as AMPQ SLL port

Configure AMPQ Broker in vCloud Director

To configure AMPQ Broker in vCD Login with provider credentials

Click on Administration – Extensibility – AMPQ Broker

Click on Edit to configure the AMPQ host address which is Load Balanced URL in our case with SSL port and the Load Balancer URL certificate. Also provide the credentials to connect with RabbitMQ cluster.

Save and Test your configuration to check the connectivity

Also, you can see the Queues details in RabbitMQ management Web UI

In upcoming blogs, we will discuss more on VMware vCloud Director. To make this vCD series more understandable, I am splitting this into multiple blogs

Part 1 – What is VMware vCloud Director

Part 2 – VMware vCloud Director Standalone Installation

Part 3 – VMware vCloud Director Installation with High Availability

Part 4 – VMware vCloud Director – Install and Configure RabbitMQ Cluster

Part 5 – VMware vCloud Director Cell Certificates & Load Balancing

Part 6 – VMware vCloud Director Cell Role Switchover

Part 7 – VMware vCloud Director Basic Configuration

Part 8 – VMware vCloud Director Org vDC Configuration

Part 9 – VMware vCloud Director Virtual Data Center

Part 10 – VMware vCloud Director Branding

Thanks,

If you have any comments, please drop me a line.
I hope this article was informative, and don’t forget to buy me a coffee if you found this worth reading.



Leave a Reply

Your email address will not be published. Required fields are marked *


This site uses Akismet to reduce spam. Learn how your comment data is processed.