Deep Dive in to Virtualization & Cloud

VMware Cloud on AWS

VMware Cloud on AWS is an integrated cloud offering jointly developed by AWS and VMware delivering a highly scalable, secure and innovative service that allows organizations to seamlessly migrate and extend their on-premises VMware vSphere-based environments to the AWS Cloud running on next-generation Amazon Elastic Compute Cloud (Amazon EC2) bare metal infrastructure. VMware Cloud on AWS is ideal for enterprise IT infrastructure and operations organizations looking to migrate their on-premises vSphere-based workloads to the public cloud, consolidate and extend their data center capacities, and optimize, simplify and modernize their disaster recovery solutions.

VMware Cloud on AWS is a hybrid cloud service that runs the VMware software-defined data center (SDDC) stack in the AWS public cloud, enables the combination of a wide variety VMware tools and interfaces with the flexibility and power of the AWS cloud. AWS provides an elastic, bare metal infrastructure on top of which VMware positions its SDDC stack, which includes NSX, vSphere and vSAN. Customers can manage clusters in their own data centers and clusters running in VMware Cloud on AWS from the same interface in vCenter.

Computing – vSphere Cluster Configuration

At initial availability, the VMware Cloud on AWS base cluster configuration contains 2TB of memory and four hosts. Each host is configured with 512GB of memory and contains dual CPU sockets that are populated by a custom-built Intel Xeon Processor E5-2686 v4 CPU package. Each socket contains 18 cores running at 2.3GHz, resulting in a physical cluster core count of 144.

The VMware vSphere Distributed Resource Scheduler (vSphere DRS) cluster uses a default configuration the migration threshold is set to the default vSphere DRS level three to avoid excessive VMware vSphere vMotion operations. VMware creates and operates a separate resource pool to manage customer workloads. Customers have the option to create child resource pools but cannot configure cluster affinity rules at initial availability.

VMware vSphere High Availability (vSphere HA) provides high availability for VMs by leveraging hosts and resources of a cluster to reserve capacity so workloads can fail over in case of host failures. Hosts in the cluster are monitored; in the event of a failure, the VMs on a failed host are restarted on alternative hosts. Host failure remediation is the responsibility of VMware.

Storage – VMware vSAN

The SDDC cluster includes a vSAN all-flash array. At initial availability of VMware Cloud on AWS, each host is equipped with eight NVMe devices and a total of 10TB of raw capacity, not including the cache capacity of the vSAN datastore, for the VMs to consume. Within a VMware Cloud on AWS four-host cluster configuration, 40TB of raw capacity, comprising all 32 encrypted NVMe devices, is available for the VMs to consume. The management VMs consume .9 percent of the vSAN datastore capacity. If the cluster is expanded to 16 hosts, 160TB of raw capacity is available for the VMs to consume, along with 128 encrypted NVMe devices. For all cluster configurations, the usable VM storage capacity depends on the per-VM storage policy.

Each host contains eight NVMe devices distributed across two vSAN disk groups. Within a disk group, the write-caching tier leverages one NVMe device with 1.7TB of storage; the storage capacity tier leverages the other three NVMe devices with a combined 5.1TB of storage. Although default storage policy configuration settings are in place, users can configure their own storage policies to provide the appropriate protection level against host and component failure. The default storage policy setting for fault tolerance is RAID 1, but users can select RAID 5 or RAID 6 instead, depending on the number of hosts in the cluster. VMware monitors the health and performance of the vSAN datastore; therefore, vSAN Health Monitoring and vSAN Performance Service are not exposed to the end user.

Datastore-level encryption with vSAN encryption, or VM-level encryption with vSphere VM encryption, is not available at initial availability of VMware Cloud on AWS. To provide data security, all local storage NVMe devices are encrypted at the firmware level by AWS. The encryption keys are managed by AWS and are not exposed to or controlled by VMware or VMware Cloud on AWS customers.

All VMs running inside the cloud SDDC consume storage capacity and leverage storage services from the vSAN datastore. Management workloads, and the workloads belonging to a single VMware Cloud on AWS customer, are located on the same vSAN cluster. However, the cloud SDDC introduces a new vSAN capability that provides two logical datastores instead of one. One of these datastores is used to store the management VMs; the other datastore is used for the customer VMs.

At initial availability, clusters are restricted to a single AWS region and availability zone (AZ). Failed hardware can be automatically detected, and automated remediation enables failed hosts to be automatically replaced by other cloud hosts and vSAN datastores to be automatically rebuilt—without user intervention.

Networking – VMware NSX

NSX is a key ingredient of VMware Cloud on AWS. It is not only optimized, along with vSphere, to work in the AWS environment, but it also provides all VM networking in VMware Cloud on AWS. NSX connects the VMware ESXi™ host and the abstract Amazon Virtual Private Cloud (VPC) networks. It enables ease of management by providing logical networks to VMs and automatically connecting new hosts to logical and VMkernel networks as clusters are scaled out. NSX is delivered using an “as a service” cloud model, and the version used in VMware Cloud on AWS provides compatibility between it and other vSphere products used on premises, such as vSphere vMotion.

VMware has introduced a basic networking service to ease the learning curve and enable everyone who uses vSphere to consume VMware Cloud on AWS as readily as possible. Cloud network administrators log in to the VMware Cloud on AWS portal and configure the network (“pre-creating”). They perform tasks such as establishing VPN connectivity and configuring firewall access rules. Next, cloud administrators log in to the vCenter Server platform with a VMware vSphere Web Client instance and consume the networks that the cloud network administrator created (“creating”). Although the cloud administrator can perform tasks such as creating logical networks and connecting VMs, the cloud network administrator permits traffic through the firewall and across the VPN networks.

To provide connectivity to VMware Cloud on AWS, two gateways are created. The management edge gateway (MGW) utilizes VMware NSX Edge™ to enable users to connect to the vCenter Server instance. They can configure firewall rules, an IPsec VPN, and DNS for the management gateway. The customer gateway (CGW) utilizes an NSX Edge instance and a distributed logical router (DLR) to enable ingress and egress of VM network traffic. Users can configure firewall rules, inbound NAT, VPN connections, DNS, and public IP addresses for their compute gateway. The initial customer configuration supports a single customer gateway. By default, all NSX Edge instances are large sized and are monitored for utilization. A default logical network is DHCP enabled and is provisioned with source NAT to provide outbound Internet connectivity.

An IPsec layer 3 VPN is set up to securely connect the on-premises vCenter Server instance with the management components running on the in-cloud SDDC cluster. A separate IPsec layer 3 VPN is set up to create connectivity between the on-premises workloads and the VMs running inside the in-cloud SDDC cluster. NSX is used for all networking and security and is decoupled from Amazon VPC networking. The compute gateway and DLR are preconfigured as part of the prescriptive network topology and cannot be changed by the customer. Customers must provide only their own subnets and IP ranges.

The Encrypted vMotion feature was introduced in VMware vSphere 6.5. It does not require a third-party key manager. It is set on a per-VM basis as one of the VM options. Encrypted vMotion encrypts the data traversing the vSphere vMotion network—not the network itself. It therefore requires no special configuration other than enabling it in the VM options. Encrypted vSphere vMotion migration between hosts inside the cloud SDDC is offered at initial availability of VMware Cloud on AWS.

Host Capacity and Availability Management

As hosts are added to the cluster, VMware Cloud on AWS automatically configures every VMkernel and logical network. After additional hosts are connected to the network, the vSAN datastore automatically expands, enabling the clusters to consume the new storage capacity.

VMware Cloud on AWS uses vSphere HA to ensure that outages are minimized. In the case of a failed host, VMs are automatically restarted on the surviving hosts. vSAN software ensures that any VM configured with a policy of one or more host failures does not lose data. VMware Cloud on AWS then examines the host and either reboots it, in the case of a transient failure, or replaces it, in the case of hardware fault. In either case, the SDDC continues to run and vSphere DRS optimizes VM placement to minimize impact on the running VMs. In the case of a degraded host, such as a failed disk, VMware Cloud on AWS efficiently removes the host by putting it in maintenance mode before eliminating it from the cluster. Customers are never billed for hosts that are added to a cluster for maintenance or fault tolerance reasons.

Hybrid Cloud Operations

At initial availability, only cold migration is available to transfer workloads to the cloud SDDC. However, cross-cloud vSphere vMotion migration will be available in future VMware Cloud on AWS releases, as well as per-VM Enhanced vMotion Compatibility, to provide proper vSphere vMotion compatibility between the in-cloud ESXi host’s CPU architecture and the customer’s on-premises ESXi host’s CPU architecture.

VMware Cloud on AWS is designed to provide single pane of glass monitoring for hybrid cloud management. The new Hybrid Linked Mode (HLM) feature enables on-premises and in-cloud vCenter Server instances to share data while maintaining some level of administrative separation. It also enables the linking of vCenter Server instances across different single sign on (SSO) domains, versions, and topologies. In addition, it provides operational consistency between vSphere environments on premises and multiple SDDC vCenter Server instances.

Hybrid Linked Mode enables users to complete the following functions:

  • Log in to the vCenter Server instance in their SDDC using their on-premises credentials
  • View and manage the inventories of both their on-premises data center and the cloud SDDC from a single vSphere client interface
  • Cold-migrate workloads between their on-premises data center and the cloud SDDC

To run HLM, users must have on-premises vCenter Server 6.5d or later, as well as layer 3 network connectivity. Because of the restrictive access model of VMware Cloud on AWS, HLM is restricted to connecting one on-premises Enhanced Linked Mode domain and does not have synchronized roles.

The VMware vCenter® content library feature effortlessly distributes and automatically synchronizes content—such as OVAs, ISO images, and scripts—between on-premises and cloud SDDC deployments. In addition, template support will be available in future VMware Cloud on AWS releases.

Operations Model

VMware Cloud on AWS is sold and operated as a service. To ensure that all environments perform correctly, VMware manages the systems exclusively. Likewise, VMware is the sole contact point for customers. In case of hardware failure, VMware interacts with AWS on the customer’s behalf, streamlining communication and remediation. The VMware Cloud on AWS service is also responsible for cloud SDDC software patching and for the application of updates.

The VMware Cloud on AWS service introduces a new cloud administrator role to the traditional vCenter Server user model and extends the roles and permissions scheme. This is to ensure that the cloud SDDC infrastructure is configured in a prescriptive deployment architecture and that the customer cloud administrator cannot reconfigure the management appliances. Within this model, the customer cloud administrator has full control over their workload while having a read-only view of management workloads and infrastructure.


VMware Cloud on AWS provides dedicated, single-tenant cloud infrastructure with support for up to 16 host vSphere clusters, delivered on the next-generation bare metal AWS infrastructure based on the latest Amazon EC2 Storage Optimized high I/O instances and featuring low-latency Non-Volatile Memory Express (NVMe) based SSDs. You can scale capacity by adding and removing hosts from clusters (3-16 hosts per cluster). VMware Cloud on AWS runs the VMware Software-Defined Data Center (SDDC) software stack directly on host servers without nested virtualization. You can move existing workloads between your existing VMware environment and VMware Cloud on AWS through cold migration, VM template migration, or even while a workload is running through live migration (vMotion).

Bare Metal Cloud Infrastructure – VMware Cloud on AWS provides the VMware SDDC software stack to the highly scalable AWS Cloud, including vSphere, vSAN, NSX, and vCenter Server. Each SDDC consists of 3 to 16 hosts, each with 36 cores, 512 GB of memory, and 15.2TB of raw NVMe storage. You can deploy a fully configured VMware SDDC Cluster in under a few hours, and scale host capacity up and down in minutes.

Flexible Storage Options – Each SDDC cluster utilizes an “all flash” vSAN storage solution built on NVMe instance storage. Each ESXi host has NVMe storage. You can also take advantage of advanced data services, including Quality of Service, snapshots, erasure coding, and VMware APIs for third-party data protection (VADP). Storage per host ranges from 15 to 35 TB in increments of 5 TB.  User chooses amount of storage desired and used on all hosts within the cluster.

Dedicated High Performance Networking – VMware Cloud on AWS provides separate, dedicated high performance networks for management and application traffic, connected through the VMware NSX networking platform, and provides support for networking multicasting. ESXi hosts are connected to an Amazon Virtual Private Cloud (VPC) through Elastic Networking Adapter (ENA), which supports throughput up to 25 Gbps.

NSX and AWS Direct Connect Integration – Now generally available, NSX integrates with AWS Direct Connect for end-to-end private networking. This is ideal for customers with traffic-heavy workloads. This enables private and consistent connectivity between VMware workloads running on AWS and those running on-premises and accelerates migration to cloud and enables multi-tier hybrid applications. Customers can now use AWS Direct Connect for all their hybrid connectivity requirements.

Security and Compliance – VMware Cloud on AWS lets you benefit from the AWS security-first approach, including IPsec VPN connectivity between your on-premises environment and VMware Cloud on AWS. You can utilize Network Address Translation (NAT) to establish connectivity for workloads running in your private subnet with VMware Cloud on AWS, and leverage network ACLs to control traffic to and from the subnet.

Micro-segmentation with NSX – Offers granular protection for each application workload, preventing the lateral spread of threats in cloud environments. This feature enables granular control over East-West traffic between application workloads running in the VMware Cloud on AWS SDDC. Security policies are dynamically enforced at the VM-level, preventing security threats from spreading across network.

vSAN encryption with AWS Key Management System (KMS) – This enables encryption of data at rest with AWS’s managed service for creating and controlling the encryption keys. All data in VMware Cloud on AWS is encrypted at no additional cost.

On-Demand Licensing – VMware Cloud on AWS supports custom-sized VMs, runs any OS supported by VMware, and makes use of single-tenant bare metal AWS infrastructure so that you can bring your Windows Server licenses to the AWS Cloud.

3rd Party Software Integration – Support for leading ISV partner solutions across categories including Data Protection, DevOps, Cloud Migration and Security. For details, see the VMware web site here.

Single Host SDDC – Single Host SDDC is our low-cost gateway into the VMware Cloud on AWS hybrid cloud solution. Typically purchased as a 3+ host service, it is the perfect way to test your first workload and leverage the additional capability and flexibility of VMware Cloud on AWS for 30 days.


If you have any comments, please drop me a line.
I hope this article was informative, and don’t forget to buy me a coffee if you found this worth reading.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.