Tips for using USB Network Adapters with VMware ESXi

Running Intel NUCs and other SFF systems with ESXi is a proven standard for virtualization home labs. One major drawback is that most of the available SFF systems only have a single Gigabit network adapter. This might be sufficient for a standalone ESXi with a few VMs, but when you want to use shared Storage or VMware NSX, you totally want to have additional NICs.

This article explains some basics to consider when running USB-based network adapters with ESXi.

VMKUSB NIC Fling Installation and Update

The USB Native Driver Fling is a popular ESXi driver by Songtao Zheng and William Lam that adds support for USB-based Network Adapters. Currently, it is the only driver that uses the Native Driver model, which is required to be compatible with vSphere 7.0.

Download: USB Network Native Driver for ESXi (VMware Flings)

The installation or update is very simple. Just download the driver for your ESXi version (available for ESXi 7.0, 6.7, and 6.5), run the following command, and reboot your ESXi Host. Make sure to always use the full path to the bundle when installing with esxcli.

# esxcli software vib install -d /path/ESXi700-VMKUSB-NIC-FLING-39035884-component-16770668.zip

If you are running ESXi 7.0, you can also use the new component feature:

# esxcli software component apply -d /path/ESXi700-VMKUSB-NIC-FLING-39035884-component-16770668.zip

Create a Customized ESXi Installer including the Driver

You can also create a customized ESXi Installer that contains the driver. This option can help when you want to run ESXi on a server that has no compatible network adapter.

Creating a customized ESXi Installer is very easy with PowerShell. Use the following snippet to create an Image containing the driver. You can change the Base Profile to any available Version (ESXi Image Profiles). Make sure to use uncomment the Fling version that fits your ESXi version.

# ESXi 7.0 Image with USB NIC Fling 1.6:

$baseProfile = "ESXi-7.0.0-15843807-standard"   # See https://www.virten.net/vmware/vmware-esxi-image-profiles/ for available Image Profiles
$usbFling = "ESXi700-VMKUSB-NIC-FLING-39035884-component-16770668.zip"        # Uncomment for ESXi 7.0
#$usbFling = "ESXi670-VMKUSB-NIC-FLING-39203948-offline_bundle-16780994.zip"  # Uncomment for ESXi 6.7
#$usbFling = "ESXi650-VMKUSB-NIC-FLING-39176435-offline_bundle-16775917.zip"  # Uncomment for ESXi 6.5

Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
Export-ESXImageProfile -ImageProfile $baseProfile -ExportToBundle -filepath "$($baseProfile).zip"
Remove-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
Invoke-WebRequest -Method "GET" https://download3.vmware.com/software/vmw-tools/USBNND/$($usbFling) -OutFile $($usbFling)
Add-EsxSoftwareDepot "$($baseProfile).zip"
Add-EsxSoftwareDepot $usbFling
$newProfile = New-EsxImageProfile -CloneProfile $baseProfile -name $($baseProfile.Replace("standard", "usbnic")) -Vendor "virten.net"
Add-EsxSoftwarePackage -ImageProfile $newProfile -SoftwarePackage "vmkusb-nic-fling"
Export-ESXImageProfile -ImageProfile $newProfile -ExportToIso -filepath "$($baseProfile.Replace("standard", "usbnic")).iso"
Export-ESXImageProfile -ImageProfile $newProfile -ExportToBundle -filepath "$($baseProfile.Replace("standard", "usbnic")).zip"

Note: The ESXi Installer might get stuck at 81% when a system has only USB-based network adapters during installation. See this article for instructions to install ESXi with USB NICs only.

Label your Network Adapters

When using multiple USB network adapters, it is a good practice to label them. That helps to identify the correct adapter when doing the cabling, switch configuration, and adapter mapping. I personally mark all adapters with:

  • MAC Address
  • ESXi Hostname
  • vusbX Adapter Number

Create Adapter to vusbX mappings

The adapter to vusb number mapping is not persistent during reboots. That might create an issue when adapters are connected to switch ports with different VLANs, or when cross-connecting adapters. Starting with Version 1.6, you can now persist the mapping based on the adapters MAC Address.

Identify adapter MAC addresses:

# esxcli network nic list |grep vusb |awk '{print $1, $8}'
vusb0 00:23:54:8c:43:45
vusb1 a0:ce:c8:cd:3d:5d

Make the current mapping persistent by adding the following option to the vmkusb_nic_fling module:
vusb0_mac=00:23:54:8c:43:45 vusb1_mac=a0:ce:c8:cd:3d:5d

# esxcli system module parameters set -p "vusb0_mac=00:23:54:8c:43:45 vusb1_mac=a0:ce:c8:cd:3d:5d" -m vmkusb_nic_fling

Please note that you always have to define all adapters, using a space as separator. If you want to add additional mappings, you have to include all previous options, or it will overwrite the previous.

Use esxcli system module parameters list -m vmkusb_nic_fling to verify the mapping:

# esxcli system module parameters list -m vmkusb_nic_fling
Name                        Type    Value              Description
--------------------------  ------  -----------------  -----------
usbCdromPassthroughEnabled  int                        Enable USB CDROM device for USB passtrough: 0 No, 1 Yes
vusb0_mac                   string  00:23:54:8c:43:45  persist vusb0 MAC Address: xx:xx:xx:xx:xx:xx
vusb1_mac                   string  a0:ce:c8:cd:3d:5d  persist vusb1 MAC Address: xx:xx:xx:xx:xx:xx
vusb2_mac                   string                     persist vusb2 MAC Address: xx:xx:xx:xx:xx:xx
vusb3_mac                   string                     persist vusb3 MAC Address: xx:xx:xx:xx:xx:xx
vusb4_mac                   string                     persist vusb4 MAC Address: xx:xx:xx:xx:xx:xx
vusb5_mac                   string                     persist vusb5 MAC Address: xx:xx:xx:xx:xx:xx
vusb6_mac                   string                     persist vusb6 MAC Address: xx:xx:xx:xx:xx:xx
vusb7_mac                   string                     persist vusb7 MAC Address: xx:xx:xx:xx:xx:xx

If you want to make the current mapping persistent, just use the following one-liner:

# esxcli system module parameters set -p "$(esxcli network nic list |grep vusb |awk '{print $1 "_mac=" $8}' | awk 1 ORS=' ')" -m vmkusb_nic_fling

You can also change Module Parameters with PowerShell:

Identify Adapters:

PS> Get-VMHostNetworkAdapter -VMHost esx2.virten.lab -Physical -Name vusb* |select Name,Mac 
Name  Mac
----  ---
vusb0 00:23:54:8c:43:45
vusb1 a0:ce:c8:cd:3d:5d

Configure Mapping:

PS> get-vmhost esx2.virten.lab | Get-VMHostModule vmkusb_nic_fling | Set-VMHostModule -Options "vusb0_mac=00:23:54:8c:43:45 vusb1_mac=a0:ce:c8:cd:3d:5d"

Verify Mapping:

PS> get-vmhost esx2.virten.lab | Get-VMHostModule vmkusb_nic_fling |ft -AutoSize
Name             Options
----             -------
vmkusb_nic_fling vusb0_mac=00:23:54:8c:43:45 vusb1_mac=a0:ce:c8:cd:3d:5d

Test Adapter Performance

To avoid problems, check the adapter's performance. My testbed is two Intel NUCs with 3 Network Adapters. Each adapter is connected to a dedicated vSwitch and each switch has a VMkernel adapter.

  • Intel NUC's embedded NIC (Connected to a Switch)
  • 1 Gbit StarTech USB NIC (Connected to a Switch)
  • 2.5 Gbit CableCreation (cross-connected)

The first test measures the latency using ping. I'm doing 50 pings and check the min/avg/max values.

# ping [Address] -c 50
min avg max
Embedded NIC (Connected to a Switch) 0,158 0,189 0,234
1 Gbit/s USB (Connected to a Switch) 0,399 0,491 0,575
2.5 Gbit/s USB (Cross-connected) 0,134 0,206 0,549

The second test is a bandwidth test using iperf3, which is available on ESXi. To use iperf3, you have to make a copy of the binary:

# cp /usr/lib/vmware/vsan/bin/iperf3 /usr/lib/vmware/vsan/bin/iperf3.copy

Start the server on the first ESXi:

# /usr/lib/vmware/vsan/bin/iperf3.copy -s

Start the client on the other ESXi Host. I'm running the test for 5 minutes (-t 300) with samples every 10 seconds (-i 10), to get a reliable result:

# /usr/lib/vmware/vsan/bin/iperf3.copy -i 10 -t 300 -c [SERVER-IP]
Bandwidth
Embedded NIC (Connected to a Switch) 931 Mbits/sec
1 Gbit/s USB (Connected to a Switch) 814 Mbits/sec
2.5 Gbit/s USB (Cross-connected) 1.97 Gbits/sec

For the last test, I'm using vMotion to migrate a virtual machine. The virtual machine to migrate was the vCenter Server with 12 GB Memory.

Bandwidth Time
Embedded NIC (Connected to a Switch) 112 MB/s 124 seconds
1 Gbit/s USB (Connected to a Switch) 99 MB/s 138 seconds
2.5 Gbit/s USB (Cross-connected) 230 MB/s 57 seconds

Conclusion: As you can see from the results, USB Network adapters are slightly slower in bandwidth and latency, but you can get good results with cross-connected 2.5 Gbit adapters, which might be interesting for a small 2-node vSAN deployment.

Check Adapter Compatibility

Not all USB adapters are compatible with the Fling. Make sure to check the Requirements page for a full list of compatible chipsets. The following adapters are proven to work:

Speed Price
Anker AK-A7610011 1GbE $18 USD
StarTech USB31000S 1GbE $20 USD
CableCreation CD0673 2.5GbE $25 USD
CLUB 3D CAC-1420 2.5GbE $45 USD
Cable Matters 201090-N 2.5GbE $50 USD
QNAP QNA-UC5G1T 5GbE $80 USD
StarTech US5GA30 5GbE $110 USD

Additional Tip: Adapters are available with USB-A or USB-C plugs. If you need the other type, just get a USB-A to USB-C or USB-C to USB-A adapter.

Switch options with Multi-Gigabit support

When you want to connect more than 2 systems you need a Multi-Gigabit capable network switch. As this technology is not very common, there are only a few options available at the moment.

MikroTik CRS305-1G-4S+IN ($130 USD)
4 Port switch with 2.5GBASE-T, 5GBASE-T, and 10GBASE-T support. Ports are in SFP+ format, so you additionally need an SFP (MikroTik S+RJ10) which costs $60 USD per port.

MikroTik CRS309-1G-8S+IN ($260 USD)
8 Port switch with 2.5GBASE-T, 5GBASE-T, and 10GBASE-T support. Ports are in SFP+ format, so you additionally need an SFP (MikroTik S+RJ10) which costs $60 USD per port.

Netgear MS510TX ($260 USD)
10 Port Switch with 1x up to 10G, 2x up to 5G, 2x up to 2.5G, 4x 1G, and 1x SFP+

TRENDnet TEG-30102WS ($450 USD)
10 Port 2.5GBASE-T Switch.

Consider Cross-Connect for 2-Node setups

When you only want to connect 2 systems, you can cross-connect adapters. This is very helpful when you want to accelerate vSAN or vMotion with Multi-Gigabit adapters in a 2-node environment. You don't need special cables to cross-connect.

Double-Check MTU Size when enabling Jumbo Frames

When you change the MTU size on a Virtual Switch, the configuration is automatically pushed to all connected physical network interfaces. Make sure to verify that the NIC has actually been configured to the correct MTU as when the adapters don't support the configured MTU, it will silently fail.

In that example, the vusb0 adapter has been configured to 1600 which is fine and sufficient for an NSX-T overlay network:

[root@esx4:~] esxcfg-nics -l
Name    PCI          Driver      Link Speed      Duplex MAC Address       MTU    Description
vmnic0  0000:00:1f.6 ne1000      Up   1000Mbps   Full   00:1f:c6:9c:47:13 1500   Intel Corporation Ethernet Connection (2) I219-LM
vusb0   Pseudo       uether      Up   1000Mbps   Full   00:24:9b:1a:bd:18 1600   ASIX Elec. Corp. AX88179
vusb1   Pseudo       uether      Up   1000Mbps   Full   00:24:9b:1a:bd:19 1500   ASIX Elec. Corp. AX88179

If you try to change the MTU size to 9000, you do not see any error messages on the dvSwitch, but the vmkernel.log reveals that the MTU could not be set:

2020-07-19T16:10:42.344Z cpu6:524356)WARNING: vmkusb: Set MTU 9000 is not supported: Failure
2020-07-19T16:10:42.344Z cpu6:524356)WARNING: Uplink: 16632: Failed to set MTU to 9000 on vusb0

You also want to verify that your configuration works as expected. You can check the MTU with ping, using a size of MTU minus 28 (8 Bytes for the ICMP-Header and 20 Bytes for the IP-Header). With a configured MTU of 1600, the size to verify is 1572.

[root@esx2:~] vmkping ++netstack=vxlan 192.168.225.10 -d -s 1572 -I vmk10
PING 192.168.225.10 (192.168.225.10): 1572 data bytes
1580 bytes from 192.168.225.10: icmp_seq=0 ttl=64 time=0.680 ms

[root@esx2:~] vmkping ++netstack=vxlan 192.168.225.10 -d -s 1573 -I vmk10
PING 192.168.225.10 (192.168.225.10): 1573 data bytes
sendto() failed (Message too long)
  1. I have the RTL8153 USB NIC running with fling driver on NUC7i7DNHE host, works fine with 1GB.
    However, I am not able to passthrough that to a VM. Is it passthrough capable?
    At the same time I can passthrough the onboard Intel l219-LM NIC.

    • Passthrough is for PCI VMNICs only. That does not work with USB devices.

      If you want to passthrough USB devices, you don't need the Fling. You just let the usbarbitrator claim the Usb device (It does that by default automatically) and add it as USB device to the VM. If you have the Fling installed, you cant pass the USB NIC to a virtual machine.

  2. Thanks, currently I am not able to pass it as USB device to my Win10 VM.
    Probably because Fling has reserved it to the host?
    Should I uninstall Fling and try again?

    Background: I wanted to sniff an external switch traffic together with distributed switch traffic on Wireshark on Win10 VM. Initially I wanted to feed external switch traffic via Fling NIC to Distributed switch, but somehow I was not able to get that external fling port connected to my virtual DSwitch. Then the passthrough was the workaround plan.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This site uses Akismet to reduce spam. Learn how your comment data is processed.