NSX-T

Direct Org Network to TKC Network Communication in Cloud Director 10.2

Since VMware has introduced vSphere with Tanzu support in VMware Cloud Director 10.2, I'm struggling to find a proper way to implement a solution that allows customers bidirectional communication between Virtual Machines and Pods. In earlier Kubernetes implementations using Container Service Extension (CSE) "Native Cluster", workers and the control plane were directly placed in Organization networks. Communication between Pods and Virtual Machines was quite easy, even if they were placed in different subnets because they could be routed through the Tier1 Gateway.

With Tanzu meeting VMware Cloud Director, Kubernetes Clusters have their own Tier1 Gateway. While it would be technically possible to implement routing between Tanzu and VCD Tier1s through Tier0, the typical Cloud Director Org Network is hidden behind a NAT. There is just no way to prevent overlapping networks when advertising Tier1 Routers to the upstream Tier0. The following diagram shows the VCD networking with Tanzu enabled.

With Cloud Director 10.2.2, VMware further optimized the implementation by automatically setting up Firewall Rules on the TKC Tier1 to only allow the tenants Org Networks to access Kubernetes services. They also published a guide on how customers could NAT their public IP addresses to TKC Ingress addressed to make them accessible from the Internet. The method is described here (see Publish Kubernetes Services using VCD Org Networks). Unfortunately, the need to communicate from Pods to Virtual Machines in VCD seems still not to be in VMware's scope.

While developing a decent solution by using Kubernetes Endpoints, I came up with a questionable workaround. While I highly doubt that these methods are supported and useful in production, I still want to share them, to show what actually could be possible.

Read More »Direct Org Network to TKC Network Communication in Cloud Director 10.2

Client VPN with WireGuard in VMware Cloud Director backed by NSX-T

When you are running an NSX-T backed VMware Cloud Director, your customers might have the requirement to implement a remote access VPN solution. While Client VPN was possible with NSX-v, it is completely missing in NSX-T. This article explains how a customer can implement a Client VPN solution based on WireGuard. The implementation is fully in the hand of the customer and does not need any VCD modification or Service Provider interaction.

Wireguard is not part of VMware Cloud Director but can be installed easily on a Virtual Machine in the tenant's network. The following diagram shows the environment that is used throughout the article.

Read More »Client VPN with WireGuard in VMware Cloud Director backed by NSX-T

Filter specific Domains (FQDN) with NSX-T Distributed Firewall

This article explains how to set up Firewall Rules in NSX-T that allow users to only access specific domains. In many high-security environments, outgoing traffic is filtered using a firewall. When you want to access an external service, you usually create IP-based firewall rules. In some cases, you don't know which IP addresses hide behind a domain. This is where domain filters come in handy.

While this feature has been available in NSX-T for a while, it was limited to a predefined set of domains. With the Release of NSX-T 3.1, you can finally define your own FQDN lists.

In this example, I'm going to set up NSX-T Distributed Firewall to only allow access to www.virten.net and reject all other domains.

Read More »Filter specific Domains (FQDN) with NSX-T Distributed Firewall

Deploy NSX-T Edge VM SSH Keys with Ansible

While working with NSX-T, there are many reasons to access edge appliances using SSH. Most troubleshooting options are only available using nsxcli on the appliance itself. During the deployment, each appliance has 3 user account: root, admin, and audit. Alle Accounts are configured with password-based authentication. In a previous article, I've already described how to deploy SSH Keys using nsxcli, which allows a secure and comfortable authentication method. In this article, I'm explaining how to use ansible to deploy SSH public keys to NSX-T Edges. This option allows you to easily manage keys on a large platform.

Read More »Deploy NSX-T Edge VM SSH Keys with Ansible

Error when connecting Virtual Machine to NSX-T Segments

When you try to connect an NSX-T based Segment to a virtual machine, the task fails with the following error message:

Reconfigure virtual machine - An error occurred during host configuration

In the nsx logfile on the ESXi host where the VM is located, the following error is displayed:

/var/log/nsx-syslog.log
2021-03-13T19:00:36Z nsx-opsagent[527252]: NSX 527252 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="527596" level="ERROR" errorCode="MPA44211"] [PortOp] Failed to create port 780b915d-1479-4eed-8e29-2364d9563f95 with VIF f3f605f2-38a1-4263-bbbd-81b189077f69 because DVS id is not found by transport-zone id 1b3a2f36-bfd1-443e-a0f6-4de01abc963e
2021-03-13T19:00:36Z nsx-opsagent[527252]: NSX 527252 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="527596" level="ERROR" errorCode="MPA42001"] [CreateLocalDvPort] createPort(uuid=780b915d-1479-4eed-8e29-2364d9563f95, zone=1b3a2f36-bfd1-443e-a0f6-4de01abc963e) failed: Failed to create port 780b915d-1479-4eed-8e29-2364d9563f95 with VIF f3f605f2-38a1-4263-bbbd-81b189077f69 because DVS id is not found by transport-zone id 1b3a2f36-bfd1-443e-a0f6-4de01abc963e

 

Read More »Error when connecting Virtual Machine to NSX-T Segments

How to Backup and Restore NSX-T

NSX-T is a critical infrastructure component and it is crucial to have a working backup and restore plan. With complex products, the backup and restore strategy gets more complicated. When working with Virtual Machines, the backup is usually done with VMware Snapshots, which is super convenient. Unfortunately, with the complexity of NSX-T which has many components like clustered Managers, Transport Nodes, and ESXi Kernel Modules, you can't use snapshots as a backup strategy.

This article provides an overview of how to backup NSX-T, and how the restore is done properly.

Read More »How to Backup and Restore NSX-T

Heads Up: NAT Configuration Changed in Cloud Director 10.2

With the release of Cloud Director 10.2, a major change to the NSX-T based NAT configuration has been implemented. The change affects how you set up DNAT and has caused some confusion after the upgrade.

In previous versions, the Application Profile (eg. SSH, HTTP, or HTTPS) defined the external and internal port. With the optional "Internal Port" setting it was possible to configure a custom internal port.

With Cloud Director 10.2, the Application profile defines the internal port only. If you do not fill in the "External Port" configuration, which is exactly in the same position as the "Internal Port" setting on previous versions, it translates ALL external ports to the configured Application. This is something you absolutely do not want to have and I've seen a lot of false configured NATs since Cloud Director 10.2.

Read More »Heads Up: NAT Configuration Changed in Cloud Director 10.2

NSX-T: Client 'admin' exceeded request rate of 100 per second.

NSX-T has a default API rate limit of 100 requests per second, per client. This limit is sometimes already triggered when you are using the GUI with multiple people using the admin account. If you are using the API to get status information or configure your platform, you very likely know the error. When you exceed the limit, the following message is displayed.

Client 'admin' exceeded request rate of 100 per second.

This article shows a couple of methods to mitigate the limit.

Read More »NSX-T: Client 'admin' exceeded request rate of 100 per second.

NSX-T: How to create a Principal Identity

In NSX-T, you can't create local users. Except for the three default users admin, root, and audit, you have to connect to an LDAP server or integrate NSX-T with VMware Identity Manager to authenticate with additional users. Additionally to normal users, NSX-T has the concept of Principal Identities, which are certificate-based users that can create objects that only the user itself can modify or delete.

This article explains how to create and work with a principal identity.

Read More »NSX-T: How to create a Principal Identity