VMware NSX-T 3.1 Edge Node Sizing

Edge Nodes in NSX-T 3.1 are available as Virtual Machines and Bare Metal Edges. When you deploy a Virtual Edge Node using the embedded deployment function in NSX-T, you can choose between 4 sizes - Small, Medium, Large and Extra Large. In this article, I'm trying to collect information about the different sizing options, what they are intended for and how to resize Edge Nodes.

NSX-T Edge VM Sizing Options

NSX-T Edge VM sizing options with their recommended bandwidth range and service usage as a table.

Description

• NSX Edge Small - Proof-of-concept deployments only.
• NSX Edge Medium - Suitable when only L2 through L4 features such as NAT, routing, L4 firewall, L4 load balancer are required and the total throughput requirement is less than 2 Gbps.
• NSX Edge Large - Suitable when only L2 through L4 features such as NAT, routing, L4 firewall, L4 load balancer are required and the total throughput is 2 ~ 10 Gbps. It is also suitable when L7 load balancer, for example, SSL offload is required.
• NSX Edge Extra Large - Suitable when the total throughput required is multiple Gbps for L7 load balancer and VPN.

Reference: NSX Edge VM System Requirements

NSX-T Bare Metal Edge Sizing

A Bare Metal Edge is typically deployed, where higher performance and faster North-South convergence is desired. The Operating System used for NSX-T Edges 3.1 Edges is Ubuntu 18.04, which is important when you want to purchase Servers. Make sure to use Ubuntu certified hardware.

Reference: NSX Edge Bare Metal Requirements

NSX-T Load Balancer Sizing

When you want to use the NSX-T embedded Load Balancer Service (Not NSX-ALB / AVI), which runs on top of a Tier-1 Gateway, you can select between Small, Medium, Large and X-Large. The LB Service size dictates how many Virtual Servers, Pools and Pool Members are supported.

These limits are Hard Limits. NSX-T prevents you from adding more Virtual Servers with the following error message:

Exceed maximum number of virtual servers on the Load balancer service. The maximum size of virtual servers for SMALL load balancer service form factor is 20, current size of virtual servers is 21. (code 502090)

The limit of Virtual Services per Edge Node is also limited. The following table shows which LB Instances can be deployed with each available Edge Size:

These limits are Hard Limits. NSX-T prevents you from adding more LB Services with the following error message:

There is no available capacity on edge node [NODE Path] to deploy a LARGE load balancer service, the form factor of this edge node is LARGE_VIRTUAL_MACHINE, the existed load balancer services on the edge node are [LIST].

You can get a capacity report using the NSX-T API with either Manager or Policy mode. With Manager Mode, you just need the Edge VM UUID (System > Fabric > Nodes > Edge Transport Nodes > Edge > ID). The report shows the current and remaining capacity:

# curl -k -u admin:password https://[NSX-Manager]/api/v1/loadbalancer/usage-per-node/76d7e438-c901-415a-b3b9-ffb854ba31a3
{
  "form_factor" : "LARGE_VIRTUAL_MACHINE",
  "edge_cluster_id" : "5fa37405-5c9b-47f3-aec8-81f0eb507519",
  "current_credit_number" : 2,
  "remaining_credit_number" : 38,
  "usage_percentage" : 5.0,
  "severity" : "GREEN",
  "current_pool_members" : 9,
  "current_virtual_servers" : 22,
  "current_pools" : 5,
  "current_small_load_balancer_services" : 2,
  "current_medium_load_balancer_services" : 0,
  "current_large_load_balancer_services" : 0,
  "current_xlarge_load_balancer_services" : 0,
  "remaining_small_load_balancer_services" : 38,
  "remaining_medium_load_balancer_services" : 3,
  "remaining_large_load_balancer_services" : 0,
  "remaining_xlarge_load_balancer_services" : 0,
  "remaining_pool_members" : 7491,
  "type" : "LbEdgeNodeUsage",
  "node_id" : "76d7e438-c901-415a-b3b9-ffb854ba31a3"
}

With Policy Mode, the call is slightly more complex as you have to provide an enforcement point, which is the edge node path including the edge cluster. Example (The UUID used here is the Edge Cluster UUID):

# curl -k -u admin:password https://[NSX-Manager]/policy/api/v1/infra/lb-node-usage?node_path=/infra/sites/default/enforcement-points/default/edge-clusters/5fa37405-5c9b-47f3-aec8-81f0eb507519/edge-nodes/0
{
  "form_factor" : "LARGE_VIRTUAL_MACHINE",
  "edge_cluster_path" : "/infra/sites/default/enforcement-points/default/edge-clusters/5fa37405-5c9b-47f3-aec8-81f0eb507519",
  "current_load_balancer_credits" : 2,
  "load_balancer_credit_capacity" : 40,
  "usage_percentage" : 5.0,
  "severity" : "GREEN",
  "current_pool_member_count" : 9,
  "current_virtual_server_count" : 22,
  "current_pool_count" : 5,
  "pool_member_capacity" : 7500,
  "current_small_load_balancer_count" : 2,
  "current_medium_load_balancer_count" : 0,
  "current_large_load_balancer_count" : 0,
  "current_xlarge_load_balancer_count" : 0,
  "remaining_small_load_balancer_count" : 38,
  "remaining_medium_load_balancer_count" : 3,
  "remaining_large_load_balancer_count" : 0,
  "remaining_xlarge_load_balancer_count" : 0,
  "resource_type" : "LBEdgeNodeUsage",
  "node_path" : "/infra/sites/default/enforcement-points/default/edge-clusters/5fa37405-5c9b-47f3-aec8-81f0eb507519/edge-nodes/0"

The informal from both API calls are the same. You can see the current and remaining LB counter. From the call, you can also learn that NSX-T uses "Credits" to manage the Load Balancer capacity. For each service, the following amount of credits are accounted on the Edge VM:

The available Credits per Edge VM Size is basically the same:

Resize NSX-T Load Balancer Service Size

The Load Balancer Service size can be easily changed after the deployment. Please keep in mind that changing and redeploying the service will close all active sessions. There is also a warning when you try to change the size (Networking > Network Services > Load Balancer > Load Balancers > Edit > Size).

Resize NSX-T Edge VMs

There is no official documentation on how to change the size of Edge VMs. If you strictly follow the documentation, the only valid option is to deploy a new Edge VM with the new size and use the "Replace Edge Cluster Member" function. Procedure:

1. Deploy a new Edge VM (System > Fabric > Nodes > Edge Transport Nodes > Add Edge Node)
2. Place the old Edge VM into Maintenance Mode (System > Fabric > Nodes > Edge Transport Nodes > Mark the Node > Actions > Enter NSX Maintenance Mode)
3. Replace the old Edge VM (System > Fabric > Nodes > Edge Clusters > Mark the Edge Cluster > Actions > Replace Edge Cluster Member)
4. Select the old and new Edge VM and press SAVE

Reference: Replace an NSX Edge Transport Node Using the NSX Manager UI

For non-production environments, you can also shut down the Edge VM and just change the Virtual Machine Size from vCenter. I've done that multiple times and never had any issues. However, I've not found any information that this is a supported procedure. After changing the size, using the lb-node-usage API call explained above, you can see that the Edge VM is indeed able to detect the new Form Factor.