Edge Nodes in NSX-T 3.1 are available as Virtual Machines and Bare Metal Edges. When you deploy a Virtual Edge Node using the embedded deployment function in NSX-T, you can choose between 4 sizes - Small, Medium, Large and Extra Large. In this article, I'm trying to collect information about the different sizing options, what they are intended for and how to resize Edge Nodes.
NSX-T Edge VM Sizing Options
NSX-T Edge VM sizing options with their recommended bandwidth range and service usage as a table.
| Memory | CPU | Disk | Bandwidth | NAT/Firewall | L4 LB | L7 LB | Multi-Gbps L7 LB / VPN |
|
| Small | 4 GB | 2 | 200 GB | < 2Gbps | PoC Only | |||
| Medium | 8 GB | 4 | 200 GB | 2 Gbps | YES | YES | NO | NO |
| Large | 32 GB | 8 | 200 GB | 2-10 Gbps | YES | YES | YES | NO |
| X-Large | 64 GB | 16 | 200 GB | > 10 Gbps | YES | YES | YES | YES |
Description
- NSX Edge Small - Proof-of-concept deployments only.
- NSX Edge Medium - Suitable when only L2 through L4 features such as NAT, routing, L4 firewall, L4 load balancer are required and the total throughput requirement is less than 2 Gbps.
- NSX Edge Large - Suitable when only L2 through L4 features such as NAT, routing, L4 firewall, L4 load balancer are required and the total throughput is 2 ~ 10 Gbps. It is also suitable when L7 load balancer, for example, SSL offload is required.
- NSX Edge Extra Large - Suitable when the total throughput required is multiple Gbps for L7 load balancer and VPN.
Reference: NSX Edge VM System Requirements
NSX-T Bare Metal Edge Sizing
A Bare Metal Edge is typically deployed, where higher performance and faster North-South convergence is desired. The Operating System used for NSX-T Edges 3.1 Edges is Ubuntu 18.04, which is important when you want to purchase Servers. Make sure to use Ubuntu certified hardware.
| Memory | CPU | Disk | |
| Minimum Requirements | 32 GB | 8 | 200 GB |
| Recommended | 256 GB | 24 | 200 GB |
Reference: NSX Edge Bare Metal Requirements
NSX-T Load Balancer Sizing
When you want to use the NSX-T embedded Load Balancer Service (Not NSX-ALB / AVI), which runs on top of a Tier-1 Gateway, you can select between Small, Medium, Large and X-Large. The LB Service size dictates how many Virtual Servers, Pools and Pool Members are supported.
| Virtual Servers | Pools | Pool Members | |
| Small | 20 | 60 | 300 |
| Medium | 100 | 300 | 2000 |
| Large | 1000 | 3000 | 7500 |
| X-Large | 2000 | 4000 | 10000 |
These limits are Hard Limits. NSX-T prevents you from adding more Virtual Servers with the following error message:
Exceed maximum number of virtual servers on the Load balancer service. The maximum size of virtual servers for SMALL load balancer service form factor is 20, current size of virtual servers is 21. (code 502090)
The limit of Virtual Services per Edge Node is also limited. The following table shows which LB Instances can be deployed with each available Edge Size:
| Pool Members | Small LB | Medium LB | Large LB | X-Large LB | |
| Small | 0 | 1 | 0 | 0 | 0 |
| Medium | 2000 | 10 | 1 | 0 | 0 |
| Large | 7500 | 40 | 4 | 1 | 0 |
| X-Large | 10000 | 80 | 8 | 2 | 1 |
| Bare-Metal | 30000 | 750 | 75 | 18 | 9 |
These limits are Hard Limits. NSX-T prevents you from adding more LB Services with the following error message:
There is no available capacity on edge node [NODE Path] to deploy a LARGE load balancer service, the form factor of this edge node is LARGE_VIRTUAL_MACHINE, the existed load balancer services on the edge node are [LIST].
You can get a capacity report using the NSX-T API with either Manager or Policy mode. With Manager Mode, you just need the Edge VM UUID (System > Fabric > Nodes > Edge Transport Nodes > Edge > ID). The report shows the current and remaining capacity:
# curl -k -u admin:password https://[NSX-Manager]/api/v1/loadbalancer/usage-per-node/76d7e438-c901-415a-b3b9-ffb854ba31a3
{
"form_factor" : "LARGE_VIRTUAL_MACHINE",
"edge_cluster_id" : "5fa37405-5c9b-47f3-aec8-81f0eb507519",
"current_credit_number" : 2,
"remaining_credit_number" : 38,
"usage_percentage" : 5.0,
"severity" : "GREEN",
"current_pool_members" : 9,
"current_virtual_servers" : 22,
"current_pools" : 5,
"current_small_load_balancer_services" : 2,
"current_medium_load_balancer_services" : 0,
"current_large_load_balancer_services" : 0,
"current_xlarge_load_balancer_services" : 0,
"remaining_small_load_balancer_services" : 38,
"remaining_medium_load_balancer_services" : 3,
"remaining_large_load_balancer_services" : 0,
"remaining_xlarge_load_balancer_services" : 0,
"remaining_pool_members" : 7491,
"type" : "LbEdgeNodeUsage",
"node_id" : "76d7e438-c901-415a-b3b9-ffb854ba31a3"
}
With Policy Mode, the call is slightly more complex as you have to provide an enforcement point, which is the edge node path including the edge cluster. Example (The UUID used here is the Edge Cluster UUID):
# curl -k -u admin:password https://[NSX-Manager]/policy/api/v1/infra/lb-node-usage?node_path=/infra/sites/default/enforcement-points/default/edge-clusters/5fa37405-5c9b-47f3-aec8-81f0eb507519/edge-nodes/0
{
"form_factor" : "LARGE_VIRTUAL_MACHINE",
"edge_cluster_path" : "/infra/sites/default/enforcement-points/default/edge-clusters/5fa37405-5c9b-47f3-aec8-81f0eb507519",
"current_load_balancer_credits" : 2,
"load_balancer_credit_capacity" : 40,
"usage_percentage" : 5.0,
"severity" : "GREEN",
"current_pool_member_count" : 9,
"current_virtual_server_count" : 22,
"current_pool_count" : 5,
"pool_member_capacity" : 7500,
"current_small_load_balancer_count" : 2,
"current_medium_load_balancer_count" : 0,
"current_large_load_balancer_count" : 0,
"current_xlarge_load_balancer_count" : 0,
"remaining_small_load_balancer_count" : 38,
"remaining_medium_load_balancer_count" : 3,
"remaining_large_load_balancer_count" : 0,
"remaining_xlarge_load_balancer_count" : 0,
"resource_type" : "LBEdgeNodeUsage",
"node_path" : "/infra/sites/default/enforcement-points/default/edge-clusters/5fa37405-5c9b-47f3-aec8-81f0eb507519/edge-nodes/0"
The informal from both API calls are the same. You can see the current and remaining LB counter. From the call, you can also learn that NSX-T uses "Credits" to manage the Load Balancer capacity. For each service, the following amount of credits are accounted on the Edge VM:
| Small Load Balancer Service | 1 Credit |
| Medium Load Balancer Service | 10 Credits |
| Large Load Balancer Service | 40 Credits |
| X-Large Load Balancer Service | 80 Credits |
The available Credits per Edge VM Size is basically the same:
| Small Edge VM | 1 Credit |
| Medium Edge VM | 10 Credits |
| Large Edge VM | 40 Credits |
| X-Large Edge VM | 80 Credits |
Resize NSX-T Load Balancer Service Size
The Load Balancer Service size can be easily changed after the deployment. Please keep in mind that changing and redeploying the service will close all active sessions. There is also a warning when you try to change the size (Networking > Network Services > Load Balancer > Load Balancers > Edit > Size).
Resize NSX-T Edge VMs
There is no official documentation on how to change the size of Edge VMs. If you strictly follow the documentation, the only valid option is to deploy a new Edge VM with the new size and use the "Replace Edge Cluster Member" function. Procedure:
- Deploy a new Edge VM (System > Fabric > Nodes > Edge Transport Nodes > Add Edge Node)
- Place the old Edge VM into Maintenance Mode (System > Fabric > Nodes > Edge Transport Nodes > Mark the Node > Actions > Enter NSX Maintenance Mode)
- Replace the old Edge VM (System > Fabric > Nodes > Edge Clusters > Mark the Edge Cluster > Actions > Replace Edge Cluster Member)
- Select the old and new Edge VM and press SAVE
Reference: Replace an NSX Edge Transport Node Using the NSX Manager UI
For non-production environments, you can also shut down the Edge VM and just change the Virtual Machine Size from vCenter. I've done that multiple times and never had any issues. However, I've not found any information that this is a supported procedure. After changing the size, using the lb-node-usage API call explained above, you can see that the Edge VM is indeed able to detect the new Form Factor.
This is very helpful.
Do you have also any information about the IPsec throughout and the Edge Nodes Size Dependency?
Hard to tell because it does not solely depend on throughput. The type of traffic (Concurrent Sessions, Packet Size) and of course the cryptographic configuration is also important. According to VMware, you should use the X-Large for "Multiple Gbps VPN". With a single large stream, you can have 3-4 GBps even with a Medium Edge.
GET https://>NSX-MGR>/policy/api/v1/infra/lb-node-usage-summary?include_usages=true
will give you the usage for all edge nodes in one call. If you have several edge clusters,it's listed and sorting might be necessary to have a clear view per EC.