NSX-T: Client 'admin' exceeded request rate of 100 per second.

NSX-T has a default API rate limit of 100 requests per second, per client. This limit is sometimes already triggered when you are using the GUI with multiple people using the admin account. If you are using the API to get status information or configure your platform, you very likely know the error. When you exceed the limit, the following message is displayed.

Client 'admin' exceeded request rate of 100 per second.

This article shows a couple of methods to mitigate the limit.

General Information
The following limits are configured by default. To view the limit, log in to an NSX-T Manager with SSH using the admin user and run the following command: get service http

nsx1> get service http
Sun Nov 29 2020 UTC 09:09:45.147
Service name:                     http
Service state:                    running
Logging level:                    info
Session timeout:                  0
Connection timeout:               30
Client API rate limit:            100 requests/sec
Client API concurrency limit:     40 connections
Global API concurrency limit:     199 connections
Redirect host:                    (not configured)
Basic authentication:             enabled
Cookie-based authentication:      enabled

NSX-T has three settings to limit API requests:

Client API rate limit - A per-client rate limit, in requests per second. If a client makes more requests than this limit in one second, the API server will refuse to service the API request and will return an HTTP 429 Too Many Requests Error.

Client API concurrency limit - This is the maximum number of outstanding requests that a client can have. When this limit is exceeded, the server returns a 429 Too Many Requests error to the client.

Global API concurrency limit - An overall maximum number of concurrent requests. This is the maximum number of API requests that can be in process on the server. If the server is at this limit, additional requests will be refused and the HTTP error 503 Service Unavailable will be returned to the client.

While it is possible to configure these rate limits, it is not recommended. Instead, you should try to design your API client to gracefully deal with situations where limits are exceeded. If that is not an option, you change the limits with the following command:

nsx1> set service http client-api-concurrency-limit [number]
nsx1> set service http client-api-rate-limit [number]
nsx1> set service http global-api-concurrency-limit [number]

You can also completely disable limits:

nsx1> set service http client-api-rate-limit 0

Note: When you change a configuration, the http service reloads which results in a short UI downtime.

You can also view the occurrence of 429 errors in the http service access log. The log file is located at /var/log/proxy/localhost_access_log.txt (Login to the Manager using SSH as root)

nsx1:/# grep 429 /var/log/proxy/localhost_access_log.txt
2020-11-29T13:49:30.469Z - "GET /api/v1/upgrade/functional-state HTTP/1.1" 429 122 0 0
2020-11-29T13:49:30.470Z - "GET /api/v1/telemetry/agreement HTTP/1.1" 429 122 2 0
2020-11-29T13:49:30.472Z - "GET /api/v1/cluster/backups/status HTTP/1.1" 429 122 1 1
2020-11-29T13:49:30.472Z - "GET /api/v1/cluster/restore/status HTTP/1.1" 429 122 1 1
2020-11-29T13:49:30.473Z - "GET /api/v1/licenses HTTP/1.1" 429 122 1 0

Additional tips to mitigate the API rate limit

1. Use LDAP or VMware Identity Manager to enable user-based authentication. Each user a per user limit.
2. Use Principal Identities when working with the API.
3. Build throttling into the client code.
4. Use API batching (POST /api/v1/batch or POST /policy/api/v1/batch) to reduce the number of API calls.