F5 Guardian Unity Partner
Home » Resources » K15831: How the BIG-IP VE system enforces the licensed throughput rate

Published Date: Sep 21, 2015
Updated Date: Nov 30, 2024

Topic

This article explains how the licensed throughput rate is enforced on a BIG-IP Virtual Edition (VE) system.

Description

The BIG-IP VE product license determines the maximum allowed throughput rate. When calculating throughput, the BIG-IP VE system separately accounts for ingress and egress traffic. Additionally, the licensed throughput rate for ingress and egress is enforced separately. For example, if you have a 200 Mbps license, ingress into the Traffic Management Microkernel (TMM) has a limit of 200 Mbps and egress from TMM also has a limit of 200 Mbps.

Ingress packets to be processed by the system are queued in a buffer that is polled frequently. The traffic shaper admits ingress traffic to the rest of the system at the licensed rate from this buffer. This is sometimes enough to shape the traffic with no further action. If bursts of traffic arrive, the buffer begins to fill, and at a threshold some packets begin to be dropped to signal to sending TCP stacks that congestion is occurring. Normally drops are very light and TCP retransmissions replace them. Light drops are a normal part of the traffic shaper mechanism and are more common with lower-licensed rates and higher numbers of TMM threads. Drops are tied to buffer usage and not to any measurement or logging of ingress traffic in a period, although there is a clear relationship between the two.

For ingress, the system is very similar in behavior to a network switch, accepting traffic over a high-bandwidth link and passing it on through a lower-bandwidth bottleneck link.

Throughput for egress traffic is determined by the number of bytes sent over predefined, sliding time windows. If the number of bytes sent surpasses the limit for a given time window, the rate shaper starts limiting the egress traffic.

The rate shaper throttles throughput to the licensed rate at the packet level, rather than at the connection level. Ingress and egress packets may be dropped during the limiting process and no management is done at the connection level.

Traffic due to config sync, mirroring and monitors is not counted against the licensed bandwidth. For versions from BIG-IP 15.1.6.1, traffic that has VLAN or destination MAC (layer 2 address) not used by BIG-IP will be dropped, and is no longer counted against the licensed bandwidth.

Note: The licensed bandwidth is dynamically distributed among TMM threads several times a second. This allows for a single connection to consume most of the bandwidth on an otherwise quiet BIG-IP VE while still reasonably distributing licensed bandwidth during typically busy periods. Before BIG-IP version 11.6.1, each TMM thread has a fixed, equal share of the bandwidth.

Beginning in BIG-IP 11.4.0, when the throughput rate exceeds 75 percent of the maximum allowed throughput rate licensed, a notification similar to the following example is logged in the /var/log/ltm file:

01010045:5: Bandwidth utilization is 8 Mbps, exceeded 75% of Licensed 10 Mbps

Note: Bandwidth utilization measurements and logging are based on 0.5 second intervals. You should judge the log messages collectively and not individually. You can better determine whether or not the BIG-IP VE is congested or busy over meaningful time periods by looking at throughput graphs that average over longer time periods: 10 seconds for three-hour graphs, 30 seconds for 24-hour graphs, one minute for seven-day graphs, and 10 minutes for 30-day graphs. Bursts of traffic may register in the logs but appear increasingly lower in graphs because they are averaged over increasingly longer periods as the data ages. It’s also normal for bandwidth to reach licensed limit for larger data transfers (relative to the bandwidth), particularly if the licensed limit is a bottleneck bandwidth, and be shaped to that limit with few drops as long as the BIG-IP VE system is not congested.

Note: It’s typically most useful to calculate a packet drop rate between two samples. This example doesn’t record packets through the system (though the tmm_stat table does), but a quick way to approximate packet drop is to divide ingress_shaped_bytes by 1500 to get a probably low estimate of packets that have passed through the shaper. You can express the ingress_drops divided by this packet number as a percentage. Usually, drops greater than about 1% are considered high, although this is an approximation not suited for every situation. Unshaped traffic, such as performing a ConfigSync, does not count against your license.

Note: A correctly sized system may have a non-zero number of drops. A light rate of packet drops is a normal part of traffic shaper operation and is not detrimental to TCP traffic. Heavy dropping, particularly coupled with obviously high use in throughput graphs, likely indicates need for increased bandwidth. It’s not typical, but, for example, if traffic arrives in a substantially synchronized way every minute at the top of the minute, then you may need to over-provision bandwidths to meet requirements for latency and drops at these times.

Supplemental Information