TCP window size and the bandwidth-delay product
A single TCP connection can never move data faster than its window divided by the round-trip time. Work out the bandwidth-delay product of your path, compare it with the window your hosts allow, and you know at once whether TCP settings are what is slowing you down.
What the window is
TCP lets a sender transmit a certain amount of data before it has to stop and wait for an acknowledgment. The receiver advertises how much it can accept: the receive window. The sender also keeps its own limit, the congestion window, which it grows while the path looks clear and cuts after loss. The amount actually in flight is the smaller of the two, and both are bounded by the socket buffers on each host.
Once the sender has a full window in flight, it waits. On a path with a 50 ms round trip, it can send at most one window every 50 ms, whatever the speed of the link.
The formula
Maximum throughput of one TCP connection ≈ window size ÷ round-trip time.
Turned around, the window you need to fill a path is the bandwidth-delay product (BDP):
BDP (bits) = bandwidth (bits per second) × RTT (seconds). Divide by 8 for bytes.
Use the bandwidth of the slowest link on the path and the RTT that ping reports between the two hosts.
Worked BDP examples
| Bandwidth | RTT | Arithmetic | BDP |
|---|---|---|---|
| 100 Mbit/s | 20 ms | 100,000,000 × 0.020 = 2,000,000 bits | 250 KB |
| 1 Gbit/s | 10 ms | 1,000,000,000 × 0.010 = 10,000,000 bits | 1.25 MB |
| 1 Gbit/s | 50 ms | 1,000,000,000 × 0.050 = 50,000,000 bits | 6.25 MB |
| 10 Gbit/s | 50 ms | 10,000,000,000 × 0.050 = 500,000,000 bits | 62.5 MB |
| 10 Gbit/s | 100 ms | 10,000,000,000 × 0.100 = 1,000,000,000 bits | 125 MB |
| 100 Gbit/s | 100 ms | 100,000,000,000 × 0.100 = 10,000,000,000 bits | 1,250 MB |
The last row is larger than the biggest window TCP can use (about 1,074 MB; see below), so a single stream cannot fill a 100 Gbit/s path across 100 ms. That is one reason high-speed transfer tools use several parallel streams.
The other direction: what a window allows
Without window scaling, the TCP header's 16-bit window field allows at most 65,535 bytes. Here is what that means at different distances:
- 10 ms RTT: 65,535 × 8 ÷ 0.010 ≈ 52 Mbit/s
- 50 ms RTT: 65,535 × 8 ÷ 0.050 ≈ 10.5 Mbit/s
- 100 ms RTT: 65,535 × 8 ÷ 0.100 ≈ 5.2 Mbit/s
Microsoft gives the same example in its Windows Server tuning guide: with a fixed 64 KB window, a 1 Gbit/s connection with 10 ms latency reaches only about 51 Mbit/s. A 4 MB buffer limit on a 50 ms path allows about 4,000,000 × 8 ÷ 0.05 = 640 Mbit/s, still short of 1 Gbit/s.
Window scaling (RFC 7323)
To get past 65,535 bytes, TCP uses the Window Scale option, first defined in RFC 1323 and now in RFC 7323 (September 2014). Each side announces a shift count in its SYN segment, and the window values it sends afterwards are multiplied by 2 to the power of that count. Key rules:
- The option is exchanged only in the SYN and SYN-ACK, so the scale is fixed when the connection opens. Both sides must send it for scaling to be used.
- The maximum shift is 14, giving a maximum window of about 1 GiB (65,535 × 16,384 = 1,073,725,440 bytes, since a shift of 14 multiplies by 16,384).
- To advertise a 6.25 MB window you need 6,250,000 ÷ 65,535 ≈ 95 times the base, so a shift of at least 7, which multiplies by 128. For 125 MB: 125,000,000 ÷ 65,535 ≈ 1,907, so a shift of 11, which multiplies by 2,048.
Middleboxes. If a firewall or load balancer strips or rewrites the Window Scale option, connections silently fall back to a 64 KB window. Microsoft's documentation notes that some network devices do not comply with the option. A transfer that is stuck near 64 KB ÷ RTT on every attempt is a strong hint; a packet capture of the SYN packets will confirm it.
From BDP to buffer settings
- Buffers need headroom. On Linux, part of the socket buffer holds kernel bookkeeping, not data, so the buffer must be somewhat larger than the window you want. ESnet suggests that the optimal buffer is about twice the BDP.
- Raise the maximum, not the default. Autotuning grows buffers per connection up to a limit. Set that limit to cover your largest BDP; ESnet's suggested maximums are typically between 32 MB and 128 MB.
- Parallel streams share the load. Four streams each need only a quarter of the BDP. ESnet advises lower per-connection limits if you use parallel streams, to avoid exhausting memory.
- Do not oversize blindly. Huge buffers on a congested path add queueing delay without adding speed.
Platform steps are in Linux tuning and Windows and macOS.
Check it with iperf3
Run a test and compare the Cwnd column with your BDP. If Cwnd stays flat well below the BDP with no retransmissions, a buffer limit is holding you back. If Cwnd keeps collapsing with retransmissions, the problem is loss, and bigger buffers will not help. See Reading results.