TCP tuning for fast and long-distance networks
On a fast link between distant sites, a single TCP connection often moves data far slower than the link allows. The cause is usually not the network but the TCP settings on the two hosts. This section explains why, and how to fix it on Linux, Windows and macOS.
Why defaults can hold you back
TCP may only send a certain amount of data before it must wait for an acknowledgment. That amount, the window, is limited by the socket buffers on both hosts. On a short path, acknowledgments come back quickly and a modest window keeps the link busy. On a long path, the sender spends most of its time waiting. The result: the same two servers that copy at 1 Gbit/s across a data center may manage only a fraction of that across a continent.
The bandwidth-delay product
To fill a path, the window must be at least the bandwidth multiplied by the round-trip time. This is the bandwidth-delay product (BDP). For 1 Gbit/s and a 50 ms RTT: 1,000,000,000 bits/s × 0.05 s = 50,000,000 bits, which is 6.25 MB. A host whose buffers top out at 4 MB cannot keep that path full with one stream, however clean the path is. Window size and BDP works through more examples.
Autotuning does most of the work
Modern systems size TCP buffers automatically for each connection. ESnet notes that autotuning arrived in Linux 2.6, Windows Vista, Mac OS X 10.5 and FreeBSD 7.0, so you rarely need to set default buffer sizes by hand. What still matters is the maximum that autotuning may grow to: on many systems it is too small for 10 Gbit/s or long paths, and raising it is the main job of TCP tuning.
Congestion control: CUBIC and BBR
The congestion control algorithm decides how fast TCP speeds up and how hard it backs off after loss. CUBIC has been the Linux default since kernel 2.6.19 and is a sound choice for most hosts. BBR, developed at Google and in Linux since 4.9, takes a different approach; ESnet notes that it may reduce the impact of packet loss, though it does not eliminate it. ESnet also warns that the original BBR (v1) competes unfairly with CUBIC on shared links, so test before you switch. On Windows, the algorithm appears as the congestion provider; see the platform pages.
When tuning matters
- Worth it: data transfer nodes, backup and replication servers, research hosts and any machine moving large files over paths with tens of milliseconds of RTT at 1 Gbit/s or more.
- Probably not: laptops, Wi-Fi clients and hosts on home broadband, where the access link is the limit. ESnet specifically advises against applying its high-speed settings to Wi-Fi laptops.
- Not a cure for loss: bigger buffers cannot fix a path that drops packets. Measure first with iperf3 and fix loss before tuning.
Guides in this section
- Window size and BDPHow the TCP window limits throughput, how to calculate the bandwidth-delay product with worked examples, and how RFC 7323 window scaling lifts the 64 KB limit.
- Linux tuningTune Linux for fast, long paths: rmem_max, wmem_max, tcp_rmem and tcp_wmem values from ESnet, a sysctl.d file, fq pacing, CUBIC versus BBR, and what to leave alone.
- Windows and macOSCheck and reset Windows receive window autotuning with netsh and Get-NetTCPSetting, and raise macOS TCP buffer limits with sysctl when a fast Mac needs it.
- MTU and MSSEthernet's 1,500-byte MTU, jumbo frames, IPv4, IPv6 and TCP header sizes, how to calculate MSS, path MTU discovery and black holes, MSS clamping and ping tests.