How web caching works: HTTP caching, proxies and CDNs

Updated November 2026

A web cache keeps a copy of a response and reuses it for later requests, so the page loads faster and the server does less work. Browsers, proxies like Squid and content delivery networks (CDNs) all follow the same rules, set out in RFC 9111 (June 2022): a stored response is reused while it is fresh, checked with the server once it is stale, and never stored at all when the server says so.

Private and shared caches

RFC 9111 defines two kinds of cache:

The difference matters for privacy. A response marked Cache-Control: private may be kept by your browser but must not be stored by a shared cache, so one user never receives another user’s page. MDN splits shared caches further into proxy caches, which the site owner does not control, and managed caches such as CDNs and reverse proxies, which the site owner deploys and configures.

Freshness: how long a copy is good

A cache works out a freshness lifetime for each response and compares it with the response’s age. RFC 9111 checks these in order:

  1. s-maxage (shared caches only), then
  2. max-age in Cache-Control, then
  3. the Expires date (the older HTTP/1.0 mechanism), and only if none is present,
  4. a heuristic.

Example: a response with Cache-Control: max-age=604800 stays fresh for 604,800 seconds, which is 7 days (7 × 24 × 3,600). A cache that received it three days ago sends it with an Age: 259200 header and reuses it without asking the server.

Heuristic freshness

If a response has no explicit lifetime but has a Last-Modified date, RFC 9111 lets caches guess, and suggests no more than a fraction of the time since the last change, with 10% as a typical setting. A file last changed 30 days ago could be treated as fresh for 3 days (30 × 10%). Squid’s default refresh_pattern . 0 20% 4320 uses 20% of that interval, capped at 4,320 minutes (3 days). MDN calls heuristic caching a legacy fallback and advises sending an explicit Cache-Control header on every response.

Validation: ETag and 304

A stale copy is not thrown away. The cache asks the server whether it changed, using a conditional request:

GET /app.css HTTP/1.1
Host: www.example.com
If-None-Match: "v42"

HTTP/1.1 304 Not Modified
ETag: "v42"
Cache-Control: max-age=3600

The ETag is an opaque version label chosen by the server. If it still matches, the server answers 304 Not Modified with no body, and the cache refreshes and reuses its copy. The older validator is Last-Modified with If-Modified-Since; MDN recommends sending both, and when both are present If-None-Match takes precedence. A 304 costs one round trip but saves the whole download.

The main Cache-Control directives

DirectiveMeaning
max-age=NFresh for N seconds.
s-maxage=NOverrides max-age for shared caches only.
no-cacheMay be stored, but must be revalidated before every reuse.
no-storeMust not be stored by any cache.
privateShared caches must not store it.
publicMay be stored even where it normally would not be (for example, with an Authorization header).
must-revalidateOnce stale, must not be reused without a successful check.
immutableWill not change while fresh; browsers skip revalidation even on reload.

A common mistake is to use no-cache when you mean no-store, or the reverse. no-cache still allows a fast 304; no-store forces a full download every time and, as MDN notes, also loses features such as the browser’s back/forward cache.

Vary: one URL, several versions

When the server sends different content for the same URL depending on request headers, it lists those headers in Vary. With Vary: Accept-Encoding, a cache keeps the compressed and uncompressed versions apart and only reuses a copy when the new request’s Accept-Encoding matches. Vary: * never matches, so the response is effectively never reused without revalidation. Avoid Vary: User-Agent: there are so many different user-agent strings that the cache will rarely get a hit.

A sensible setup

MDN’s recommended pattern, which works with browsers, proxies and CDNs alike:

Proxies and CDNs

A forward proxy such as Squid sits near the users and caches whatever they fetch over plain HTTP. Since most of the web moved to HTTPS, which a forward proxy only tunnels, this kind of caching saves much less than it once did. A CDN is a managed shared cache that sits near the users but works for the site: the site owner points its domain at the CDN, which holds the TLS certificate and serves copies from many locations. The same RFC 9111 rules apply, and s-maxage lets a site give the CDN a longer lifetime than browsers.

History: cache hierarchies and ICP

In the mid-1990s, when international links were slow and expensive, researchers built hierarchies of cooperating caches. The Internet Cache Protocol (ICP) was first developed in the Harvest research project at the University of Southern California. Duane Wessels and kc claffy of NLANR at UC San Diego documented version 2 in RFC 2186 and RFC 2187 (both September 1997, Informational).

ICP is a small UDP query: one cache asks its neighbors “do you have this URL?” and gets a HIT or MISS back within a second or two. Neighbors are parents (one level up, which fetch on your behalf even on a miss) or siblings (same level, used only for hits). The NSF-funded NLANR cache hierarchy had been running since December 1995; by May 1998 UC San Diego described it as involving about 1,100 caches in 40 countries.

ICP has no authentication and carries no request method, which RFC 2187 lists among its limits. Squid still supports it (icp_port 3130 and the cache_peer directive), along with the newer HTCP, but most sites today run a single proxy or rely on CDNs. More background is in our NLANR history page.

Sources