Passive measurement: capturing and analyzing traffic
Passive measurement observes traffic that is already on the network instead of sending test packets. You either record the packets themselves (with tcpdump or Wireshark) or let routers and switches summarize them as flow records (NetFlow, IPFIX, sFlow). It shows what users actually experience, but it also sees their data, so it comes with privacy and legal duties.
Packets or flows?
| Packet capture | Flow records | |
|---|---|---|
| What you get | Every packet, headers and (optionally) payload | One summary per conversation: addresses, ports, protocol, bytes, packets, times |
| Data volume | Large: a busy 1 Gbit/s link can fill a disk in hours | Small: a fraction of the traffic |
| Best for | Debugging one problem in detail | Who talks to whom, traffic mix, trends, anomalies |
| Where | On a host, a mirror (SPAN) port or a network tap | Exported by routers and switches to a collector |
Capturing with tcpdump
tcpdump reads packets from an interface through the libpcap library. Capturing usually needs root (or equivalent capabilities); reading a saved file does not.
# list interfaces
tcpdump -D
# watch DNS traffic, no name lookups
sudo tcpdump -i eth0 -n port 53
# save 1,000 packets to a file, then read it back
sudo tcpdump -i eth0 -c 1000 -w dns.pcap port 53
tcpdump -n -r dns.pcap
# ring buffer: 10 files of 100 MB, oldest overwritten
sudo tcpdump -i eth0 -w ring.pcap -C 100 -W 10
Useful details from the manual:
-ssets the snapshot length, the number of bytes kept per packet. The default is 262,144 bytes, which keeps whole packets.-s 128keeps roughly the headers only: less disk and much less private data.-Ccounts in millions of bytes (1,000,000, not 1,048,576), so-C 100 -W 10keeps at most about 1 GB.-i anycaptures on all interfaces at once.-wwrites raw packets in pcap format. tcpdump, Wireshark and most analysis tools read both pcap and the newer pcapng.
Capture filters
Capture filters decide which packets are recorded at all. They use the pcap-filter language (often called BPF syntax), shared by tcpdump and Wireshark’s capture dialog:
host 192.0.2.10 # to or from one host
net 198.51.100.0/24 # one network
tcp port 443 and host 192.0.2.10 # HTTPS with one host
udp portrange 7000-7009 # a range of UDP ports
not port 22 # everything except SSH
tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 # connection starts and ends
Always filter out your own remote session (not port 22) when capturing over SSH, or the capture records itself.
Analyzing in Wireshark
Wireshark decodes most protocols you are likely to meet and is the easiest way to read a capture. It uses display filters, a different language from capture filters, to hide packets that are already recorded:
ip.addr==192.0.2.10
tcp.port in {80, 443, 8080}
dns
A practical routine is to capture on the server with tcpdump (no graphical session needed), copy the file to your workstation, and open it there. For packet and frame sizes on the wire, see packet sizes explained.
Flow data: NetFlow, IPFIX and sFlow
A flow is a set of packets passing an observation point during a time interval that share common properties, typically the same source and destination addresses, ports and protocol. The router keeps a counter per flow and exports a record when the flow ends or times out.
- NetFlow version 9
- Cisco’s format, described in RFC 3954 (October 2004, Informational). Records are template-based, so the collector learns which fields are present, and are usually sent over UDP.
- IPFIX
- The IETF standard built on NetFlow v9: RFC 7011 (September 2013), Internet Standard STD 77. Its version number is 10, one higher than NetFlow v9. It can run over SCTP, TCP or UDP. A device that sends records is an Exporter; one that receives them is a Collector.
- sFlow
- Developed by InMon and described in RFC 3176 (September 2001). Instead of tracking every flow, the switch samples 1 in N packets and also exports interface counters, over UDP. Sampling keeps the load low on fast switches.
Sampling needs arithmetic. With 1-in-1,000 sampling, 2,500 sampled packets from one host stand for about 2,500 × 1,000 = 2.5 million real packets. The estimate is good for large flows and poor for small ones: a 50-packet connection will often not be sampled at all.
Privacy and the law
Packet captures can contain passwords, messages, cookies and personal data, and even headers and flow records show who talked to whom and when. Before you capture:
- Capture only on networks you own or are explicitly authorized to monitor, ideally in writing.
- Collect the minimum: a tight capture filter, a short snaplen, a short time window. The IETF’s privacy guidance (RFC 6973) calls this data minimization.
- Store captures securely and delete them when the job is done.
- Anonymize addresses before you share a trace with anyone else.
Interception is a crime in many places. In the United States, the federal Wiretap Act (18 U.S.C. § 2511) makes intentional interception of electronic communications illegal, with narrow exceptions such as a party’s prior consent or a service provider protecting its own service. Other countries have similar laws plus data-protection rules. When in doubt, ask your organization’s legal team first.
A note on history
Passive measurement on research backbones was one of NLANR’s main activities. Its measurement group at the San Diego Supercomputer Center ran passive monitors at high-performance connection sites and published traces for researchers; its 2002–2006 NSF award planned to extend passive measurement to study packet flows through routers. See the NLANR history page.
Sources
- tcpdump(1) manual page, tcpdump.org
- pcap-filter(7) manual page, tcpdump.org
- Filtering packets while capturing, Wireshark User’s Guide
- Building display filter expressions, Wireshark User’s Guide
- RFC 3954: Cisco Systems NetFlow Services Export Version 9, IETF
- RFC 7011: IP Flow Information Export (IPFIX) protocol, IETF
- RFC 3176: InMon Corporation’s sFlow, IETF
- RFC 6973: Privacy Considerations for Internet Protocols, IAB
- 18 U.S.C. § 2511, Legal Information Institute, Cornell Law School
- Award 0129677: NLANR Measurement and Network Analysis group, National Science Foundation