ISMA-97 Report

• Backbone Panel
• Measurement Infrastructure Panel
• Overview of CAIDA Tool Taxonomy
• Internet Analysis, Simulation and Modeling
• Routing Analysis Tools
• Open Panel Discussion of Performance Tools
• Flow Characterization / Analysis
• End User Tools / Evaluating ISP Performance
• Panel Discussion on Internet Measurement: Conclusions and Next Steps

ISMA-97 Workshop Report is available at http://www.nlanr.net/ISMA/isma97_report.html.

Mike O'Dell (UUnet) listed three tasks that need Internet measurement:

For ISPs, useful tools will describe macroscopic performance. (Examples using infrastructure data: Tony Bates' CIDR Report and Curtis' Overlap Report.) The ISPs all agree on the need for tools for large-scale simulation of the global Internet routing infrastructure. ISPs are willing to pay for network elements that instrument features such as queue length histograms. Today's network devices are (mostly) manageable, but are not measurable.

Randy Bush (Verio) explained that Verio uses homegrown MRTG-based for doing traffic shaping and related analyses. They are also using a commercial simulation tool, but have not found it to be very satisfactory. As a small ISP, funding internal tool development is not an option. 

Cleveland Mickles (MCI) described current MCI measurements, with tools almost entirely internally developed. MCI would prefer public tools, in part due to support related issues. In particular, MCI would be most interested in tools to:

• parse and graphically display collected data
• measure routing stability/convergence
• aid in security investigations

Henry Kilmer (SprintLink) showed an MRTG-based graphical monitoring tool that shows capacity utilization by color on a web page. This tool assists in analyzing/justifying bandwidth upgrades for SprintLink, and was useful in depicting effects of the bogus route announcements following the AS7007 incident in April. This tool also helps to visually communicate network conditions to management through weekly summary reports. Also of use to SprintLink are the Concord exception and health reports. SprintLink is using NetFlow and Cflowd to develop Autonomous System (AS) matrices for various peering points.

Peter Lothberg (Sprint ICM) is particularly interested in topology changes and their effect on network stabililty. Sprint ICM uses all home grown tools, mostly focused on the tradeoff between bandwidth efficiency and packet loss, a well-known challenge in the face of bursty data. He emphasized the need for more robust simulation tools for routing analysis, including the ability to reflect peer cooperation, although admittedly the latter goal is hindered by the inadequacy of the current Internet Routing Registry (IRR).

Curtis Villamizar (ANS) explained that generating reports is easy with access to raw measurement data. ANS uses MIB II and the DS1/DS3 MIBs, looking particularly at errored seconds, although they admittedly don't map well to percent packet loss. ANS manages OSPF and BGP stability and identifies events in real-time. Difficulties include the differences in enterprise MIBs which frustrate cross-platform data consolidation. Network management machines at POPs monitor latencies between probes pairs. Some data of interest does exist in enterprise MIBs, specifically queue loss and load.

Critical data not readily available include:

• Adjacency traps in OSPF
• Link Quality Monitoring (LQM) over PPP, ATM, or FDDI
• Net-to-Net and AS-to-AS Matrices
• Measurements of length of flows
• Measurements of connection rates
• Means of using statistical traffic samples to trace back things like SYN attacks

Devices that require SNMP sub-agents for management also aggravate monitoring activity.

John Hawkinson (BBN Planet) pointed out that they collect utilization data, but SNMP agents are not designed to count all data. Using NetFlow/Cflowd at certain exchange points, BBN derives AS information and is hoping that Cisco will soon enhance NetFlow to enable aggregation router to aggregation router analysis. John shares the concerns expressed by the other ISPs regarding the absence of publicly available measurement and analysis tools. As one of the older ISPs, and a company with R&D roots, BBN Planet has a long history of developing internal tools to meet its needs.

In general, the consensus of the panel was that public tools are inadequate for ISPs' measurement, simulation, and management needs. Most tools are internally developed and not made public. Statistical packages such as SAS are used; NTP statistics and SNMP data are also gathered routinely. Cascades' Openview data is also viewed as helpful, and Netcol is used by some ISPs to correlate events. Large ISPs monitor the DMZs of their peers. Several are beginning to use NetFlow and Cflowd to obtain AS matrices, yet this code currently only works with Cisco routers and has limitations (see discussion below).

Panelists were critical of current end-to-end measurement tools, most of which do not use consistent metrics, nor do they provide sufficient information to enable users to accurately interpret resulting data. AS traffic matrices and VC statistics, and queue length statistics are the most important data missing today. And ISPs concurred that there is no acceptable tool for accurately measuring packet loss across ISP clouds.

One participant mentioned a technique they use to assess packet loss across paths and test inter-ISP performance, using a Java script on a PC containing two modem ports, each connected to a different ISP and each sending/receiving packets. This approach ensures important consistencies in test h/w, s/w, connections, etc.

The backbone panelists agreed that the industry is still at the point where virtually any ISP investment in tools has positive return, and that there is a great need for stronger cooperation and collaboration in this arena to continue evolution of network measurement/management tools. In. particular, ISPs would benefit most from macro-level tools that enhance understanding IP traffic/routing on large networks, including simulation tools that can facilitate the study of real BGP data, such as the vast amounts of historic BGP data warehoused by Merit.

Guy Almes (ANS) described the Surveyor measurement infrastructure project, a joint initiative of Advanced Network & Services (ANS) and the Common Solutions Group (CSG) which emphasizes early implementation of metrics developed within the IP Performance Metrics (IPPM) working group of the Internet Engineering Task Force (IETF). The CSG is a collaboration among 23 participating universities to jointly address challenges relating to networking. ANS is working with CSG to deploy measurement boxes (Surveyor Tools) at each university. Several traffic measurements will be taken from these sites, including estimates of one-way delay and packet loss.  The boxes are PCs (running FreeBSG), with global positioning system (GPS) antennae added in order to achieve accurate (+/- 50 microsecond) time stamps. An Oracle-based database is being developed to analyze data, with a web server serving as a front end permitting access to data by participants. Currently there are machines installed at the University of Chicago and at Carnegie Mellon University. By the end of May, additional units will be operational at Pennsylvania State University, the University of Michigan, and George Washington University.

Bill Cerveny (ANS) provided details on the CSG boxes' configuration and deployment. Problems and hurdles include antenna placement, cable length (150 ft max), deployment lead time, SSH portability, and support for ATM (expected in 1997). ANS/CSG also want to introduce the measurement machines at or near exchange points. However, there are significant scaling problems involved with this effort. With 25 sites (excluding exchanges) there would be 625 paths potentially generating about 703 GB of data annually -- 625 records per second would be recorded. Guy and Bill presented several graphs of preliminary delay statistics generated using MS Excel. The power point presentation on the CSG initiative is available at http://www.advanced.org/csg-ippm/isma97.

Jamshid Mahdavi and Matt Mathis (PSC) provided an overview of the National Internet Measurement Infrastructure (NIMI). The goals for NIMI are to provide a scalable measurement strategy along the lines of an industrial strength Network Probe Daemon (NPD). Timing and scaling are inherent problems with any end-to-end path performance measurements; NIMI is seeking better packet filtering techniques to more precisely timestamp packets on the wire. Data collection and storage will be distributed. NIMI itself is not focused on hardware and might be run on some of the CGS boxes. Like CGS, they will have to address issues relating to permissions and privacy.

NIMI is being designed for scalability, e.g., it could be deployed on every campus/subnet and, theoretically, scaled to a million machines globally. Designed to optimize low bandwidth and provide high yield measurements, NIMI will include challenges of: developing a measurement archive; discovering the topology of the NIMI mesh and the underlying Internet (required for path decomposition and statistical decomposition of end-to-end tests); dealing with permissions and data scope of the distributed data; and using caching to relate user queries to summaries of measurement results. NIMI is not a large- scale hardware deployment, it is not centrally managed, it is not associated with the IETF's IPPM working group, and it is not for profit.

The NIMI presentation is available at http://www.nlanr.net/ISMA/isma97_mahdavi. A related paper by Vern Paxson on creating a national measurement infrastructure is available at http://www.nlanr.net/ISMA/isma97_paxson.html.

Mitchell Baltuch (Unidata Program Center/NCAR) spoke about the Java-based Internet Data Distribution System (IDD) that distributes meteorological (weather) data throughout the U.S. Unidata moves large data streams 30-35 gigabytes daily, in real-time to approximately 120 sites. Unidata/NCAR is not yet utilizing multicast for this effort due to concerns about its reliability.

IDD measures delay from the seven originating sites to the end-sites. Specific points of interest include latency and end-to-end performance (less concerned with packet loss). All sites report hourly on their maximum and average latency. If the latency is greater than 3600 seconds then data has been lost. Routing, availability of links, etc. are not critical measurement parameters. Unidata wrote a perl wrapper for traceroute and ping, called netcheck. Run hourly as a cron job, netcheck permits Unidata to track problems related to data delivery at each leg of the route, from the campus net through the backbone and across the Net. Mitch noted that IDD's data illustrates frequent performance problems at the public exchange points. Other diagnosis and measurement tools developed by Unidata (e.g., sysstat for the iostat, vmstat, etc.) are available publically at http://www.unidata.ucar.edu/. Ideally, Unidata would like to be able to measure without cooperation of the intermediate routers, i.e., incorporate both end-to-end and hop-to-hop measurements into its monitoring. This presentation is available at http://www.unidata.ucar.edu/staff/mitch/isma97.ps

ISP representatives expressed concern that the research sector is not sufficiently articulating what problems they address with their measurement efforts. ISPs also indicated that they would like to see meaningful measurements that can provide customers a quantitative way to define/evaluate their received service. In addition to defining terms like IP cloud and path, we need standardized metrics for measuring them. Panelists explained that the IPPM is tasked with defining Internet metrics and that these requirements are beyond the scopes of the projects described during this session.

The issue of an Internet 611 help line was introduced, as were suggestions that end-users should have tools that would inform them when they should be contacting their ISP. Participants also discussed the varying methods of processing data, e.g., Unidata has centralized processing of information, NIMI will encourage distributed acquisition and processing. CSG is still grappling with the best approach to data processing given the daunting scalability issues.

ISPs also explained that measurements that rely the IP address of an exchange point often result in faulty data. Routers at a DMZ are configured to facilitate throughput, not to respond to direct queries (or pings). In order to get reliable measurement results, it is critical that equipment be placed near an exchange point, not at the exchange. According to participants, certain existing measurement methodologies and tools that collect data from machines at the Network Access Points (NAPs) should be evaluated to ensure that their data accurately reflect those metrics that they are attempting to measure, e.g., end-to-end throughput, packet loss, etc.  

There was also discussion about what these campus-to-campus measurement efforts really communicate about user-perceived performance. Several ISPs expressed the desire for metrics reflecting performance straight to the end user's box.

k claffy and Jon Kay provided an overview of the CAIDA Tool Taxonomy, see http://www.nlanr.net/Caidants/meastools.html. The taxonomy is a living site, maintaining up-to-date summaries of existing and emerging Internet measurement and analysis tools and initiatives. The taxonomy covers many categories of tools, one dimension of which is whether they are end-point centric (e.g., end-to-end measurements of latency or routing stability) or network-centric (e.g., flow characterization, topology mapping, capacity planning). The largest gaps appear to be in statistically respectable user-oriented measurements and in infrastructure (big picture) visualizations/techniques for capacity engineering. This tool taxonomy effort is sponsored through assistance from the NSF and CISCO.

kc also summarized the goals of the Cooperative Association for Internet Data Analysis (CAIDA) - see http://www.caida.org.

Current CAIDA initiatives include:

• Collaborating on the development of measurement tools, e.g., OC3mon/Coral;
• Developing visualizations of Internet traffic flows, including real-time traceroutes traffic at http://www.caida.org/Pathchar;
• Hosting neutral, networked collaborative environments (a text based object-oriented multi-user domains or MOOs) provided for use in real-time network management, operation, analysis, and research discussions, examples include a tool developer MOO used in designing Cflowd and related code, the SquidMOO used in the development of public software used in the NLANR global caching hierarchy; and the Merit-CAIDA collaboration relating to inter-provider notifications known as the IPNmoo);
• Fostering information exchange between segments of the Internet community and among Internet providers through hosting events like ISMA-97 and the BORG.

 Discussion:

Identification and evaluation of Internet measurement and analysis tools (along the lines of what CAIDA is doing) is a fundamental first step in this sector. In addition to CAIDA's current efforts, the Trans-European Research and Education Networking Association's (TERENA) newly-formed measurement task force is also assessing available tools as part of a process of defining a tool set for deployment across Europe's R&E networks. More information on this initiative is available at http://www.terena.nl/task-forces/tf-etm/.

In terms of tool development, CAIDA is particularly interested in macro-level tools to facilitate engineering analysis and planning by ISPs. kc recognized the legitimacy of the complaints from ISP's that academic studies of traffic have historically been of little use to ISPs, and even commercial 'measurement suite' tools offer barely enough in the way of useful functionality to keep large backbones alive and evolving without constant vigilance and intuition on the part of weary, under-slept engineers. (Sort of reminds one of the street people that run up and clean your windshield (really badly) when you're stopped at a light in New York city, and then ask you to pay for it). But, unlike critical operational tools, ISP management is generally less willing to invest scarce resources in the development of these capacity planning /engineering tools. CAIDA wants to avoid the `useless windshield cleaning' situation by making sure that the network engineering expertise at the ISPs is in fact steering the research agenda based on their inter-ISP as well as intra-ISP needs. There was strong consensus on the profound need for ISPs to collaborate more directly with researchers and vendors to ensure the development of technical sound, responsive engineering tools. [A list of requisite tools/capabilities specified by ISMA ISPs is available in the ISMA Summary Report.]

Karen Sage (Cisco/Netsys) described the importance of understanding the limitations of tools, as well as recognizing their benefits. [During the break, network management/simulation tools from CISCO, MakeSys, and MIL3 were demonstrated.]

k claffy explained why visualizing Internet traffic is so difficult and progress is so slow. Examples of Internet visualization developed by CAIDA/NLANR include: Planet Multicast, daily visualizations of traffic across the Global Caching Hierarchy, real-time visualization of traceroutes, and an elementary macroscopic depiction of BGP traffic flows across Autonomous Systems or ASes (3-D imagery).  The absence of lat/long data from the DNS database, however, makes reconstructing traffic visualization a lengthy, manual process. Recommendations for including these data in the DNS configuration records are in RFC 1876.

Deborah Estrin (USC) spoke about the Virtual InterNet Testbed (VINT), a DARPA-funded collaborative effort between USC, ISI, LBL, and Xerox PARC. VINT seeks to model the dynamics of Internet protocols and traffic behavior, invaluable with algorithmic problems associated with introducing new protocols, e.g.

• Congestion control associated with TCP dynamics of different complex traffic mixes and paths
• Integrated Services with different admission control policies over interesting topologies
• Multiple transports (e.g., reliable multicast, tag switching) across large topologies

Currently the VINT project focuses on evaluating protocol designs "in context" (e.g. reliable multicast protocols over dynamic multicast routing). Immediate goals include composing simulations based on LBL's Network Simulator (ns) and developing libraries of topologies, workloads, scenarios and protocols. Current models include TCP with various queuing mechanisms (FIFO, RED, CBQ), real-time flow monitoring, and protocol animation. Work on RSVP and web caching is also planned. VINT is not designed to address parallel simulations or distributing simulations. More details on VINT are at http://netweb.usc.edu/vint.

The ISPs praised recent advancements by network simulation vendors, but urged these companies to develop capabilities to incorporate data across multiple vendors (most ISP use more than one) and across multiple ISPs (i.e., exchanging OSPF and BGP, simulate results without forcing ISPs to exchange sensitive data).

Participants discussed LBL's (reasonably) flexibile and scalable public network simulator ns (the network simulator developed by LBL). Deborah encouraged ISMA ISPs to use it and provide feedback to her and Steve McCanne (mccanne@cs.berkeley.edu). The ns framework and C++ code permits significant flexibility for the user, it also scales easily, increasing to 1000+ nodes.

  Presenter: Craig Labovitz (Merit / University of Michigan)

Craig Labovitz (Merit) outlined a new framework within Merit's Internet Performance, Measurement and Analysis (IPMA) effort for collecting distributed data. This new collaboration with the University of Michigan includes efforts relating to measurement/analysis of routing instability; topology; routing accuracy/problems, latency/loss/availability, throughput (passive and active), and real-time vs. historical analysis of data.

IPMA is reviewing the UARC project for insights on data collection and dissemination. UARC was developed to support distributed collaborative laboratories and focus on real-time data collection, scalability and robustness. In this model, distributed servers replicate data and deal with scoping and access control issues. The "Salamander" service is responsible for gathering information from probe machines and replicating the data. Data is filtered through Java-based report generators, databases, and e-mail scripts.

Organizations contributing data have the ability to provide the encryption key or define the scope (community) that will have access to it. The Salamander servers peer with ISP routers for data stability, topology, availability and accuracy. They do not use GPS antennae for timing, but can produce acceptable time measures as long as the round trip is symmetric (e.g. via a dial up POTS line).

Routing instability was discussed briefly. A surprisingly large portion of BGP traffic is either pathological or redundant. `BGP withdraw' problem gives rise to orders of magnitude more route withdraws than expected, and even just the announcement traffic turns out to be quite more `dynamic' than one might imagine. For example, one would expect that 40 thousand routes, with two flaps/day on average, would result in approximately 100 thousand route updates during the day. In reality, up to 3 million occur. There also appears to be a 30 second frequency dependency, possibly associated with the periodic aspects of the underlying protocol, which has been shown to lead to very wide-spread synchronization (see Floyd, S., and Jacobson, V., The Synchronization of Periodic Routing Messages, April 1994. )

The raw BGP data collected by Merit during its NSF-supported Routing Arbiter project is available for research use by the general public. A paper detailing the observed effects of BGP updates is available at http://www.merit.edu/ipma/analysis.

Overall, our current knowledge regarding the topological significance of route changes is minimal; it is an important area for networking research. ISMA participants emphasized the importance of the R&E community and vendors taking advantage of the historical BGP data available in Merit's archives to conduct simulations in order to predict the intricacies of route behaviors. Other participants requested that Merit/U. of Michigan explore the creation of a simple, queriable server where end-users can find out whether portions of the Net are up or down.

Questions were posed about the significance of numerous withdraws, e.g., the difficulty of some routers keeping up with the volume. It was noted that newer releases of Cisco's IOS fix one large source of the BGP withdraws from that architecture. The basic problem however, still exists and, rather than simply filtering out the withdraws, it would seem wiser to characterize the events that give rise to and follow these withdraws, and how they affect Internet performance from a larger perspective.

Vern Paxson (Lawrence Berkeley National Laboratory, LBL) provided an overview of pathchar and tcpanaly -- two important new tools currently under development at LBL.

Pathchar is being developed by Van Jacobson to estimate bandwidth capacity and queueing details via a hop-by-hop analysis. Vern nicknamed pathchar `traceroute on steroids'. Currently, run time tends to be long, from minutes to days, depending on path characteristics, rendering it an unlikely candidate for an everyday user tool unless this can be lowered. Pathchar has accurately identified 100 Mbps FDDI and 25 Mbps ATM links downstream from a 10 MB Ethernet, a pretty amazing feat (x-ray vision!). Problems with pathchar include dealing with timing noise and also with hidden hops, e.g., in ATM, frame relay, bridges (anything with store and forward delays unseen at the network layer). As long as the bandwidth between the hidden hops is large, however, their presence has little effect on the model. Future plans include: distributing this release, porting the tool to FreeBSD 2.2, Linux 2.x and Solaris; improving queueing and loss-rate analysis; and addressing the hidden hop issue. Copies of Van's pathchar transparencies are available at ftp://ftp.ee.lbl.gov/pathchar. Related visualizations using pathchar are available at http://www.caida.org/Pathchar and http://sitka.triumf.ca/net/pathchar.html.

Vern described his current work on analyzing TCP and end-to-end network path behavior, using a tool called tcpanaly. Tcpanaly recognizes nine different vendor TCP implementations, and how each manages their congestion window. The tool takes tcpdump format files as input. If given a single trace file recorded at either a sending or receiving TCP, then it analyzes the TCP's sending or receiving behavior. If given a pair of files recorded at both sender and receiver, then it also analyzes the network dynamics of the path between the two: one-way delay, retransmissions and loss, timing compression, bottleneck bandwidth, and queuing and available bandwidth. It calibrates the trace files in terms of checking for packet filter drops, clock adjustments, and clock skew. A copy of Vern's presentation is available at http://www.nlanr.net/ISMA/isma97_paxsonsl.ps.gz.

Andy Bierman responded to questions about the status of the IETF RMON working group. RMON-1 collects data about each packet (1 million elements of data) at very fine granularity of data. RMON-1 is based on SNMP data, therefore it has the same scaling problems of SNMP. Subnet or similar aggregation is needed in order to make RMON features easier to use. The RMON working group is also working on switch features for collection.

Guy Almes noted a legitimate need for users to know how the Internet is operating, but scaling user measurements is very difficult. While we don't want end users to be running traceroute or TReno at every URL request, everyone seemed to agree on the need to empower network managers at campuses and other large institutions with appropriate knowledge regarding performance and measurement that they can communicate to their user community. Scaling end-user measurements is a serious concern, but should not prevent user groups from conducting measurements independent of the information that they may receive from their ISPs. If ISPs made more and better performance data available to users, one ISP representative suggested, fewer users would feel compelled to take the measurements themselves.

During a discussion of topology mapping, John Hanley (Yahoo) suggested that a wrapper might be added around traceroute to find the ASes of the nodes on a route, plus the latitude/longitude of the node (by starting from the DNS entry), resulting in the address of a nearby reverse traceroute server, see additional comments.

Stephen Ladouceur (Bell Canada/Canarie) stressed that ISPs have a responsibility to be honest with customers, but don't have to waves flags for every mistake. It's human nature that if customers actively receive information about the existence of a problem, they may become overly conscious of similar problems that they wouldn't have otherwise noticed. ISP help desks and NOCs, Steve urged, should be provided with tools to simulate what the customer sees, and encouraged to communicate their findings to customers. Bell Canada utilizes tools similar to this to dial in and simulate what the customer sees, even utilizing similar workstations.

Panelists also emphasized the need for better measurement coverage of Internet infrastructure-wide performance, e.g., congestion at major exchanges and geographic regions, more public outage reporting, etc. References were again made to the 611 telephone repair line. Today's Internet users contact their ISP to report service difficulties, yet many of these problems are not local and are beyond the ability of individual ISPs to fix.

The group discussed the apparent disconnect between: (1) current measurement initiatives and tool development efforts and, (2) today's level of understanding regarding both Internet metrics and the questions that require measurements. Participants concurred regarding the growing urgency for metric definitions and standards from the IETF/IPPM working group. However, several participants stressed the importance of collecting data on traffic metrics that can be reliably measured in the hope that these efforts will converge with emerging data requirements from users and ISPs.

Suggestions made during these discussions included:

• Given their wide availability and end-to-end functionality, browsers (e.g., Netscape and Internet Explorer) might be configured to perform some basic measurements for end-users.
• Clever caching might ameliorate some of the perceived delay problems on the Internet (although some participants noted that most ISPs would have a disincentive to save bandwidth in this manner).

The airport and telephone analogies surfaced during these discussions. Participants pointed out that Internet end-users do not necessarily want to diagnose problems, but simply to know that their plane will be late and kept informed of congestion or other problems. On the other hand, airport and telephone users do not call those institutions to demand that certain equipment be bought or that upgrades be made, as happens with ISP customers. Airport and telephone customers also do not call the switches directly to ask what's happening during problems.

Regarding third party monitoring outfits reporting on ISPs to consumers, Bill Woodcock observed:

Hilarie Orman (DARPA) asked participants for recommendations on how to best develop predictive models of traffic behavior based on measurement data from trunks and interconnect/peering points. In particular, we must be able to describe both expected performance (throughput, delay, etc.) and network paths and topologies (ring, multiplexing, etc.). Internet tomography analysis, multi-network simulation and other forms of intra- and inter-cloud analysis are vital but daunting tasks, and the complexity of realistic topologies hinders measurement and modeling of paths and traffic behavior. While little is understood about how to address these topics, participants agreed that correlating output from various tools (active and passive) is an integral step. 

  Moderator: k claffy (UCSD/NLANR/CAIDA)

  Presenters: Joel Apisdorf (MCI), Nevil Brownlee (U. of Auckland), John Hawkinson (BBN Planet), Jamshid Mahdavi (PSC), Daniel McRobb (ANS)

Nevil Brownlee (University of Auckland) described the goals and activities of the IETF's Realtime Traffic Flow Measurement (RTFM) working group, which he co-chairs, see http://www.auckland.ac.nz/net/Internet/rtfm and the status of efforts to develop NeTraMet and the Internet Traffic Flow Measurement suite of standards. He differentiated among Accounting (understanding what is going on or "counting"); Billing (one application of accounting); and Flows (related to user-specific traffic/tasks). With respect to flows, one can relate timings within a flow to system performance, and one can also select "interesting" flows within a busy network. Both have their uses.

The University of Auckland meters internal links at sites, and international links at the NZ Internet Exchange (NZIX). They use third percentile day measures to get around burstiness and encourage spreading of the load. The university experimented with higher percentiles but found that it did not seem to make a big difference. The higher quartiles are more permissive without load spreading. Issues associated with traffic flow measurement include:

• granularity - i.e., adjacent address to/from router, peer address (IP address), transport address (protocol type, port), any combination of the above;
• accuracy & reliability - do we need: every packet, redundant meters and/or collectors, strong robustness to unusual conditions or failures?
• data reduction - front-end (at meter) by selecting flows of interest, using bi-directional flows, computing derived information using scripts and cron jobs, building distributions, e.g. packet size, inter-arrival time; standard data formats are required (RTFM meter MIB - RFC 2064 specifies data within the meter; NeTraMet produces flow data in a simple, self-describing file format, but this is not a standard).

Nevil also described efforts underway to port NeTraMet from a free-standing DOS program to a statistics module for OC3mon, see http://www.nlanr.net/NA/Oc3mon/oc3-api.html. Details on NeTraMet's Flow Data File Format are available at http://www.nlanr.net/NA/Oc3mon/ntm-flowdata.html. Nevil's presentation is available at http://www.auckland.ac.nz/net/Internet/rtfm/isma-flows.html

Joel Apisdorf (MCI/vBNS) described the architecture and capabilities of the Coral monitoring tools (OC3mon and OC12mon). The OC3mon monitors ATM over SONET on an OC3 tail circuit, acting as a passive, non-intrusive measure based on a PC (200 MHz Pentium) and two ATM NICs (for data send/receive). The unit is reasonably low cost ($6000), and works by tapping into the optical fiber and siphoning light off the circuit, reconstructing it into bits, cells, packets, and then flows, often exceeding 350,000 flows in any 5-minute interval.

All necessary aspects of the hardware are in the public domain, including the board layout, source code for FPGAs, and schematics. Collected flow data (see www.vbns.net for vBNS flow data,paper on tool at http://www.nlanr.net/NA/Oc3mon) include: time series information by application and protocol, AS matrix reports, and heavy-hitters (IP address-specific) reports. Reports on total usage over time (queries on total application use) are also available, sorted by flows, packets, bytes or duration. Currently each vBNS supercomputing center node has an OC3MON on each of its serial OC3 links to other sites. MCI also has OC3MON boxes monitoring two sample points on its commercial network.

OC12mon is in development, using prototype cards from Applied Telecom, and MCI-developed software. Initial deployment expected by July 1997, OC12MON resource constraints may require that each flow direction require a separate PC. Each OC12MON will require almost 3x the RAM (256-384 MB RAM) used in OC3MON, with each card costing about $7,000 as opposed to the $900 for FORE's OC-3 NIC cards.

All aspects of the OC12 card will be open, except for the daughter card that turns light into SONET payload bytes. Diagrams of the OC3 and OC12 cards and other details on their implementation can be found at http://www.nlanr.net/NA/Oc3mon/coralwksp.html. Future plans include a port to Unix (FreeBSD), drivers for FDDI, serial T3, encapsulations like DEC gigaswitch, and a playback mode to allow recreation of a traffic scenario. For presentation slides, see http://www.vbns.net/presentations/joel/index.htm.

Daniel McRobb (ANS) described efforts by himself and John Hawkinson (BBN Planet) to develop Cflowd to analyze data derived from NetFlow statistics output on CISCO routers. Material from a February 1997 NANOG presentation on these activities is at http://www.academ.com/nanog/feb1997/flow-switching.html. Daniel and John have worked closely with CISCO personnel (particularly Darren Kerr) to develop Cflowd and to identify changes in the NetFlow code necessary to support it. The motivation for developing Cflowd is to improve data available for capacity planning and trend analysis, not necessarily to analyze individual flows. Plots of Cflowd AS matrix information are available at http://figaro.ans.net/cflowd/.

Daniel reviewed statistics graphs of ANS' traffic, revealing interesting variations from typical Internet traffic, e.g., traffic for AOL (ANS' primary customer base) has a mean packet size of 500 bytes and dialup traffic peaks 7:00 p.m.and 2:00 a.m. They also found unexplained peaks every 30 minutes and 45 minutes, possibly resulting from cron jobs doing backups.

One challenge that developers grapple with is the question of what NetFlow data to capture. For example, Daniel showed plots of measurements at MAE-East based on flows coming into a FDDI at 60 MB/sec and going out via two DS-3 lines at 30 packets/sec. Scaling these measurements to dozens of sites across the Internet will require data prioritization and massive

Jamshid Mahdavi (PSC) talked about the important performance information that can be derived passively, e.g., via observing FTP sessions. PSC collects for each FTP flow out of their main servers: number of bytes and packets, duration, average packet size, flow goodput, and peak performance over 10 sec interval. Jamshid compares TCP header sequence number (min and max) information with IP statistics to derive a measure of "goodput". In the example shown (from December 1996 end of April), the resulting goodput measure was 87%.

A decrease in TCP sequence number may indicate reordering or dropped packets. RTT and loss rates can facilitate a comparison of expected vs. actual TCP performance. (The example shown was not a Sack TCP, so it did not achieve the predicted best case values.)

PSC's time series data revealed conditions worsening until they changed providers. He also suggested that providers might use this technique to detection trends early.

Currently, CISCO is the only router vendor with significant statistics functionality in its equipment. Participants noted the utility of having flow data from various routers to incorporate into simulators (Opnet, NetMaker, NetSys, etc.) Currently, simulators typically offer a set of preset probability distribution functions (PDFs), as opposed to empirically based distributions.

The Cflowd collection hosts queries summary data, not raw flows; the latter are kept in the router cache's circular buffer and force-dumped every 30 minutes. As this cache fills up, it simply exports the raw data faster.

Jamshid explained that he choose the 3rd quartile as a parameter for his FTP analysis after comparing it to higher percentiles and finding little difference above the 70th percentile. Given peoples' familiarity with quartiles, he felt that it was a reasonable choice.

  Moderator: Tracie Monk (UCSD/NLANR/CAIDA)

Les Cottrell (SLAC/HEPNRC) described a comprehensive measurement methodology based on ping statistics that provide short and long-term measures of bandwidth, response time, packet loss, reachability, and unpredictability. These monitoring efforts are conducted on behalf of DOE's Energy Science network (ESnet) and global High Energy Physics (HEP) community.

As end-users, the HEP community is primarily interested in measuring network performance as seen by the applications. Ping is well suited for diagnosing layer 2 performance and can be used in negotiating contracts, setting user expectations, and choosing ISPs. If, for example, ping delay data over a 10-week period that indicates three standard deviations difference from the mean (or even median) might trigger an alert to the ISP regarding the performance degradation.

Les reviewed several graphs of long-term trends, indicating that packet loss between the HEP sites he monitors grew from 2% to 8% over the last 6 months. Their data also includes monthly averages over the past two years, which indicate conditions improving within the U.S., but deteriorating over international links. The saw tooth data depicted in these graphs (available at http://www.slac.stanford.edu/grp/scs/net/talk/isma-sdo-may97/index.htm) reflect capacity upgrades or peering configuration changes.

SLAC coordinates multiple collection and analysis sites, and a web page for public access and review of RTT and loss rates. The collection architecture is hierarchical, avoiding the ominous O(n^2) ping matrix. They use a simplified scale based on packet loss (0-1%, good; 1-5%, acceptable; 5-12%, poor; 12-25%, bad; and >25%, unacceptable), which is particularly useful when communicating results to funding and management personnel. To calibrate, at 3-5% packet loss, video conferencing becomes very difficult. At 5-10% packet loss, video conferencing is not usable. ESnet's international links are particularly high-loss.

Les explained that the HEP community has agreed that ping is an adequate if coarse tool to measure performance, but that one must carefully choose targets to monitor (need consistency in the hosts, i.e., general web servers are bad; DNS servers and mail servers are somewhat better). Long-term uses of these data include in negotiating ISP contract parameters, setting user expectations, and sizing ESnet connections (which universities receive are to receive ESnet connections). In the future, SLAC will extend the monitoring to other HEP sites in Europe and elsewhere, and will arrange for a constant-size web page at each site to measure calibrated web server performance. The web pages for these monitoring activities are at http://www.slac.stanford.edu/comp/net/wan-mon.html.

Brian Kenner (InterVU) described the importance of optimizing Internet performance as it relates to the delivery of certain user-oriented services over the Net. Performance of the infrastructure directly affects InterVU's livelihood: effective delivery of streaming video media over the Net. To improve the quality of video reception by end-users, the company deploys distributed servers throughout the infrastructure, and mechanisms for the client to select the `optimal' server in real time.

InterVU has developed a special application (SmartVU), using traceroutes and 50KB http transactions, to assist users in choosing the optimal server. In testing, using the tool to select a server yields an approximate 10% performance improvement in video stream latency. Early testing also revealed problems with suboptimal configurations of certain OSes as shipped, and InterVU is working directly with vendors to fine-tune the servers. (Information on the diagnosis and resolution of these problems is available from Brian; his presentation slides are at: http://www.nlanr.net/ISMA/isma97_kenner/.

John Leong (Inverse Network Technologies) reported on an interesting dial-up ISP benchmarking service designed to measure the performance of dial-up ISPs (e.g. AT&T, CompuServe, AOL, CNC, Netcom, etc.) in terms of connectivity rates, service qualities, and related metrics. Inverse utilizes a custom software tool, with a bank of Windows 95 PCs with US Robotics 33.6 modems and Netscape browsers, to dial into over 600 POPs across the U.S. (18 ISPs, 20 to 40 POPs per ISP). The test environment is designed to replicate typical connection environments seen by users. A typical test run will yield between 80,000 to 100,000 samples per month, with which Inverse identifies the number of connection successes/failures (answers, busy signals, ability to log-on, etc.), as well as an ISP's modem connection speeds. For successful connections, Inverse's tool also measures fails/retries, response time (latency), and throughput for DNS, Web pages download and e-mail performance.

John showed graphs of performance differences among various ISPs. e.g. AOL showed a 60% failure rate during peak hours (6:00 p.m. to 12:00 a.m.) in March. [Which is why terms such as 'prime time' should be dropped from measurement activities. Fixed time windows, e.g., 8 or 12 hrs, are preferable.] Review of the AOL data showed failure rates as high as 75% in early evening. The good news for AOL is that they have steadily improved since December (got lots of new modems), at least until this report was released (May 97). IBM proved to be surprisingly robust during this period of the day, consistent with their focus on a business vs. consumer market. Most ISPs show connections at 28.8 kb; PacBell was an exception in March, with significant connections at or above 31.2 kb.

Graphs of DNS performance by ISP illustrated that while DNS servers response times are about equal (350 - 500 ms), the number of retries required before successfully resolving a query can be as high as 13% for some ISPs, perhaps due to packet loss in the network rather than to specific DNS server problems. John reviewed Web throughput data by time of day, noting that Web throughput for most ISPs starts to decline by 9:00 a.m., staying relatively low until 10 p.m.

John also presented a matrix of ISPs versus URLs tested and noted that some URLs have high throughput performance regardless of ISP and pointing out that two ISPs demonstrate consistently poor performance regardless of which URL is used. Finally, John noted that based on his Carnegie Mellon experience, it is inherently difficult to distinguish between applications/servers and network problems and suggested that performance measurement tools should be provided for both key distributed applications/servers as well as for the network. Furthermore, integration of computing user service help desks with networking trouble desks can assist in distinguishing networking problems from server problems. John's slides are to be posted on http://www.inversenet.com.

Jeff Sedayao (Intel) represents a larger customer of several ISPs, and advocates the use of statistical process control techniques to attribute network problems to causes, optimally in conjunction with one's ISP. While latency delays to some sites may be relatively unimportant, Intel needs to know when key sites, e.g., microsoft.com, become difficult to reach, and more importantly, why such problems occur. To this end, Jeff has created a tool, Imeter, that pings specific sites measuring delay, packet loss, and unreachables. Imeter can detect problems and monitor boundaries between service events. Cindy Bickerstaff (Intel) has developed a tool called Timeit which does HTTP GETs of pages from Intel sites, then times the response. It separates out the DNS response and reports on DNS lookup, connect time, delivery rates, and fraction errors. Intel uses this tool for experimental design of proxy configurations. Its results can be aggregated by country or by geographic region.

Both tools are used as components in selecting ISPs and verifying commitments in ISP Service Level Agreements. Intel also uses the data to trigger more thorough testing, trouble ticket opening, or alerts to ISP, and is exploring capabilities to generate alerts through scripts examining outlier data. Both tools are available from ftp://ftp.intel.com/pub/ietf/ippm and ftp://ftp.va.pubnix.com/pub/uunet. Jeff's presentation is available at http://www.nlanr.net/ISMA/isma97_sedayao/.

Considerable concern exists regarding scalability of active measurements, especially if every web user were to start doing such measurements, e.g. with NetMedic.

However, most felt that the two efforts described here (SLAC, Intel) are being pursued in a responsible fashion. Intel even blocks out superfluous measurements using tools such as NetMedic (http://www.vitalsigns.com/), since they would not add any more value to the testing already being done, and in fact would incur considerable unwanted network traffic.

One participant asked if anyone had experience with 'long distance calls impacting the numbers with transients' (lower quality connections that cause the modems to step the speed down). While participants were unfamiliar with this particular problem, most acknowledged that interaction between telephony and IP infrastructure is poorly understood. Busy signals are by no means the only problem with failed connections to dial-in modem pools.

Inverse responded to a question regarding the confidence interval of its measurements, particularly the variations between those ISPs that share the same infrastructure. ISPs urged Inverse to share its findings with providers in order to resolve unexplained anomalies.

Concerns were expressed about possible distortions from performance surveys that assume a single brand of modem, e.g., ISPs utilizing similar hardware will appear to be have better performance due to the interaction of like systems. Situations where ISPs may call-forward a customer to an unbusy POP and transfers to lower grade service as networks' load increases may also lead to distortions and complicate the tradeoffs between performance and access. These issues make conducting measurements challenging, explained participants.

Trust: Would customers trust statements made by their ISPs? Would users trust measurements made by third party groups? The group seemed to think that, in fact, when ISPs explained conditions or performance results to customers, these customers tend to believe them. The problem lies in the dearth of information currently available to users, in the absence of which, users will obtain data themselves. Participants also felt that as reliable third party measurement services emerge and mature, major users (including Intel and the HEP community) will be inclined to outsource their performance monitoring/analysis requirements.

Participants supported the need for content providers and delivery firms, e.g., InterVU, to conduct performance measurements aimed at improving services. However, it's unclear how representative a 50 Kb test file throughput is of a 5Mb video stream, and folks weren't thrilled about the idea of injecting additional traffic onto the net solely for the sake of comparing ISPs.

Two ISPs indicated that they occasionally place servers on major customer sites that monitor other servers at different sites (same customer; same ISP). These measurements are used in assessing whether the customer's performance expectations are being met; performance under par may yield rebates on the customer's contract.

Participants discussed meaningless measurements being taken simply because the data exist, as opposed to solving traffic performance or engineering problems. One ISP compared the current state-of-the-art in Internet measurements as akin to putting probes in the chassis of a television set with no overall concept of the picture being displayed (i.e. no big picture).

  Moderator: k claffy (UCSD/NLANR/CAIDA )

A summary of 'ISMA Threads' covering both inputs on the surveys and discussions during the two days was used to frame discussions during this session. The following section is organized around those summary slides.

• predict in order to maximize scarce resources
• facilitate network operation, planning, design
• competitive advantage in service provision
• support billing/usage-based services
• ISPs should be active in defining metrics and developing analysis techniques, as well as collecting data and educating users on correlation between metrics and actual network performance  

Doing measurement doesn't mean anything, explained several ISPs, unless you can leverage it to change the way your network runs. Yet for these ISPs, almost any measurement tools they deploy, and new data they receive, seems to have value.

One difficulty is that our basic understanding of the Internet is still weak. We need a model of the Internet to put these traffic measurement data in perspective. This would be of tremendous value to ISPs as they plan for and instrument their networks.

• traffic engineering mechanisms (e.g., CflowD, AS matrices)
• ways to characterize/track path performance wrt QoS/bottlenecks
• better integration of measurement tools into networking equipment
• closer cooperation/collaboration among communities (R&E, ISP, vendor, customer) wrt designing tools, data acquisition efforts (better coupling with what problems are being solved), postprocessing/analysis
• development of high fidelity multi-platform simulation tools that permit real time feedback from multiple ISPs while ensuring data confidentiality

Participants entered into a lively discussion about the merits of active versus passive measurements. The focus of much of ISMA has been on active techniques, yet these are much more complicated than passive techniques, which have little standardization in terms of definitions or accepted measurement practices. Active measurements are also not readily scalable and run the risk of influencing the performance of the traffic being measured.

Passive measurements also have problems, however, touching upon privacy concerns and necessitating a degree of cooperation from ISPs that is not necessary with active end-to-end performance measurements. Passive measurements tend not to reflect performance as seen by the user, and the math becomes increasingly complex as one attempts to extrapolate performance measurements out from the probe device to other parts of the network.

The possibility of 'parasitic' measurements was discussed. This topic may be of growing interest as our understanding of how to piggy-back on protocols improve. According to Matt Mathis, a TCP MIB might be developed to serve this function (note that Matt is working on such a MIB). Having HTTP timestamp packets might also provide some insights.

The problem of measuring the Internet with broken TCP implementations was also discussed. In this situation, one might find better performance on a broken Internet than on a non-broken Internet. [Note that measurement problems resulting from alternative, often faulty TCP implementations, surfaced during several ISMA sessions. This is particularly problematic for end-users using Windows; these platforms are notorious for their poor TCP implementations.]

Measurement/analysis of global routing information was cited as an important area promising benefits to both end users and ISPs. ISPs also stressed that they are willing to pay for additional router features that improve performance measurement capabilities. Specifically, queue length measurement showing queue length histograms are needed. Participants urged router vendors to focus more on the operational challenges facing ISPs.

what investments would yield maximum value for end users?

• performance monitoring tools/data that assist in making competitive vendor selection / contract negotiation and in ongoing monitoring of ISP services
• tools that assist in identifying/diagnosing problems, e.g., id of bottleneck on lan vs. isp vs. general Internet
• end users are most interested in metrics observed at the edges of the network & should be involved in the definition of these metrics

Having end-users conduct performance measurements/diagnosis, and possibly all contacting ISPs' NOCs requesting that problems be tracked/fixed, does not scale. However, having a small group of skilled firms specializing in these activities is a viable option. The presence of tools such as the recently released NetMedic, that are not scalable, nor are they particularly accurate in their diagnoses, suggests the need for more attention to this area. The theme of this discussion was that, in the absence of data, end-users will resort to whatever tools are available. It is also important to insert realism into users' expectations which having an intermediary third party may facilitate, e.g., with verifying that a problem actually exists and participating in the diagnosis and correction.

Several groups are beginning to survey available tools and evaluate their strengths and weaknesses, including the CAIDA Tool Taxonomy and efforts by TERENA's measurement task force. 

Participants also discussed the use of pathchar by end-users, agreeing that it is a subtle tool, therefore unlikely to have much value for the average end user. Potentially, however, the community could use this tool to identify fault with respect to congestion, packet loss or other problems. It could eventually have as influential an impact on network diagnosis as traceroute, and may receive considerable visibility once Van issues a Windows version. Several people suggested that what is needed to limit the number of independent measurements of the same problem, particularly given the long run-time requirements of pathchar, is distributed databases from which users can request results of recent measurements. 

• before you collect, clearly articulate what question you are trying to answer, or what hypothesis you are testing
• collection, aggregation, correlation, dissemination of data on traffic loads, performance levels, network outages
• in addition to new tools (e.g., pathchar, tcpanaly), need better interpretation of existing tools (traceroute, ping, ttcp, dig), perhaps augment their functionality/deployment so end-users can offer ISPs more useful feedback
• vendor access to data for equipment design/implementation (but vendors should take advantage of the wealth of data currently available, e.g., RA BGPstuff)
• confidentiality/privacy, tools to anonymize collected data
• improve the ability of network elements (routers, switches, etc.) to accurately measure and report network performance data, e.g., embedded RMON technology

Since significant tax dollars are being spent, participants all agreed that they have a responsibility/desire to help ensure that they are spent wisely. ISPs, in particular, questioned the role of government in conducting measurements on the commercial Internet, although most supported government efforts to expand the overall knowledge base of Internet traffic behavior, particularly relating to routing.

• monitor end-to-end performance, react/communicate w. their ISP when not meeting expectations/guarantees
• regular feedback to tool developers to review tools' goals, purposes, users
• content providers should be directly involved in traffic measurements; results applied to understanding customer needs and to improve protocols and implementation of products

• end-to-end traffic measurement and analysis
• identifying appropriate metrics/meaningful measurement (IPPM has lead in the IETF, but other communities should take a more proactive role in this process)
• address evolving nature of the Internet, e.g., measurement implications of IPSEC, higher speeds (modem and wires), queuing, scaling issues...
• developing tools/simulators that reflect real-world situations, e.g., brownouts and similar 'soft' outages, and tools that are not overly invasive to the net (passive monitoring techniques where feasible)
• better real-time diagnosis of problems, especially routing-related
• standardization of methodologies, protocols, data formats, etc. making tools widely available

The greatest challenge facing us today, ventured participants, is the growing complexity and scaling of the Internet infrastructure. More protocols, more complex infrastructure, IP/anything, growing user demands (streaming video) all dynamically changing make achieving a set baseline for the Internet, then measuring it, very, very difficult. The Net still holds many secrets (anomalies, unexpected behavior) that, once uncovered, can be fixed. If not uncovered however, they will continue to complicate effective operation and scaling of the Internet. Continuing to study, qualify, and quantify Internet traffic phenomena is critical.

• little agreement here... except wrt significant challenges
• coordination of disparate efforts
• techniques as passive as possible
• distributed acquisition, warehousing, analysis
• information sharing/privacy
• communities working together

Measuring the performance of the "cloud" abstraction is particularly difficult, e.g., 'where do you clamp the probes?'

• very different opinions depending on community, e.g for....
• ISPs - homegrown measurement/analysis tools the norm, albeit increasing interest in some flow measurement tools (CflowD; OC3mon)
• end-users - ping and traceroute are important (however, accurate interpretation of results is very difficult!)
• for researchers - new tools such as pathchar and tcpanaly have promise; distributed, scalable collection of measurement data, warehousing and meaningful analysis and simulation of traffic behavior are challenges
• commercial & research measurement tools are being summarized in CAIDA Tool Taxonomy ( www.nlanr.net/Caidants/meastools.html ) -- send inputs to INFO@CAIDA.ORG

'If the Internet had a research arm,' kc asked, 'what would panelists have it do in the next three years?' Several people expressed concern that commercial router vendors are not moving as quickly to introduce new technologies as they did prior to 1994. It might therefore be useful to have Internet researchers get more involved with investigation/specifying/building high performance hardware, as well as encourage industry collaborations (a la the CISCO, ANS, BBN Planet collaboration on NetFlow/Cflowd) and equipment vendor (e.g., Bay and Juniper) participation.

Concerns were expressed about the lead times inherent in today's Internet: elapsed time between identifying a congested link and having a new circuit in place is remarkably long, generally 9-18 months for T3 and above. [Another ISP ventured that this used to be in the order of 2-4 weeks, but for their company, it is now almost a year.] Anticipated shortage of fiber is expected to aggravate this problem in the short- to medium-term.

Two of the ISPs acknowledged that they are now offering service `guarantees' to their customers (for a premium), via a `rebate' for underperforming intervals. In response to questions about their offering end-to-end performance guarantees across multiple networks, they explained that the topic had not presented itself (although they acknowledged that their sales personnel would most likely be willing to negotiate such an arrangement). In dealing with two providers, they suggested, customers would need to get agreement on performance to a common point and have the ISPs jointly agree on penalties for non-compliance.  

Last updated 14 June 1997
Questions or comments should be directed to Tracie Monk at tmonk@nlanr.net