Network Analysis Infrastructure

Towards a systemic understanding of the Internet organism:
a framework for the creation of a Network Analysis Infrastructure

Initial version, needs to more refinements

10 May 1998

  1. Introduction

    At the current time, where the scope of the Internet is dramatically expanding in terms of global ubiquity, performance and degree of interconnections, the understanding of the increasing complexity and decreasing manageability at its systemic level is decreasing. The services of the network today are assessed in perceptions and probabilities, drifting from the fairly pure mathematical science of assessing early and simpler networks more and more towards the unpredictabilities of the complexity of a biologic organism. An example result is that more and more service providers fight public information about user perceptions that put their services into a bad light in terms of reported performances, defending themselves that their component network is in good shape, and that the problem must be somewhere else. In the same sense, participants in the global Internet are often inconsiderate of a common fate-sharing model where the union of ISPs has to provide and prove predictable and verifiable performances.

    A quick verification of global routing tables in http://moat.nlanr.net/ASPL/ shows that commonly the distance between BGP-advertised Internet entities traverses 4 or 5 or more Autonomous System numbers, roughly mappable to service providers. This illustrates that from a user's point of view, the singular service parameters of an ISPs are important, but typically not reflective of the end-to-end performance the user can expect.

    In response to these issues, NLANR is attempting to create a network analysis infrastructure, focusing on the vBNS sites, but extending its notion of research to include at least the HPC community. Not to give all-encompassing answers, but to gather more insight and knowledge of the inner workings of the Internet. This NLANR activity is complemented by and collaborating with CAIDA (Cooperative Association for Internet Data Analysis). While NLANR is focused on a work scope surrounding the NSF academic research agenda and clientele, CAIDA is an organization extending the scope of collaborative measurements and network analysis to private Internet service providers.

    An analysis infrastructure has the objective to operate beyond the means of network measurements, as it is not good enough to create large data silos containing alot of measurement data that is hardly ever been looked at. The definitions of analysis objectives is critical early on, while enough flexibility has to be retained to adopt to new and extended requirements.

  2. Measurement objectives

    The parameter space for Internet measurements has to encompass at least three dimensions:

    Three areas for measurement data currently being considered are passive workload profile assessments, active performance measurement, and stabilities and status of Internet routing. Activities in all those areas will be part of the NLANR work scope, thought the initial primary focus will likely be on passive workload profile assessments, given that there is quite a bit of activities in the other two areas at other sites throughout the Internet.

    1. Passive workload profile assessments

      The passive measurements are done non-invasive relative to the observed networking environment. It will not impact the performance of the network while measurements are being done. An example of this are OCXmon monitors which tap into the light of a fiber interconnection by means of optical splitters, and collect packet header traces.

      Packet header traces can generate an immense amount of data, and, unless the whole data set is desired to be kept, it is usually critical to abstract the data as quickly, locally, and as much as possible, which then of course looses information contained in the full traces. Abstractions can happen at a central data collection location, or, if possible to also avoid high data volumes to be transfered, at the location where the data is being collected. Sometimes it may be necessary to abstract already abstracted data further, which may include such processing at both the data generating, as well as a central collection site.

      If data abstraction happens at the location where the data is being collected, the two opportunities are to do it in real-time or non-real time.

      Real-time abstractions allow for a continuous operation, but are limited by the availabilities of local CPU and memory resources to accommodate the analysis at the speed where new data is coming in. This becomes an especially difficult issue is complex analyses for in-depth understanding of the networking environment are desired, or in high traffic load situation. In such cases the local collection/analysis machine turns into a bottleneck.

      A non-real-time analysis assumes staggered data collection and data analysis phase, presumably followed by a phase where the analyzed data be transferred to a central location, and the data collection be restarted. However even in this case highly complex analyses can take a very long time. This methodology would reflect a reality sampled in time, as the measurement phases are interleaved with the analysis phases.

      The prime example for a passive measurement tool for the NLANR activity is OCXmon, which allows for OC3 and OC12 based environments to be monitored, and traffic traces be analyzed, without impact to the network by the measurement itself. Strategic network-centric locations of significant traffic aggregation will form the nucleus for deployment of these data collection tools.

      An analysis application of OCXmon considers active flows, based on work done at SDSC starting in the early 90's.

      This model of flows considers transaction impacts as the network sees them, and is useful for a first level of traffic aggregation properties. It assumes a flow to start when a specific criteria is met, and continue until a timeout expires for the particular flow, at which time the timeout is subtracted from the flow duration. Flows can be defined and possibly aggregated relative to multiple criteria, such as source hosts, destination hosts, host reductions to networks, quintuples of hosts, protocol, and ports, and so on. Flows can allow insight into packets, bytes, and durations of network transactions, relative to a specified flow criteria.

      It should be pointed out that there are other alternatives for workload profile measurements, such as Cisco flow monitors, which are not entirely passive, insofar as they impact the packet forwarding performance.

    2. Active performance measurements

      Common applications of active performance measurements assess host reachability, packet losses, and throughput measurements. The throughput measurements may be windowed (e.g., using TCP) or non-windowed (e.g., plain UDP). These kind of activities are getting more and more wide spread, as any "common user" can undertake these tests by himself from his host to other hosts quite easily. Some (a quite increasing number) of Internet sites attempt to assess Internet performance figures by the aforementioned means and by probing many sites from usually a handful of machines, and turn them into "Internet Weather Reports."

      In the NLANR case we will measure performance parameters from within the network, i.e., via probes deployed within the infrastructure itself. Strategic locations will be selected to be of use to our objectives.

    3. Stabilities and status of Internet routing

      Internet routing considers today's complex mesh of interconnections between component networks and service providers. Since the ARPAnet days of a hierarchically structured core network are long over, loops across networks can form, end local change events can get globally distributed.

      Of concern is the volume of routed identifiers (IP address/mask pairs), as well as their churn, as events cause changes, such as up/down events. The sheer size of the tables increases the burden on the packet forwarding engines, the churn impacts the ability to keep the global routing system current, and also introduces traffic loads as routing information is being exchanged.

      The routing table system also reflects the addressable network entities that can be reached, and what address occupation of the IP space they utilize. A snapshot evaluation is demonstrated in http://moat.nlanr.net/IPaddrocc.

    Another important area of consideration is measurement and analysis work in web areas, such as covered in the IRCache activity. While this is not a direct part of NLANR/MOAT, both projects will be able to benefit from each other.

    The goal of these activities is to produce public analysis results, data sets, and methodologies of use to the Internet community. This is different from the CAIDA objectives and its focus on using analysis results, data sets, and methodologies in an inter-ISP environment, but not necessarily for public distribution.

  3. Overall network analysis environment

    To successfully design the a system for the analysis infrastructure, layers of functions will have to be considered:

    The overall analysis infrastructure requires sets of machines and tools with various functionalities, including:

    The following depicts a conceptual implementation framework:

    The analysis infrastructure itself is supported by miscellaneous monitors which collect data, with possibly some local analysis, and results sent to a centralized collector for further analysis and storage. Some of the data will be sensitive in nature, e.g., it may contain IP addresses for which privacy issues have to be considered. Machines that may contain such sensitive information are marked in red. The yellow boxes refer to data that has lost some sensitivity as a result of the initial aggregation, but may still have to be treated as potentially sensitive. Green functions would contain freely distributable and desensitized data.

    The abstracting and archiving will happen at at least three levels:

    Initial collection of data and local analysis will happen at the measurement location, followed by central analysis and data cataloging, and either archived on NLANR hard disk, SDSC HPSS storage, or other media.

    The analysis system provides data for NLANR researchers doing network analysis on behalf of the project. It also provides for public dissemination of data and information regarding the network properties.

    This system will evolve further during the duration of this project, and the diagrams above should be viewed in the context of a conceptual framework only.

    For the time being, we have a 400MHz machine used for the "first stage storage and computation" with about 20GB of disk space. The "external presentations" server is "moat.nlanr.net" with its web interface.

  4. Measurement topology

    The topology of the analysis infrastructure will evolve over time.

    Currently we expect to start with five sites using Unix based OC3mon measurement machines:

    An expanded list of the vBNS HPC sites can be found in http://www.vbns.net/logical.html .

    An example scenario for a more complex measurement environment is depicted above, and illustrates a collaboration with the NASA Ames Inter-eXchance (AIX) and the MAE-West network interconnection of Metropolitan Fiber Systems. The two environments are using four striped OC3 channel to interconnect Digital Gigaswitches, and an OC3mon measurement machines collects packet headers off the second of the four OC3 stripes.

  5. OCXmon passive traffic tracing

    OCXmon is a suite of flexible, affordable, high performance network statistics collection tools. Current development is focused on the OC3 (Coral/OC3) and OC12 (Coral/OC12) environment, though other possibilities are under consideration. This current summary focuses on OC3mon for OC3 networks.

    Coral is a continuation of the OC3mon activity of MCI, which was based on measurement and analysis tools assessing active flows for Ethernet and FDDI developed at SDSC, but adds additional analysis functionality. A more comprehensive presentation about the OCXmon environment can be found in http://moat.nlanr.net/Presentations/NAI .

    The initial development was done for a DOS platform with a pair of FORE cards (one for each traffic direction), as intelligent and programmable front-end cards by Joel Apisdorf of MCI. Jon Dugan of NCSA, based on initial work by Eric Hoffman, adapted the code to the FreeBSD 2.2.5 Unix platform, by using Joel Apisdorf's FORE firmware unmodified, and including own code to interface the FORE controllers and the firmware to the flow analysis code. Also independent code to write header traces to hard disk exists for FreeBSD.

    OC3mon utilizes optical splitters to tap non-intrusively into optical fiber, to take a small fraction of the light for its own purposes, but without impacting the operation of the network. The observed fiber may have ATM switches or ATM hosts at its ends.

    The monitoring system consists of a distributed system, where the downloaded software on the ATM cards is excerpting packet headers, then pushes it across the PCI bus into main memory. The main processor has then the responsibility for post processing, including simply writing the data to hard disk.

    The ATM card subsystem shares the host bus with the memory and the rest of the host components, but has to be able to become bus master for both its cards to control the bus for the data transfers.

    Several modules are available to be down loaded into the FORE ATM cards:

    These are tailored for different requirements. E.g., while pca200e.bin is the standard module for the collection of the first cell of each packet, skipgigx.bin may have to be used on interconnections between Digital Gigaswitch nodes, in order to skip over the MAC level information.

    Once the ATM cards have delivered data into the main memory, the host processor can access the data, and decide what to do with it.

    This may include a real-time analysis function (such as real-time flow abstractions and analysis), collection the raw data to a hard disk file, staggered collection and post processing, and so on.

    Two reference implementation for OC3mon exist, one for Unix in http://moat.nlanr.net/Coral , and the DOS version is available from http://www.mci.net/~apisdorf/coral . This software and the analysis components are still evolving, and there will be changes over time.

    OCXmon Packet generator

    As an essential tool for verifying OCXmon performance, Joel Apisdorf has extended the functionality of the FORE front-end card in OC3 monitors for being able to generate cells based on an input file read by the host processor.

    The file input format used is similar to the packet trace format that OCXmon generates. Such a file can also easily be generated from a program.

    The need for a high performance OC3 packet generator arose when test machines connected to a data collector were able to fully utilize an OC3 connection for large packets, but were only able to generate about 40,000 minimum sized packets per second, which is only a fraction of the OC3 bandwidth.

  6. Other passive traffic collection

    An obvious candidate for (close to) passive data collection is using SNMP data from participating routers. In the vBNS context this has to be considerate of what is already being collected by the MCI vBNS team, how often the collections happen, and at what locations. The NLANR measurement and analysis activity, while attempting to be an additional source of information to the community, should be complementary to the MCI work. However, minimally NLANR should store measurement data for now, and analyze based on actual needs.

  7. Active performance measurements

    Given that a great many people and institutions undertake throughput, reachability, and loss experiments, this area should probably not be the strongest initial focus for NLANR, as the primary focus should be on what is not already covered. Hence, this section needs details to be filled in over time.

  8. Status of routing

  9. Analysis objectives

Comments or questions can be sent to info@moat.nlanr.net