Technical Report (draft)
“Background reading on Internet Measurement”
2001. 8. 28.
This technical report was written for three goals. First, this report provides background
on the current Internet architecture and describes why measurements are a key
element in the development of a robust and financially successful commercial Internet.
Then, this report discusses the current activities of main Internet measurement project
underway within various forums to encourage the development and deployment of
Internet performance monitoring and workload characterization tools. And finally, this
report provides three main cases on the measurement project.
Table of Contents
2. Internet Architecture
2.1 Defining Internet Architecture
2.2 The Original Requirements
2.3 New Architecture Requirements 3. Measurement Activity
3.1 Internet Traffic Performance Measurement
3.2 Internet Traffic Flow Measurement
3.3 Active versus Passive Techniques
4. Case Study
4.1 Skitter Project
After the design of Internet technology was guided by DARPA of the US DoD , the
Internet and its architecture have grown in evolutionary fashion from modest beginnings.
The design philosophy has evolved considerably from the first proposal to the current
standards. [Clark88] The network architecture is still developing, and it represents a
set of deliberate choices out of many design alternatives. [Cerf74]
The research community that had developed the ARPAnet protocols introduced the
notion of network Architecture during the Internet research phase. Network
architecture is a set of high-level design principles that guides the technical design of
the network, especially the engineering of its protocols and algorithms.
The important features of the original Internet architecture [RFC1958] included:
? Heterogeneity is inevitable and must be supported by design.
? If there are several ways of doing the same thing, choose one.
? All designs must scale readily to very many nodes per site and to many
millions of sites
? Performance and cost must be considered as well as functionality.
? Simplicity, Modularity
? Avoid any design that requires addresses to be hard coded or stored on non-
? A single naming structure should be used.
? Upper layer protocols must be able to identify end-points unambiguously.
The [Clark88] paper catalogs one view of the original objectives of the Internet
architecture. The top-level goal for the Internet Architecture was to develop an
effective technique for multiplexed utilization of existing interconnected networks. This
goal contains the word “effectiveness”, and following list summarizes was ordered with
the most important requirement first.
? Survivability: Internet communication must continue despite loss of networks
? Type of Service: The Internet must support multiple types of communications
? Heterogeneity: The Internet architecture must accommodate a variety of
? Distributed management: The Internet architecture must permit distributed
management of its resources.
? Cost: The Internet architecture must be cost effective.
? Ease of Attachment: The Internet architecture must permit host attachment
with a low level of effort.
? Accountability: The resources used in the Internet architecture must be
The [Clark90] paper presents a set of features that are said by the authors to be the
considerations for a new generation or the next generation of Internet protocols. They
essentially propose a rework of the protocol implementation based on two main factors:
Architectural considerations and Application based framing and Architectural
considerations. They list out the need for and the important factors among these two key principles. They also consider all the functions now provided and the need to
rework these. They also present a brief section about ILP considerations.
The goal of a new-arch research program should be to define and prototype a future
architecture towards. A new architecture should lead to greater functionality, lower
costs, and increased adaptability for all types of communication. [Braden00] There
could be many causes to demand the change of the Internet architecture. For example,
the commercialization of the Internet is the main factor for the new requirement. Some
important new requirements that may influence the new architecture are as follow.
? Mobility to support flexible, efficient, highly-dynamic mobility.
? Policy-driven auto-configuration of end systems and routers, subject to policy
and administrative constraints.
? Highly time-variable resources over short time-scales due to switched
backbone links, or due to mobile devices
? Allocation of capacity among users and applications.
? Extremely long propagation delays.
Above requirements has dealt with technical requirements, still there are significant
non-technical drivers on Internet design such as commercial drivers, legal and public
Increasingly, both users and providers need information on end-to-end performance
and traffic flows, beyond the realm of what is realistically controllable by individual
networks or users. Path performance measurement tools enable users and providers to
better evaluate and compare providers and to monitor service quality. Many of these
tools treat the Internet as a black box, measuring end-to-end characteristics, e.g., response time and packet loss (ping) and reachability (traceroute), from points originating and terminating outside individual networks. [Monk97] It is detailed traffic and performance measurement and analysis that has heretofore been essential to identifying and ameliorating network problems. Trend analysis and accurate network system monitoring permit network managers to identify hot spots (overloaded paths), predict problems before they occur, and avoid congestion and outages by efficient deployment of resources and optimization of network configurations. As the nation and world become increasingly dependent on the Internet, it is critical that we develop and make available mechanisms to enable infrastructure-wide planning and analysis and to promote continued efficient scaling of the Internet. User communities are also exerting
verifiable service guarantees that are not readily available pressure on providers for
under the current Internet.
In order to achieve measurements that are readily comparable and relevant, a fundamental first step is the development of common definitions of IP metrics. The
Internet Engineering Task Force (IETF) IPPM working group[IPPM] is working to provide a more rigorous theoretical framework and guidelines for designing measurement tools robust to the wide variety of disparate signal sources in the friction Internet.[Paxson96] In late 1996, draft requests for comments (RFCs) were issued delineating metrics for connectivity [RFC2678], one-way delay[RFC2679], and empirical bulk transfer capacity[RFC3148]. A charter for continued development of the
community's understanding of performance metrics is also now available at the IPPM home page [IPPM]
The below image on next page depicts the general architecture for active and passive measurement techniques.
Well-defined metrics of delay, packet loss, flow capacity, and availability are fundamental to measurement and comparison of path and network performance. Tools that measure in a statistically realistic way are disturbingly slow to emerge, but it was started to see reasonable prototypes for measuring: TCP throughput (treno, Mathis&Mahdavi/PSC), dynamics indicative of misbehaving TCP implementations (tcpanaly, Paxson/LBL), and end-to-end delay distributions such as NetNow (Labovitz/Merit), the Imeter (Intel) and detailed ping and traceroute analysis (Cottrell/SLAC). The network time protocol (NTP) addresses many of the time issues associated with measurement tools, but some metrics (e.g., one-way delay) will require
collector node User
monitor Measurement Network
packets node Copied Active packets
synchronized infrastructure measurement probes and beacons deploying global
positioning system (GPS) and similar timing devices.
In general, these users are most interested in metrics that provide an indication of the
likelihood that their packets will be getting to the destination in a timely manner.
Therefore, estimates of past and expected performance for traffic across specific
Internet paths, not simply measures of current performance, are important. Users are
also increasingly concerned about path availability information, particularly as it affects
the quality of Internet applications requiring higher bandwidth and lower latency/jitter,
e.g., Internet Phone and videoconferencing. Availability of such data could help assist in
scheduling online events, such as Internet-based distance education seminars, and also
influence user willingness to purchase higher service quality and associated service
Traffic flow characterization measurements focus on the internal dynamics of individual
networks and cross-provider traffic flows. They enable network architects to better:
? engineer and operate networks,
? understand global traffic trends and behavior, and
? adopt / respond to new technologies and protocols as they are introduced into the
Data available from traffic flow tools include flow type (e.g., web, e-mail, FTP, real-audio, and CUSeeMe); sources/destinations of traffic; and distributions of packets sizes and duration. These measurement tools must be deployed within networks, particularly at border routers and peering points. Traffic flow characterization measurement therefore requires a higher degree of cooperation and involvement by service providers than do end-to-end performance oriented measurements. End-users or large institutional sites can also use these tools to monitor traffic, e.g., MCI has placed OC3mon flowmeters at vBNS nodes at each of the NSF-supported supercomputing centers. These devices provide detailed information on traffic flows and assist in analyzing usage and flagging anomalies.
OC3mon traffic monitor (providing realtime Flow Characterization Tools include the
monitoring of traffic at 155 Mbps speeds), developed by Joel Apisdorf and others within MCI's vBNS research team. MCI makes detailed flow data graphics publically available through the vBNS web site.[vBNS]
Nevil Brownlee of the University of Auckland, New Zealand, has also been working with the IETF Real Time Flow Meter (RTFM) working group to develop tools for accounting and related flow measurement. He has developed the NetraMet / Nifty tool, most
notably to support resource accounting in New Zealand. John Hawkinson (BBN Planet) and Daniel McRobb (ANS) are developing the Cflowd tool to augment and further analyze data provided by the netflow switching capability of Cisco routers. They presented preliminary results of their analyses at the February 1997 meeting of the North American Network Operators Group (NANOG).
Active measurements inject test packets into the network and observe their behavior. For example, the simple ping tool measures round-trip-time(RTT) of ICMP probe
packets, In the active measurement arena, commercial and research ventures are emerging.[Claffy00] Benchmarking end-to-end performance of commercial service providers (for example, transit, access, content hosting, and caching) across clearly specified paths or segments of the Internet is now a competitive market. Often spawned by end users interested in verifying performance of their Internet service, these measurements typically involve an end host sending active probe traffic out into the network and recording the delay until packets return to their source. Unfortunately, such traffic measurements inherently involve a large number of parameters that are difficult, if not impossible, to model independently; the complexity renders elusive any
comparability or useful normalization of gathered data.
In contrast, passive measurements observe actual traffic without perturbing the network. Passive monitors must process the full load on the link, which can be problematic on high-speed links. Passive measurements commonly collect traffic flow data, either from routers and switches, or from stand-alone traffic meters. [Brow01] Passive
infrastructure-wide measurements and characterization of traffic remain the purview of researchers and some large transit providers. The National Laboratory for Applied Network Research (NLANR) makes available traces from various High Performance Connection (HPC) university sites for analysis by the community. Analyses using these somewhat limited windows onto the Net suggest the emergence of new applications, such as streaming media (audio and video) and games, as well as trends in the distribution and sequencing of packet sizes. Some workload trends have clear if not ominous implications for ISPs--for example, the proportion of non-congestion controlled traffic (as derived from packet traces) directly affects infrastructural stability. The distribution of TCP flow lengths, while nontrivial, also indicates the degree to which traffic is likely to respond to network signals.
CAIDA (Cooperative Association for Internet Data Analysis) [CAIDA] is a project of the National Laboratory for Applied Network Research(NLANR) within the University of California, San Diego (UCSD).
CAIDA is meant to be inclusive -- building upon existing NLANR measurement
collaborations with supercomputing centers, ISPs, universities, vendors, and government. CAIDA is also responsive and is designed to meet evolving and future needs of Internet by encouraging continued innovation by the R&E community, tempered by the realities of the commercial Internet infrastructure.
CAIDA is a collaborative undertaking to promote greater cooperation in the engineering and maintenance of a robust, scalable global Internet infrastructure. It addresses problems of Internet traffic measurement and performance and of inter-provider communication and cooperation within the Internet service industry. It provides a
neutral framework to support these cooperative endeavors.
CAIDA's initial goals include:
? Encourage the creation of Internet traffic metrics (in collaboration with IETF/IPPM
and other organizations); and work with industry, consumer, regulatory, and other
representatives to assure their utility and universal acceptance.
? Create a collaborative research and analytic environment in which various forms
of traffic data can be acquired, analyzed, and (as appropriate) shared.
? Foster the development of advanced methodologies and techniques for: traffic
performance and flow characterization, simulation, analysis, and visualization.
Specific areas of future impact include real-time routing instability diagnosis and
evolution for next generation measurement and routing protocols (multicast and
The goal is to have both government and industry participate in CAIDA, for the benefit
of the larger Internet environment.
CAIDA’s skitter[Skitter] is a tool for actively probing the Internet in order to analyze
topology and performance from approximately 16 skitter hosts around the world to
hundreds of thousands of destinations in IPv4 address space. [Murr01] It use 52-byte
ICMP echo request packets to monitor from an IP address belonging to one of CAIDA
hosts, so skitter is one of active measurement tools. CAIDA currently maintain the
skitter hosts in London, Japan (2), Singapore, Korea, New Zealand, Canada (Ottawa),
and the United States (NY, DC, Boulder, Ann Arbor, MAE-West (2), San Diego (2) and
Champaign-Urbana.) CAIDA analyze the data to visualize macroscopic topology and
performance attributes of a large cross-section of the Internet. Skitter’s initial goals
skitter records each hop from a source to many destinations. by incrementing
the "time to live" (TTL) of each IP packet header and recording replies from
each router (or hop) leading to the destination host.
skitter collects round trip time (RTT) along with path (hop) data. skitter uses
ICMP echo requests as probes to a list of IP destinations.
skitter data can provide indications of low-frequency persistent routing changes.
Correlations between RTT and time of day may reveal a change in either
forward or reverse path routing.
By probing the paths to many destinations IP addresses spread throughout the
IPv4 address space, skitter data can be used to visualize the directed graph
from a source to much of the Internet.
Based on standards form the IETF’s IPPM working group [IPPM], Surveyor measures
performance of Internet paths among participating organizations world-wide. [Murr01] The project is also developing methodologies and tools to analyze the gathered performance data. Data per site, per path, and per calendar day are available at Surveyor site. [Surv] The project aims to create the infrastructure and tools that will improve the ability to understand and effectively engineer the Internet infrastructure. The Surveyor project consistently measures end-to-end unidirectional delay, loss, and routing among a diverse set of measurement probes throughout the Internet.[Kali99] The goal of the project is to create technology and infrastructure to allow users and service providers (at all levels) to have an accurate common understanding of the performance and reliability of paths through the Internet. Surveyor measures the One-way Delay [RFC2679] and One-way Loss [RFC2680] metrics developed by the IPPM working group of the IETF. Surveyor is currently deployed at 51 sites in the higher education and research community, including universities, US national labs, and international sites.
The NIMI (National Internet Measurement Infrastructure) [NIMI] is a software system for building network measurement infrastructure. [Adams00] Measurements include bandwidth, loss and hop counts between host pairs. NIMI was designed to be scalable and dynamic. Its scalability comes from its ability to delegate NIMI probes to administration managers for configuration and coordination of measurements and data collection. A key NIMI design goal is scalability to potentially thousands of NIMI probes within a single infrastructure. NIMI is dynamic in its architectural support for third-party measurement modules. For example, the MINC (Multicast Inference of Network Characteristics) project has used NIMI to test and calibrate the use of edge measurements to infer performance characteristics of the interior of a network. Using the experiences gained from the NPD project, with the added knowledge that NIMI must be scalable to achieve the necessary goal of global saturation, NIMI was architected to be lightweight. This would allow for NIMI to be more portable, reaching a greater level of exposure on the operating systems that compose the Internet. Likewise,
NIMI must be flexible because NIMI must be able to incorporate new performance
measurement tools as they become available. Moreover, NIMI must be fully
programmable, in both adapting to needs of the initiator of a measurement and in
controlling access – NIMI must be able to support a diverse set of policies. And finally,
NIMI must be secure to quell concerns of gathering sensitive data.
The NIMI architecture was designed to facilitate coordination and control of wide-area
Internet measurements. Derived and expanded from Vern Paxson’s Network Probe
Daemon (NPD) work [Paxson96], NIMI offers a web interface for collected
measurements. [Murr01] The NIMI infrastructure consists of a set of dedicated
measurement servers (termed NIMI probes) running on a number of hosts in a network,
and measurement configuration and control software, which runs on separate hosts.
Each NIMI platform runs a measurement server whose job is to: authenticate
measurement requests as they arrive; check requests against the platform’s policy
table; queue them for future execution; execute them at the appropriate time; bundle
and ship the results of the requests to whatever destination the request specified (a
“DAC – Data Analysis Client”); and delete the results when instructed to. [Paxson00]
Internally, the NIMI probe is divided into two distinct daemons, nimid, which is
responsible for communication with the outside world and performing access control
checks, and scheduled, which does the actual measurement scheduling, execution, and
result packaging. The CPOC (Configuration point of contact) acts as the administrative
point of contact for a set of NIMI probes within the CPOC zone of control. The CPOC’s
job is simply to provide the initial policies for each unique NIMI probe, and over time,
provide updates to these policies.
End user who wishes to use the infrastructure does so via the Measurement Client (MC).
The MC is a Unix utility that can run on whatever workstation is convenient for the end
user, providing that the workstation has access to the user’s NIMI credentials. It
communicates directly with the NIMI probe involved in the measurement; the CPOC is
not involved in the processing of individual measurement requests.
The DAC acts as a repository and post-processor of the data returned by the NIMI
probe upon completion of a measurement. This component can actually be run from the
MC to collect immediate results, or as a daemon to collect ongoing measurements.
The following image depicts the flow of messages between NIMI componens: