On Dominant Characteristics of Residential Broadband Internet Traf?c
Gregor Maier TU-Berlin/T-Labs Anja Feldmann TU-Berlin/T-Labs Vern Paxson UC Berkeley/ICSI Mark Allman ICSI
While residential broadband Internet access is popular in many parts of the world, only a few studies have examined the characteristics of such traf?c. In this paper we describe observations from monitoring the network activity for more than 20,000 residential DSL customers in an urban area. To ensure privacy, all data is immediately anonymized. We augment the anonymized packet traces with information about DSL-level sessions, IP (re-)assignments, and DSL link bandwidth. Our analysis reveals a number of surprises in terms of the mental models we developed from the measurement literature. For example, we ?nd that HTTP?ªnot peer-to-peer?ªtraf?c dominates by a signi?cant margin; that more often than not the home user??s immediate ISP connectivity contributes more to the round-trip times the user experiences than the WAN portion of the path; and that the DSL lines are frequently not the bottleneck in bulk-transfer performance.
Categories and Subject Descriptors
C.2.2 [Computer-Communication Networks]: Network
Protocols?ªApplications; C.2.3 [Computer-Communication Networks]: Network Operations?ªNetwork monitoring
Network Measurement, Application Mix, HTTP usage, TCP performance, Residential Broadband Traf?c, DSL
ing with family and friends in myriad ways. However, the nature of the connectivity differs from previously studied environments such as campus networks and enterprises in salient ways. First, users of residential broadband connections will often have different goals than those in other environments, and are not subject to the same sorts of strict acceptable use policies that may regulate their access at work or at school, such as prohibitions against accessing certain Web sites or employing certain applications. In addition, we expect that the users who set up hosts and ancillary equipment in residences often have no expertise in system administration, nor much desire to understand any more than is necessary to ??make it work??. Finally, unlike for campuses (and to a lesser extent, enterprises), researchers rarely have
large-scale access to residential traf?c, and thus its makeup, dynamics, and variations remain underexamined. In this work we present observations developed from passive packet-level monitoring of more than 20,000 residential DSL lines from a major European ISP. This unique vantage point provides a broad view of residential traf?c, enabling more comprehensive and detailed characterizations than was possible in previous work, such as Cho et al.??s studies based on backbone traces [19, 9, 10], other work that examined speci?c applications like P2P-assisted content distribution  and Skype , or studies using active measurements . In this initial exploration we focus on studying a broad range of dominant characteristics of residential traf?c across a number of dimensions, including DSL session characteristics, network and transport-level features, prominent applications, and network path dynamics. Our study discovered a number of results we found surprising in terms of the standard ??mental models?? one develops from the Internet measurement literature and by talking with operators and colleagues. For example: ? HTTP traf?c, not peer-to-peer, dominates. Overall, HTTP makes up nearly 60% of traf?c by bytes while peer-to-peer contributes roughly 14%. Even if we assume that all unclassi?ed traf?c is peer-to-peer, this latter ?gure only rises to one-quarter, con?rming contemporaneous observations by Erman et al.  for a major US broadband provider. ? DSL sessions run quite short in duration, with a median length of only 20?C30 min. The short lifetime affects the rate of IP address reassignments, and we ?nd 50% of addresses are assigned at least twice in 24 h, and 1?C5% of addresses more than 10 times, with signi?cant implications for IP address aliasing.
Residential broadband Internet connectivity is a mature service in many countries. This foundation of rich access allows users to tightly integrate network use into their lives?ªfrom checking the weather or sports scores to shopping and banking to communicat-
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for pro?t or commercial advantage and that copies bear this notice and the full citation on the ?rst page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior speci?c permission and/or a fee. IMC??09, November 4?C6, 2009, Chicago, Illinois, USA. Copyright 2009 ACM 978-1-60558-770-7/09/11 ????$10.00.
Time Aug 08 Sep 08
Duration 14x 90 min 24 h
Size Loss 100?C600 none GB ea. >4 TB several multi-second periods with no packets >4 TB see above
Time Feb 2009 Jan?CFeb 2009
Duration 10 days 6x 24 h
Loss none none
Table 2: Summary of additional anonymized DSL session information
all sessions median duration per DSL line
Table 1: Summary of anonymized packet traces ? Delays experienced from a residence to the ISP??s Internet gateway often exceed those over the wide-area path from the gateway to the remote peer. We ?nd a median local component of 46 ms (due to DSL interleaving), versus a median remote component of 17 ms. ? Users rarely employ the full capacity of their lines, con?rming observations by Siekkinen et al. . 802.11 wireless networking in customers?? homes, and TCP settings on the residential systems, appear to limit the achievable throughput. We organize the paper as follows. After giving a short overview of our datasets and terminology in Section 2, we look at DSL session characteristics in Section 3. In Section 4 we explore which applications are popular among the user population, and take a closer look at the most predominant, HTTP, in Section 5. We brie?y examine transport protocol features in Section 6, and examine path characteristics in Section 7. We summarize in Section 8.
probability density 0.2 0.3
10 15 session duration [h]
Figure 1: PDF of session durations for sessions with duration longer than 5 minutes for dataset TEN. router, for which we employed Endace DAG network monitoring cards  for traf?c capture. Immediately after capture we extract application classi?cations (using DPD ; see Section 4.1) and information such as HTTP headers from the traces using Bro , storing anonymized versions of the packet and application headers for later processing. Table 1 provides an overview of the data
traces, including when gathered and overall size. WEEK re?ects 14 intervals of 90 minutes each, gathered twice per day during the same hours over the course of one week. In addition, we gathered anonymized DSL session information, including the session start and end times, anonymized IP address, anonymized line-card identi?er, and the con?gured access-bandwidth. Along with DSL session traces for each of our packet measurements, we obtained a 10-day DSL session-only trace from Jan 2009 (TEN), as well as six separate 24h session-only traces (see Table 2). To simplify the presentation, we focus our discussion on SEP and TEN. However, we veri?ed our results across all traces and explicitly point out differences. In particular, we use the 14 samples from WEEK to verify that there are no dominant day-of-week or other biases apparent in the 24 h traces (SEP, APR). In addition, we cross-checked our results with sampled NetFlow data exported by 10 of the ISP??s routers. This further increases our con?dence in the representativeness of our application mix results.
DATA AND TERMINOLOGY
We base our study on passive, anonymized packet-level observations of residential DSL connections collected at aggregation points within a large European ISP. Overall, the ISP has roughly 10 million (4%) of the 251 million worldwide broadband subscribers . They predominantly use DSL. The monitor operated at the broadband access router connecting customers to the ISP??s backbone. The access bandwidth of the monitored lines varies between 1,200/200 Kbps (downstream/upstream) and 17,000/1,200 Kbps, with the exact rate depending on both the customer??s contract and their distance from the DSLAM (the ISP??s line-card). In the portion of the network we monitored most users had distances low enough to in principle support 17 Mbps. For clarity of exposition, we de?ne the following terms. A line denotes a physical DSL line as identi?ed by a line-card identi?er. We de?ne a DSL-level session as the period when the DSL modem and the line-card are together in operation. We refer to the network between the monitoring point and the customer as the local side, as opposed to the remote side (remainder of the Internet). Similarly, the customer sends upstream traf?c and receives downstream traf?c. A ?ow refers to unidirectional data transmission at the usual 5-tuple granularity (IP addresses, transport protocol, transport ports). A connection is a bi-directional transport-level communication channel, demarked for TCP by the usual control packets (SYN, FIN/RST) and for UDP by the the arrival of the ?rst packet and the absence of activity detected using an idle timeout of 20 s. Finally, the originator endpoint actively initiated the connection, as opposed to the responder, which passively awaited the connection request. Our monitoring vantage point allowed us to observe
more than 20,000 DSL lines from one urban area, connected to one access
3. DSL SESSION CHARACTERISTICS
We begin our study with a look at the behavior of the users?? DSL sessions (periods of connection to the ISP??s network). A ?rst basic question concerns the durations of such connections. Network analysis studies often make the assumption that one can use IP addresses as host identi?ers (for example, for studies that count the number of systems exhibiting a particular phenomenon), and previous studies have found stability in these mappings on the order of several hours to days. Moore et al. analyzed the 2001 Code
SessionTimeout 7.2% PortError 7.7% Other 1.9% IdleTimeout 1.7% UserRequest 81.5%
Figure 2: DSL (Radius) session termination causes distribution for sessions lasting longer than 5 minutes.
Figure 4: Bandwidth usage of all DSL lines across time (1 min bins).
fraction of online DSL lines [%] 42 44 46 48 50
Figure 3: Relative number of concurrent DSL lines across time for one 24h weekday period of dataset TEN. Note the base-line. Red outbreak and found that for larger timescales (days to weeks), IP addresses cannot be used as reliable host identi?ers due to IP reassignment ; they did not examine timescales below several hours. Xie et al. observed some highly volatile dynamic IP address ranges, which they attributed mainly to dial-up hosts . Thus, we expected to ?nd typical session lengths of several hours. However, we ?nd instead that many are quite short. We base our analysis on Radius  logs, which many European ISPs use for authentication and IP address leasing. Radius supports two timeouts, SessionTimeout and IdleTimeout, though the monitored ISP only makes use of the ?rst. SessionTimeout performs a role similar to the DHCP lease time, limiting the maximum lifetime of a session. The
ISP sets it to 24 hr (a popular choice among European ISPs [52, 37]). DSL home routers generally offer an option to reconnect immediately after a session expires. However, in contrast to DHCP, Radius does not provide an option to request a particular IP address (e.g., the previously used IP address), and the ISP allows addresses to change across sessions. We analyzed the DSL session duration of the Radius logs, excluding sessions lasting under 5 minutes. Surprisingly, we ?nd that sessions are quite short, with a median duration of only 20?C30 minutes. Figure 1 shows the distribution of DSL session durations for
those longer than 5 minutes, computed over all sessions, along with the distribution of the median session duration computed per DSL line. The data exhibits two strong modes around 20?C30 minutes and 24 hr (the maximum duration given the Radius setup), partitioning the DSL lines in two large groups: always-connected lines, and lines that only connect on demand and disconnect shortly after. We do not ?nd much in between (lines connected for several hours). While previous work found short sessions (70% lasting at most 1 hour) in the context of wireless university networks , we found it striking to discover such short DSL sessions in residential networks, in violation of our mental model that sessions would be signi?cantly longer-lived. To check if there is a signi?cant difference in DSL session durations for P2P users vs. non-P2P users (see Section 4), we partitioned the DSL-lines into two groups. Overall, the characteristics of the distribution are similar, with two prevalent modes. However, we ?nd that P2P users tend to have longer session durations and that a largerfraction of P2P users always remain connected. To better understand the high prevalence of short sessions, we examined the Radius termination status in the logs. Radius differentiates between 18 termination causes. Figure 2 shows the distribution of causes for sessions longer than 5 minutes. We observe that more than 80% of sessions are terminated by user request (this rises to 95% for sessions under 5 minutes). Most likely these are caused by idle timeouts in the DSL modem on the client side. While most current broadband contracts are ?at-rate, in the past time-based contracts were popular in Europe. Indeed, these latter are still offered by most European ISPs. Therefore, it is likely that consumer DSL routers come with a small idle timeout as a factory default in an effort to aid users in keeping down costs, and we veri?ed this for several popular home routers. The second most common termination cause is PortError, which likely results when users power off their DSL modem as part of powering down their entire computing setup. Since many DSL sessions are short and Radius does not preserve IP address assignments across sessions, we therefore expect (and ?nd) IP addresses used for multiple DSL lines across each dataset. During a 24 hr period we ?nd 50% of the IP addresses assigned to at least 2 distinct DSL lines, and 1?C5% to more than 10
DSL lines. These results underscore the peril involved in using an IP address as a long-term reliable host identi?er.
Previous work found that for consumers diurnal patterns start with activity in the morning, steadily increasing throughout the course of the day, with the height of activity starting in the early evening and lasting till midnight [19, 17]. We see this same overall pattern in terms of the number of active DSL sessions, as shown in Figure 3. However, we note that the variation is in fact modest, with 40% of the lines permanently connected. We also observe a slight day-of-week effect, with Sundays having larger numbers of concurrent sessions, and Friday/Saturday having lower daily maxima than other weekdays. We also observe a diurnal pattern in bandwidth usage, per Figure 4, with the relative differences now being much more pronounced. After all, keeping a session alive does not imply any bandwidth usage per se. Our data also offers us an opportunity to analyze the potential resource requirements of an ISP wide NAT deployment. In particular, we study how many public IP addresses are needed to support the traf?c on the monitored lines. For this purpose we count the number of concurrently active TCP/UDP connections and add a 5min or 10-min timeout to the duration of each 5-tuple. Doing so implies that we do not allow the immediate reuse of each 5-tuple. Under the assumption that a single public IP address can support 65,536 concurrent connections (due to available port space) we ?nd that a single public IP address suf?ces to support 1,300?C2,000 active lines with a 10-min timeout, and roughly twice that when using a 5-min timeout. Given the maximum number of concurrently connected lines, 5?C 10 public addresses would in principle suf?ce to accommodate the monitored DSL-lines?ªa huge reduction of the required public IP address space. So far we only considered outgoing connections, yet a NAT must also accommodate incoming connections. We ?nd that very few lines utilize incoming connections for traditional services such as HTTP. Most successful incoming connections are to ports commonly used for VoIP (SIP and RTP), default P2P ports, IPSec key management, and traceroute destination ports. It is plausible that P2P applications can use Universal Plug-and-Play to dynamically negotiate ports with the NAT devices. SIP and RTP include NAT traversal solutions and proxy services. In addition, we ?nd that almost all SIP connections are to/from the ISP??s SIP server, since SIP is used as a transparent VoIP replacement for end-customers. Moreover, one does not have to support traceroute. As such it appears that one would not need too many additional public IP addresses for incoming connections. While we acknowledge that more in-depth study is needed, it appears that such NAT deployment would indeed conserve a very large number of public IP addresses. Whether it proves manageable,
and/or impedes innovation, remains a separate question.
otherDPD 10% BitTorrent 8.5% eDonkey 5% NNTP 4.8% well?known 3.6%
Figure 5: Application Mix for trace SEP. deep packet inspection and traf?c management systems at selected customers sites to assess the application usage [45, 46, 40]. Cachelogic claimed that by 2006 P2P accounted for more than 70% of the traf?c, with Ipoque supporting this claim for 2007. For 2008 Ipoque found that P2P in Europe accounted for more than 50% of traf?c (with Web contributing another 25%). On the other hand, Hyun-chul et al. reported that payload-based analysis conducted in 2004 from within the PAIX backbone found almost no P2P traf?c, but more than 45% HTTP . On the other hand, the same study developed how at various university networks the traf?c differs; for example, at KAIST in 2006 they found under 10% HTTP, and 40?C50% P2P. Cho et al. [9, 10] also found in 2008 that TCP port 80 contributed only 14% of all bytes in Japanese ISP backbones (9% in 2005), with the bulk of traf?c being on unassigned ports. None of the default P2P ports contributed more 1% of the traf?c volume. (The authors point out that WINNY, the most prelevant P2P application in Japan, uses unassigned ports.) They found that residential traf?c exhibited a shift to more streaming and video content, which agrees with recent blog and news reports that claim that P2P traf?c has somewhat declined, with streaming media increasing [50, 3]. With an assumption that the unassigned ports indeed re?ected P2P, their datasets indicated that P2P dominated the total traf?c volume. From a somewhat different perspective, Kotz and Essien [29, 30] reported that 50% of wireless traf?c in 2001 on a university campus, which included residential buildings, used HTTP??s wellknown ports, with 40% of this traf?c incoming to local servers. Henderson et al.  compared these results with newer traces from 2003/2004 of the same network, ?nding major shifts in the application mix (HTTP 63%?ú27%, File systems 5%?ú19%, P2P 5%?ú22%), and that more traf?c stayed on-campus than in 2001 (70%, up from 34%). Of the P2P traf?c, 73% remained internal. Therefore, we cannot easily compare these results to residential broadband use. Finally, Fraleigh et al.  also used a port-based approach on 2001 data, ?nding that on some links 60% of the bytes come from P2P and only 30% from HTTP, although most of their traces have more than 40% HTTP. Given this context, we now turn to an analysis of application usage in our 2008/2009 residential traces.
To understand the popular applications among our user population, we examine our application classi?cations (made at datacollection time)
and anonymized application-layer header traces. We in addition assess how well purely port-based classi?cation would perform for correctly identifying residential traf?c patterns, and characterize traf?c asymmetries. Previous studies of Internet application mix found HTTP to predominate around the turn of the century. Fraleigh et al.  analyzed packet level traces recorded from the Sprint backbone in 2001, ?nding that in most traces HTTP contributed > 40% of all bytes, though several traces had P2P contributing 80%. Subsequent studies found that P2P became the dominant application. Ipoque and Cachelogic both used data from their deployed
4.1 Application usage analysis
To robustly identify application protocols, we employ the Bro system??s Dynamic Protocol Detection (DPD) . DPD essentially tries to parse each byte stream with parsers for numerous protocols, deferring determination of the corresponding application until only that application??s parser recognizes the traf?c. DPD
also uses regular expression signatures to winnow down the initial set of candidate parsers. The Bro distribution includes full DPD parsers/recognizers for BitTorrent, FTP, HTTP, IRC, POP3, SMTP, SSH, and SSL. We extended the set of detectors with partial recognizers for eDonkey and Gnutella (both based on L7-?lter signatures ), NNTP, RTP, RTSP, SHOUTcast, SOCKS, and Skype. In the SEP trace we can classify more than 85% of all bytes, with another 3.6% using well-known ports, as re?ected in Figure 5. We ?nd that HTTP, not P2P, is the most signi?cant protocol, accounting for 57% of residential bytes. We also ?nd that NNTP contributes a signi?cant amount of volume, nearly 5%. Almost all of the NNTP bytes arise due to transfers of binary ?les, with RARarchives (application/rar) being among the most common ?le types, suggesting that the traf?c re?ects the equivalent of ?le-sharing. We ?nd that P2P applications?ªBitTorrent, Gnutella, and eDonkey?ªcontribute < 14% of all bytes, with BitTorrent the most prevalent, and Gnutella almost non-existent. However, the L7-?lter signatures for eDonkey may be incomplete. We observe a significant amount of traf?c (1.2%) on well-known eDonkey ports that the classi?er fails to detect as eDonkey. The distribution of connection sizes for this traf?c closely matches that for traf?c positively identi?ed as eDonkey (and differs from other applications). If we presume that this indeed re?ects eDonkey traf?c, then the overall share of P2P traf?c increases to 17?C19%, with eDonkey??s popularity roughly the same as BitTorrent??s. But even if we assume that all unclassi?ed traf?c is P2P, the total P2P share still runs below 25%. P2P applications could also in principle use HTTP for data download, thus ??hiding?? among the bulk of HTTP traf?c and increasing the signi?cance of P2P traf?c volume. However, our indepth analysis of HTTP traf?c (Section 5) ?nds that this is not the case.
Streaming protocols1 (RTSP, RTMP, SHOUTcast) account for 5% of the traf?c in terms of bytes. We identify RTSP and SHOUTcast using partial DPD parsers, while we identify RTMP??s based only on its well-known port. We also ?nd noticeable Voice-over-IP traf?c (Skype , RTP), about 1.3% of the total bytes. In order to increase our con?dence in the representativeness of our application mix results, we analyzed sampled NetFlow data exported by 10 of the ISP??s routers. This data shows that 50% of the traf?c comes from TCP port 80. We further compared our results with those from a commercial deep-packet-inspection system deployed at a different network location, ?nding a close match. Our analysis of the other traces con?rms the ?ndings outlined above. In particular the other traces con?rm that our results are not biased by the day-of-week we choose. However, while the HTTP traf?c share in the APR trace is about the same, we ?nd slightly more unclassi?ed traf?c. We note that the overall P2P traf?c decreases somewhat, and shifts from eDonkey to BitTorrent (now 9.3%). Also the fraction of NNTP traf?c decreases. On this day it only accounted for 2.2% of the traf?c. Our hypothesis is that especially the latter observations re?ect day-to-day variations rather than indications of trends, but we will require longer-time measurements to determine this de?nitively. We might expect that application usage differs widely between users with different access speeds. Figure 6 shows the application mix seen for different downstream bandwidth rates. Although the mix does vary, the changes are modest, other than for more P2P traf?c with higher bandwidths, and much higher NNTP prevalence for the 17000 Kbps class. However, only a small percentage of lines use NNTP, so its contribution to traf?c mix can see more variation across different types of lines.
1 We do not consider video delivery via HTTP as streaming. We refer to those as progressive HTTP downloads.
percent 40 60
2300 3500 6500 access bandwidth [Kbps]
HTTP BitTorrent eDonkey NNTP otherDPD well?known unclassified
Figure 6: Relative application mix per access bandwidth. Bottom bar is HTTP, top bar unclassi?ed.
However, we do ?nd that lines with higher access bandwidth have a higher utilization in terms of average volume per line. Lines in the 3500 and 6500 Kbps categories contribute about twice as many bytes per line than lines in the 1200 Kbps class, and 17,000 Kbps lines three