summaryrefslogtreecommitdiff
path: root/doc/rfc/rfc6057.txt
diff options
context:
space:
mode:
Diffstat (limited to 'doc/rfc/rfc6057.txt')
-rw-r--r--doc/rfc/rfc6057.txt1627
1 files changed, 1627 insertions, 0 deletions
diff --git a/doc/rfc/rfc6057.txt b/doc/rfc/rfc6057.txt
new file mode 100644
index 0000000..db22db2
--- /dev/null
+++ b/doc/rfc/rfc6057.txt
@@ -0,0 +1,1627 @@
+
+
+
+
+
+
+Internet Engineering Task Force (IETF) C. Bastian
+Request for Comments: 6057 T. Klieber
+Category: Informational J. Livingood
+ISSN: 2070-1721 J. Mills
+ R. Woundy
+ Comcast
+ December 2010
+
+
+ Comcast's Protocol-Agnostic Congestion Management System
+
+Abstract
+
+ This document describes the congestion management system of Comcast
+ Cable, a large cable broadband Internet Service Provider (ISP) in the
+ U.S. Comcast completed deployment of this congestion management
+ system on December 31, 2008.
+
+Status of This Memo
+
+ This document is not an Internet Standards Track specification; it is
+ published for informational purposes.
+
+ This document is a product of the Internet Engineering Task Force
+ (IETF). It represents the consensus of the IETF community. It has
+ received public review and has been approved for publication by the
+ Internet Engineering Steering Group (IESG). Not all documents
+ approved by the IESG are a candidate for any level of Internet
+ Standard; see Section 2 of RFC 5741.
+
+ Information about the current status of this document, any errata,
+ and how to provide feedback on it may be obtained at
+ http://www.rfc-editor.org/info/rfc6057.
+
+Copyright Notice
+
+ Copyright (c) 2010 IETF Trust and the persons identified as the
+ document authors. All rights reserved.
+
+ This document is subject to BCP 78 and the IETF Trust's Legal
+ Provisions Relating to IETF Documents
+ (http://trustee.ietf.org/license-info) in effect on the date of
+ publication of this document. Please review these documents
+ carefully, as they describe your rights and restrictions with respect
+ to this document. Code Components extracted from this document must
+ include Simplified BSD License text as described in Section 4.e of
+ the Trust Legal Provisions and are provided without warranty as
+ described in the Simplified BSD License.
+
+
+
+Bastian, et al. Informational [Page 1]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+Table of Contents
+
+ 1. Introduction ....................................................2
+ 2. Applicability to Other Types of Networks ........................3
+ 3. Key Terminology .................................................3
+ 4. Historical Overview .............................................7
+ 5. Summary .........................................................8
+ 6. Relationship between Managing Congestion and Adding Capacity ....9
+ 7. Implementation and Configuration ...............................10
+ 7.1. Thresholds for Determining When a CMTS Port Is in a Near
+ Congestion State ..........................................14
+ 7.2. Thresholds for Determining When a User Is in an
+ Extended High Consumption State and for Release from
+ That Classification .......................................15
+ 7.3. Effect of BE Quality of Service on Users'
+ Broadband Experience ......................................19
+ 7.4. Equipment/Software Used and Location ......................21
+ 8. Conclusion .....................................................23
+ 9. Exceptional Network Utilization Considerations .................23
+ 10. Limitations of This Congestion Management System ..............24
+ 11. Low Extra Delay Background Transport and Other Possibilities ..24
+ 12. Security Considerations .......................................24
+ 13. Acknowledgements ..............................................25
+ 14. Informative References ........................................26
+
+1. Introduction
+
+ Comcast Cable is a large broadband Internet Service Provider (ISP),
+ based in the U.S., serving the majority of its customers via cable
+ modem technology. During the late part of 2008, and completing on
+ December 31, 2008, Comcast deployed a new congestion management
+ system across its entire network. This new system was developed in
+ response to dissatisfaction in the Internet community as well as
+ complaints to the U.S. Federal Communications Commission (FCC)
+ regarding Comcast's old system, which targeted specific peer-to-peer
+ (P2P) applications. This new congestion management system is
+ protocol-agnostic, meaning that it does not examine or impact
+ specific user applications or network protocols, which is perceived
+ as a more fair system for managing network resources at limited times
+ when congestion may occur.
+
+ It is important for readers to note that congestion can occur in any
+ IP network, and, when it does, packets can be delayed or dropped. As
+ Bob Briscoe has pointed out on an IETF mailing list, some amount of
+ packet loss can be normal and/or tolerable, noting "But a single TCP
+ flow with a round trip time (RTT) of 80 ms can attain 50 Mbps with a
+ loss fraction of 0.0013% (1 in ~74,000 packets) so there's no need to
+ try to achieve loss figures much lower than this. And indeed, if
+
+
+
+Bastian, et al. Informational [Page 2]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ flows aren't bottlenecked elsewhere, TCP will drive the system until
+ it gets such loss levels. If, instead, a customer is downloading
+ five separate 10 Mbps TCP flows still with an 80-ms RTT, TCP will
+ drive losses up to 1 in ~3,000, or 0.03%, and any lower loss rates
+ won't be able to improve performance". As a result, applications and
+ protocols have been designed to deal with the reality that congestion
+ can occur in any IP network, the mechanics of which we explain in
+ detail later in this document.
+
+ The purpose of this document is to describe how this example of a
+ large-scale congestion management system functions. This is
+ partially in response to questions from other ISPs as well as
+ solution developers, who are interested in learning from and/or
+ deploying similar systems in other networks. In addition, it is
+ hoped that such a document may help inform new work in the IETF, in
+ the hope that better systems and protocols may be possible in the
+ future. Lastly, the authors wish to transparently and openly
+ document this system, so that there could be no doubt about how the
+ system functioned.
+
+2. Applicability to Other Types of Networks
+
+ Several document reviewers and other IETF participants have pointed
+ out that, though we refer to functional elements that are specific to
+ a Data Over Cable Service Interface Specification (DOCSIS)-based
+ network implementation, this type of congestion management system
+ could be generally applied to nearly any type of network. Thus, it
+ is important for readers to take note of this and take into
+ consideration that this sort of protocol-agnostic congestion
+ management system could certainly fit in a wide variety of network
+ types and implementations.
+
+3. Key Terminology
+
+ This section defines the key terms used in this document. Some terms
+ below refer to elements of the Comcast network. As a result, it may
+ be helpful to refer to Figure 1 (see Section 7) when reviewing some
+ of these terms.
+
+3.1. Cable Modem
+
+ A device located at the customer premise used to access the Comcast
+ High Speed Internet (HSI) network. In some cases, the cable modem is
+ owned by the customer, and in other cases it is owned by the cable
+ operator. This device has an interface (i.e., someplace to plug in a
+ cable) for connecting the coaxial cable provided by the cable company
+ to the modem, as well as one or more interfaces for connecting the
+ modem to a customer's PC or home gateway device (e.g., home gateway,
+
+
+
+Bastian, et al. Informational [Page 3]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ router, firewall, access point, etc.). In some cases, the cable
+ modem function, i.e., the ability to access the Internet, is
+ integrated into a home gateway device or Embedded Multimedia Terminal
+ Adapter (eMTA). Once connected, the cable modem links the customer
+ to the HSI network and ultimately the broader Internet.
+
+3.2. Cable Modem Termination System (CMTS)
+
+ A piece of hardware located in a cable operator's local network
+ (generally in a "headend", Section 3.10) that acts as the gateway to
+ the Internet for cable modems in a particular geographic area. A
+ simple way to think of the CMTS is as a router with interfaces on one
+ side leading to the Internet and interfaces on the other connecting
+ to Optical Nodes and then customers, in a so-called "last mile"
+ network.
+
+3.3. Cable Modem Termination System (CMTS) Port
+
+ Also referred to simply as a "port". A port is a physical interface
+ on a device used to connect cables in order to connect with other
+ devices for transferring information/data. An example of a physical
+ port is a CMTS port. A CMTS has both upstream and downstream network
+ interfaces to serve the local access network, which are referred to
+ as upstream or downstream ports. A port generally serves a
+ neighborhood of hundreds of homes. Over time, CMTS ports tend to
+ serve fewer and fewer homes, as the network is segmented for capacity
+ growth purposes. Prior to DOCSIS version 3, a single CMTS physical
+ port was used for either transmitting or receiving data downstream or
+ upstream to a given neighborhood. With DOCSIS version 3, and the
+ channel bonding feature, multiple CMTS physical ports can be combined
+ to create a virtual port. A CMTS is also briefly defined in
+ Section 2.6 of [RFC3083].
+
+3.4. Channel Bonding
+
+ A technique for combining multiple downstream and/or upstream
+ channels to increase customers' download and/or upload speeds,
+ respectively. Multiple channels from the Hybrid Fiber Coax (HFC)
+ network (Section 3.11) can be bonded into a single virtual port
+ (called a bonded group), which acts as a large single channel or port
+ to provide increased speeds for customers. Channel bonding is a
+ feature of Data Over Cable Service Interface Specification (DOCSIS)
+ version 3, as described in [DOCSIS_MULPI].
+
+
+
+
+
+
+
+
+Bastian, et al. Informational [Page 4]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+3.5. Coaxial Cable (Coax)
+
+ A type of cable used by a cable operator to connect customer premise
+ equipment (CPE) -- such as TVs, cable modems (including eMTAs), and
+ Set Top Boxes -- to the HFC network. This cable may be used within
+ the home as well as in segments of the "last mile" network running to
+ a home or customer premise location. There are many grades of
+ coaxial cable that are used for different purposes. Different types
+ of coaxial cable are used for different purposes on the network.
+
+3.6. Comcast High Speed Internet (HSI)
+
+ A service/product offered by Comcast for delivering Internet service
+ over a broadband connection.
+
+3.7. Customer Premise Equipment (CPE)
+
+ Any device that resides at the customer's residence, connected to the
+ Comcast network, whether controlled by Comcast or not.
+
+3.8. Data Over Cable Service Interface Specification (DOCSIS)
+
+ A reference standard developed by CableLabs that specifies how
+ components on cable networks need to be built to enable HSI service
+ over an HFC network, as noted in [DOCSIS_CM2CPE], [DOCSIS_PHY],
+ [DOCSIS_MULPI], [DOCSIS_SEC], and [DOCSIS_OSSI]. These standards
+ define the specifications for the cable modem and the CMTS such that
+ any DOCSIS-certified cable modem will work on any DOCSIS-certified
+ CMTS, independent of the selected vendor. The interoperability of
+ cable modems and CMTSs allows customers to purchase a DOCSIS-
+ certified modem from a retail outlet and use it on their cable-
+ networked home. All DOCSIS-related standards are available to the
+ public at the CableLabs website, at http://www.cablelabs.com.
+
+3.9. Downstream
+
+ Description of the direction in which a signal travels, in this case
+ from the network to a user. Downstream traffic occurs when users are
+ downloading something from the Internet, such as watching a web-based
+ video, reading web pages, or downloading software updates.
+
+3.10. Headend
+
+ A cable facility responsible for receiving TV signals for
+ distribution over the HFC network to the end customers. This
+ facility typically also houses one or more CMTSs. This is sometimes
+ also called a "hub".
+
+
+
+
+Bastian, et al. Informational [Page 5]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+3.11. Hybrid Fiber Coax (HFC)
+
+ A network architecture used primarily by cable companies, comprised
+ of fiber-optic and coaxial cables that currently deliver Voice,
+ Video, and Internet services to customers, as defined in Section 1.2
+ of [DOCSIS_MULPI].
+
+3.12. Internet Protocol Detail Record (IPDR)
+
+ Standardized technology for monitoring and/or recording subscribers'
+ upstream and downstream Internet usage data based on their cable
+ modem. The data is collected from the CMTS and sent to a server for
+ further processing. Additional information is available at
+ http://www.ipdr.org, as well as [IPDR_Standard] and [DOCSIS_IPDR].
+
+3.13. Optical Node
+
+ A component of the HFC network generally located in customers' local
+ neighborhoods that is used to convert the optical signals sent over
+ fiber-optic cables to electrical signals that can be sent over
+ coaxial cable to customers' cable modems, or vice versa. A fiber-
+ optic cable connects the Optical Node, through distribution hubs, to
+ the CMTS, and coaxial cable connects the Optical Node to customers'
+ cable modems.
+
+3.14. Provisioned Bandwidth
+
+ The peak speed associated with a tier of service purchased by a
+ customer. For example, a customer with a 105 Mbps downstream and
+ 10 Mbps upstream speed tier would be said to be provisioned with
+ 105 Mbps of downstream bandwidth and 10 Mbps of upstream bandwidth.
+ This is often referred to as 105/10 service in industry parlance.
+
+ The Provisioned Bandwidth is the speed that a customer's modem is
+ configured (and the network is engineered) to deliver on a regular
+ basis (which is not the same as a "Committed Information Rate" or a
+ guaranteed rate). Internet speeds are generally a best effort
+ service that are dependent on a number of variables, many of which
+ are outside the control of an Internet Service Provider (ISP). In
+ general, speeds do not typically exceed a customer's provisioned
+ speed. Comcast, however, invented a technology called "PowerBoost"
+ [PowerBoost_Specification] that, for example, enables users to
+ experience brief boosts above their provisioned speeds while they
+ transfer large files over the Internet, by utilizing excess capacity
+ that may be available in the network at that time.
+
+
+
+
+
+
+Bastian, et al. Informational [Page 6]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+3.15. Quality of Service (QoS)
+
+ A set of techniques to manage network resources to ensure a level of
+ performance to specific data flows, as described in [RFC1633] and
+ [RFC2475]. One method for providing QoS to a network is by
+ differentiating the type of traffic by class or flow and assigning
+ priorities to each type. When the network becomes congested, the
+ data packets that are marked as having higher priority will have
+ higher likelihood of being serviced.
+
+3.16. Upstream
+
+ Description of the direction in which a signal travels, in this case
+ from the user to the network. Upstream traffic occurs when users are
+ uploading something to the network, such as sending email, sending
+ files to another computer, or uploading photos to a digital photo
+ website.
+
+4. Historical Overview
+
+ Comcast began the engineering project to develop a new congestion
+ management system in March 2008, the same month that Comcast hosted
+ the 71st meeting of the IETF in Philadelphia, PA, USA. On May 28,
+ 2008, Comcast participated in an IETF Peer-to-Peer Infrastructure
+ Workshop [RFC5594], hosted by the Massachusetts Institute of
+ Technology (MIT) in Cambridge, MA, USA.
+
+ In order to participate in this workshop, interested attendees were
+ asked to submit a paper to a technical review team, which Comcast did
+ on May 9, 2008, in [COMCAST_P2PI_PAPER]. Comcast subsequently
+ attended and participated in this valuable workshop. During the
+ workshop, Comcast outlined the high-level design for a new congestion
+ management system [COMCAST_P2PI_PRES] and solicited comments and
+ other feedback from attendees and other members of the Internet
+ community (presentations were also posted to the IETF's P2Pi mailing
+ list). The congestion management system outlined in that May 2008
+ workshop was later tested in trial markets and is in essence what was
+ then deployed by Comcast later in 2008.
+
+ Following an August 2008 FCC document [FCC_Memo_Opinion] regarding
+ how Comcast managed congestion on its High-Speed Internet ("HSI")
+ network, Comcast disclosed to the FCC [FCC_Net_Mgmt_Response] and the
+ public additional technical details of the congestion management
+ system that it intended to and did implement by the end of 2008
+ [FCC_Congest_Mgmt_Ltr], including the thresholds involved in this new
+
+
+
+
+
+
+Bastian, et al. Informational [Page 7]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ system. While the description of how this system is deployed in the
+ Comcast network is necessarily specific to the various technologies
+ and designs specific to that network, a similar system could be
+ deployed on virtually any large-scale ISP network or other IP
+ network.
+
+5. Summary
+
+ Comcast's HSI network has elements that are shared across many
+ subscribers. This means that Comcast's HSI customers share upstream
+ and downstream bandwidth with their neighbors. Although the
+ available bandwidth is substantial, so, too, is the demand. Thus,
+ when a relatively small number of customers in a neighborhood place
+ disproportionate demands on network resources, this can cause
+ congestion that degrades their neighbors' Internet experience. The
+ goal of Comcast's new congestion management system is to enable all
+ users of our network resources to access a "fair share" of that
+ bandwidth, in the interest of ensuring a high-quality online
+ experience for all of Comcast's HSI customers.
+
+ Importantly, the new approach is protocol-agnostic; that is, it does
+ not manage congestion by focusing on the use of the specific
+ protocols that place a disproportionate burden on network resources,
+ or any other protocols. Rather, the new approach focuses on managing
+ the traffic of those individuals who are using the most bandwidth at
+ times when network congestion threatens to degrade subscribers'
+ broadband experience and who are contributing disproportionately to
+ such congestion at those points in time.
+
+ Specific details about these practices, including relevant threshold
+ information, the type of equipment used, and other particulars, are
+ discussed at some length later in this document. At the outset,
+ however, we present a very high-level, simplified overview of how
+ these practices work. Despite all the detail provided further below,
+ the fundamentals of this approach can be summarized succinctly:
+
+ 1. Software installed in the Comcast network continuously examines
+ aggregate traffic usage data for individual segments of Comcast's
+ HSI network. If overall upstream or downstream usage on a
+ particular segment of Comcast's HSI network reaches a
+ pre-determined level, the software moves on to step two.
+
+ 2. At step two, the software examines bandwidth usage data for
+ subscribers in the affected network segment to determine which
+ subscribers are using a disproportionate share of the bandwidth.
+
+
+
+
+
+
+Bastian, et al. Informational [Page 8]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ If the software determines that a particular subscriber or
+ subscribers have been the source of high volumes of network
+ traffic during a recent period of minutes, traffic originating
+ from that subscriber or those subscribers temporarily will be
+ assigned a lower priority status.
+
+ 3. During the time that a subscriber's traffic is assigned the lower
+ priority status, their packets will not be delayed or dropped so
+ long as the network segment is not actually congested. If,
+ however, the network segment becomes congested, their packets
+ could be intermittently delayed or dropped.
+
+ 4. The subscriber's traffic returns to normal priority status once
+ his or her bandwidth usage drops below a set threshold over a
+ particular time interval.
+
+ Comcast undertook considerable effort, over the course of many
+ months, to formulate our plans for this congestion management
+ approach, adjusting them, and subjecting them to real-world trials.
+ Market trials were conducted in Chambersburg, PA; Warrenton, VA; Lake
+ City, FL; East Orange, FL; and Colorado Springs, CO, between June and
+ September 2008. This enabled us to validate the utility of the
+ general approach and collect substantial trial data to test multiple
+ variations and alternative formulations.
+
+6. Relationship between Managing Congestion and Adding Capacity
+
+ Many people have questioned whether congestion should ever exist at
+ all, if an ISP was adding sufficient capacity. There is certainly a
+ relationship between capacity and congestion. But there are two
+ types of congestion that generally present themselves in a network.
+
+ The first general type of congestion is regularly occurring and is
+ the result of gradually increasing traffic levels up to a point where
+ typical usage peaks cause congestion on a regular basis. Comcast,
+ like many ISPs, has a set capacity management process by which
+ capacity additions are automatically triggered based on certain usage
+ trends; this process is geared towards bringing additional capacity
+ to the network prior to the onset of regularly occurring congestion.
+ As such, capacity is added when needed and before it presents
+ noticeable effects. This process is in place since capacity
+ additions are not instantaneous and in many cases require significant
+ physical work.
+
+ The second general type of congestion is unpredictable congestion,
+ which can occur for a wide range of reasons. One example may be due
+ to current events, where users may be all rushing to access specific
+ content at the exact same time, and where the systems serving that
+
+
+
+Bastian, et al. Informational [Page 9]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ content may not be able to keep up with demand. Another example may
+ be due to a localized disaster, where some network paths have been
+ destroyed or otherwise impaired, and where many users are attempting
+ to communicate with one another at traffic levels significantly above
+ normal.
+
+ Thus, in both cases, even with continuous upgrades and constant
+ investment in additional capacity, the fact remains that network
+ capacity is not unlimited. A congestion management system, absent
+ superior protocol-based solutions that do not currently exist, can
+ therefore help manage the effects of congestion on users, improving
+ their Internet experience.
+
+7. Implementation and Configuration
+
+ It is important to note that the implementation details below and the
+ overall design of the system are matched to traffic patterns that
+ exist on the Internet today and that the authors believe will exist
+ in the near future. While the authors desired to make the system
+ highly adaptable and a good long-term network investment, significant
+ changes in such traffic patterns may necessitate a change in the
+ configuration of the system or, in extreme cases, a different type of
+ system altogether.
+
+ To understand exactly how these new congestion management practices
+ work, it is helpful to have a general understanding of how Comcast's
+ HSI network is designed. Comcast's HSI network is what is commonly
+ referred to as a hybrid fiber-coax network, with coaxial cable
+ connecting each subscriber's cable modem to an Optical Node, and
+ fiber-optic cables connecting the Optical Node, through distribution
+ hubs, to the Cable Modem Termination System (CMTS), which is also
+ known as a "data node". The CMTSs are then connected to higher-level
+ routers, which in turn are connected to Comcast's Internet backbone
+ facilities. Today, Comcast has over 3,200 CMTSs deployed throughout
+ our network, serving over 15 million HSI subscribers.
+
+ Each CMTS has multiple "ports" that handle traffic coming into and
+ leaving the CMTS. In particular, each cable modem deployed on the
+ Comcast HSI network is connected to the CMTS through the ports on the
+ CMTS. These ports can be either "downstream" ports or "upstream"
+ ports, depending on whether they send information to cable modems
+ (downstream) or receive information from cable modems (upstream)
+ attached to the port. (Note that the term "port" as used here
+ generally contemplates single channels on a CMTS, but these
+ statements will apply to virtual channels, also known as "bonded
+
+
+
+
+
+
+Bastian, et al. Informational [Page 10]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ groups", in a DOCSIS 3.0 environment.) Even without channel bonding,
+ multiple channels are usually configured to come out of each physical
+ port. Said another way, there is generally a mapping of multiple
+ channels to each physical port.
+
+ Currently, on average, approximately 275 cable modems share the same
+ downstream port, and about 100 cable modems share the same upstream
+ port; however, this is constantly changing (both numbers generally
+ become smaller over time, based on current DOCSIS technology). Both
+ types of ports can experience congestion that could degrade the
+ broadband experience of our subscribers and, unlike with the previous
+ congestion management practices, both upstream and downstream traffic
+ are subject to management in this new congestion management system.
+
+ Based upon the design of the network and traffic patterns observed,
+ the most likely place for congestion to occur is on these CMTS ports.
+ As a result, the congestion management system measures the traffic
+ conditions of CMTS ports, and applies any policy actions to traffic
+ on those ports (rather than some other, more distant segment of the
+ network).
+
+ To implement Comcast's new protocol-agnostic congestion management
+ practices, Comcast purchased new hardware and software that were
+ deployed near the Regional Network Routers ("RNRs") that are further
+ upstream in Comcast's network. This new hardware consists of
+ Internet Protocol Detail Record ("IPDR") servers, Congestion
+ Management servers, and PacketCable Multimedia ("PCMM") servers.
+ Further details about each of these pieces of equipment can be found
+ below, in Section 7.4. It is important to note here, however, that
+ even though the physical location of these servers is at the RNR, the
+ servers communicate with -- and manage individually -- multiple ports
+ on multiple CMTSs to effectuate the practices described in this
+ document. That is to say, bandwidth usage on one CMTS port will have
+ no effect on whether the congestion management practices described
+ herein are applied to a subscriber on a different CMTS port.
+
+ Figure 1 provides a simplified graphical depiction of the network
+ architecture just described:
+
+
+
+
+
+
+
+
+
+
+
+
+
+Bastian, et al. Informational [Page 11]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ Figure 1: Simplified Network Diagram Showing High-Level Comcast
+
+ Network and Servers Relevant to Congestion Management
+
+ -------------------------
+ / \
+ | Comcast Internet Backbone |
+ \ -----
+ +------------+ --------------------/ \
+ | Congestion | / \
+ | Management |<+++GigE++++ +---->| Internet |
+ | Server | + | | Backbone |
+ +------------+ + | \ Router /
+ + Fiber \ /
+ +------------+ + | -----
+ | QoS | + |
+ | Server |<+++GigE++++ \/
+ | | + -----
+ +------------+ + / \
+ + / \
+ +------------+ + | Regional |
+ | Statistics | +++++++>| Network |
+ | Collection |<+++GigE++++ | Router |
+ | Server | \ /
+ +------------+ +---Fiber------>\ /<------Fiber----+
+ | ----- |
+ \/ \/
+ ----- -----
+ / \ / \
+ / Local \ / Local \
+ | Market | | Market |
+ \ Router / \ Router /
+ +--------->\ /<------------+ \ /
+ | ----- | ------
+ | /\ | /\
+ Fiber | Fiber |
+ | Fiber | Fiber
+ | | | |
+ \/ \/ \/ \/
+ /------\ /------\ /------\ /------\
+ | CMTS | | CMTS | | CMTS | | CMTS |
+ \------/ \------/ \------/ \------/
+ /\ /\ /\ /\
+ | | | |
+ Fiber Fiber Fiber Fiber
+ | | | |
+ \/ \/ \/ \/
+
+
+
+
+Bastian, et al. Informational [Page 12]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ +---------+ +---------+ +---------+ +---------+
+ | Optical | | Optical | | Optical | | Optical |
+ | Node | | Node | | Node | | Node |
+ +---------+ +---------+ +---------+ +---------+
+ /\ /\ /\ /\ /\ /\
+ || || ||______ || _____|| ||
+ Coax Coax |__Coax| Coax |Coax__| Coax
+ || || || || || ||
+ \/ \/ \/ \/ \/ \/
+ +=======+ +=======+ +=======+ +=======+ +=======+ +=======+
+ = Cable = = Cable = = Cable = = Cable = = Cable = = Cable =
+ = Modem = = Modem = = Modem = = Modem = = Modem = = Modem =
+ +=======+ +=======+ +=======+ +=======+ +=======+ +=======+
+
+ ================================================================
+ + Note: This diagram is a simplification of the actual network +
+ + and servers, which in actuality includes significant +
+ + redundancy and other details too complex to represent here. +
+ ================================================================
+
+ Figure 1
+
+ Each Comcast HSI subscriber's cable modem has a "bootfile", which is
+ essentially a configuration file that contains certain pieces of
+ information about the subscriber's service to ensure that the service
+ functions properly. (Note: No personal information is included in
+ the bootfile; it only includes information about the service that the
+ subscriber has purchased.) For example, the bootfile contains
+ information about the maximum speed (what we refer to in this
+ document as the "provisioned bandwidth") that a particular modem can
+ achieve based on the tier (personal/residential, commercial, etc.)
+ the customer has purchased. Bootfiles are generally reset from time
+ to time to account for changes in the network and other updates, and
+ this is usually done through a command sent from the network and
+ without the subscriber noticing. In preparation for the transition
+ to this new congestion management system, Comcast sent new bootfiles
+ to our HSI customers' cable modems that created two Quality of
+ Service (QoS) levels for Internet traffic going to and from the cable
+ modem: (1) "Priority Best Effort" ("PBE") traffic; and (2) "Best
+ Effort" ("BE") traffic. As with previous changes to cable modem
+ bootfiles, the replacement of the old bootfile with the new bootfile
+ requires no active participation by Comcast customers.
+
+ Thereafter, all traffic going to or coming from cable modems on the
+ Comcast HSI network is designated as either PBE or BE. PBE is the
+ default status for all Internet traffic coming from or going to a
+ particular cable modem. Traffic is designated BE for a particular
+ cable modem only when both of two conditions are met:
+
+
+
+Bastian, et al. Informational [Page 13]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ o First, the usage level of a particular upstream or downstream port
+ of a CMTS, as measured over a particular period of time, must be
+ nearing the point where congestion could degrade users'
+ experience. We refer to this as the "Near Congestion State" and,
+ based on the technical trials we have conducted (further validated
+ in our full deployment), we have established a threshold,
+ described in more detail below, for when a particular CMTS port
+ enters that state.
+
+ o Second, a particular subscriber must be making an extended, high
+ contribution to the bandwidth usage on the particular port,
+ relative to the service tier they purchased, as measured over a
+ particular period of time. We refer to this as the "Extended High
+ Consumption State" and, based on the technical trials we have
+ conducted (further validated in our full deployment), we have
+ established a threshold, described in more detail below, for when
+ a particular user enters that state.
+
+ When, and only when, both conditions are met, a user's upstream or
+ downstream traffic (depending on which type of port is in the Near
+ Congestion State) is designated as BE. Then, to the extent that
+ actual congestion occurs, any delay resulting from the congestion
+ will affect BE traffic before it affects PBE traffic.
+
+ We now explain the foregoing in greater detail in the following
+ sections.
+
+7.1. Thresholds for Determining When a CMTS Port Is in a Near
+ Congestion State
+
+ For a CMTS port to enter the Near Congestion State, traffic flowing
+ to or from that CMTS port must exceed a specified level (the "Port
+ Utilization Threshold") for a specific period of time (the "Port
+ Utilization Duration"). The Port Utilization Threshold on a CMTS
+ port is measured as a percentage of the total aggregate upstream or
+ downstream bandwidth for the particular port during the relevant
+ timeframe. The Port Utilization Duration on the CMTS is measured in
+ minutes.
+
+ Values for each of the thresholds that are used as part of this
+ congestion management technique have been tentatively established
+ after an extensive process of lab tests, simulations, technical
+ trials, vendor evaluations, customer feedback, and a third-party
+ consulting analysis. In the same way that specific anti-spam or
+ other network management practices are adjusted to address new issues
+ that arise, it is a near certainty that these values will change over
+ time, as Comcast gathers more data and performs additional analysis
+ resulting from wide-scale use of the new technique. Moreover, as
+
+
+
+Bastian, et al. Informational [Page 14]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ with any large network or software system, software bugs and/or
+ unexpected errors may arise, requiring software patches or other
+ corrective actions. As always, Comcast's decisions on these matters
+ are driven by the marketplace imperative that we deliver the best
+ possible experience to our HSI subscribers.
+
+ Given our experience as described above, we determined that a
+ starting point for the upstream Port Utilization Threshold should be
+ 70 percent and the downstream Port Utilization Threshold should be
+ 80 percent. For the Port Utilization Duration, we determined that
+ the starting point should be approximately 15 minutes (although some
+ technical limitations in some newer CMTSs deployed on Comcast's
+ network may make this time period vary slightly). Thus, over any
+ 15-minute period, if an average of more than 70 percent of a port's
+ upstream bandwidth capacity or more than 80 percent of a port's
+ downstream bandwidth capacity is utilized, that port is determined to
+ be in a Near Congestion State.
+
+ Based on the trials conducted and operational experience to date, a
+ typical CMTS port on our HSI network is in a Near Congestion State
+ only for relatively small portions of the day, if at all, though
+ there is no way to forecast what will be the busiest time on a
+ particular port on a particular day. Moreover, the trial data and
+ operational experience indicate that, even when a particular port is
+ in a Near Congestion State, the instances where the network actually
+ becomes congested during the Port Utilization Duration are few, and
+ managed users whose packets may be intermittently delayed or dropped
+ during those congested periods perceive little, if any, effect, as
+ discussed below.
+
+7.2. Thresholds for Determining When a User Is in an Extended High
+ Consumption State and for Release from That Classification
+
+ Once a particular CMTS port is in a Near Congestion State, the
+ software examines whether any cable modems are consuming bandwidth
+ disproportionately. (Note: Although each cable modem is typically
+ assigned to a particular household, the software does not and cannot
+ actually identify individual users or the number of users sharing a
+ cable modem, or analyze particular users' traffic.) For purposes of
+ this document, we use "cable modem", "user", and "subscriber"
+ interchangeably to mean a subscriber account or user account and not
+ an individual person. For a user to enter an Extended High
+ Consumption State, he or she must consume greater than a certain
+ percentage of his or her provisioned upstream or downstream bandwidth
+ (the "User Consumption Threshold") for a specific length of time (the
+ "User Consumption Duration"). The User Consumption Threshold is
+ measured as a user's consumption of a particular percentage of his or
+ her total provisioned upstream or downstream bandwidth. That
+
+
+
+Bastian, et al. Informational [Page 15]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ bandwidth is the maximum speed that a particular modem can achieve
+ based on the tier (personal/residential, commercial, etc.) the
+ customer has purchased. For example, if a user buys a service with
+ speeds of 50 Mbps downstream and 10 Mbps upstream, then his or her
+ provisioned downstream speed is 50 Mbps and provisioned upstream
+ speed is 10 Mbps. It is also important to note that because the User
+ Consumption Threshold is a percentage of provisioned bandwidth for a
+ particular user account, and not a static value, users of higher-
+ speed tiers have correspondingly higher User Consumption Thresholds.
+ Lastly, the User Consumption Duration is measured in minutes.
+
+ Following lab tests, simulations, technical trials, customer
+ feedback, vendor evaluations, and an independent third-party
+ consulting analysis, we have determined that the appropriate starting
+ point for the User Consumption Threshold is 70 percent of a
+ subscriber's provisioned upstream or downstream bandwidth, and that
+ the appropriate starting point for the User Consumption Duration is
+ 15 minutes (this has been further validated in our full deployment).
+ That is, when a subscriber uses an average of 70 percent or more of
+ his or her provisioned upstream or downstream bandwidth over a
+ particular 15-minute period, that user is then in an Extended High
+ Consumption State. Therefore, this is a consumption-based threshold
+ and not a peak-speed-based threshold. Thus, the Extended High
+ Consumption State is not tied to whether a user has bursted once or
+ more above this 70% threshold for a brief moment. Instead, it is
+ consumption-based, meaning that a certain bitrate must be exceeded
+ over at least the entire User Consumption Duration.
+
+ The User Consumption Thresholds have been set sufficiently high that
+ using the HSI connection for Voice over IP (VoIP), gaming, web
+ surfing, or most streaming video cannot alone cause subscribers to
+ our standard-level HSI service to exceed the User Consumption
+ Threshold. For example, while one of Comcast's common HSI service
+ tiers has a provisioned downstream bandwidth of 22 Mbps today,
+ streaming video (even some HD video) from Hulu uses less than
+ 2.5 Mbps, a Vonage or Skype VoIP call uses less than 131 kbps, and
+ streaming music uses less than 128 kbps (in this example, 70 percent
+ of 22 Mbps is 15.4 Mbps). As noted above, these values are subject
+ to change as necessary in the same way that specific anti-spam or
+ other network management practices are adjusted to address new issues
+ that arise, or should unexpected software bugs or other problems
+ arise.
+
+ Based on data collected from the trial markets where the new
+ congestion management practices were tested (further validated in our
+ full deployment), on average less than one-third of one percent of
+ subscribers have had their traffic priority status changed to the BE
+ state on any given day. For example, in Colorado Springs, CO, the
+
+
+
+Bastian, et al. Informational [Page 16]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ largest test market, on any given day in August 2008, an average of
+ 22 users out of 6,016 total subscribers in the trial had their
+ traffic priority status changed to BE at some point during the day.
+
+ A user's traffic is released from a BE state when the user's
+ bandwidth consumption drops below 50 percent of his or her
+ provisioned upstream or downstream bandwidth for a period of
+ approximately 15 minutes. These release criteria are intended to
+ minimize (and hopefully prevent) user QoS oscillation, i.e., a
+ situation in which a particular user could cycle repeatedly between
+ BE and PBE. Thus, without this lower release criteria, we were
+ concerned that certain users would oscillate between BE and PBE
+ states for an extended period, without clear benefit to the system
+ and other users, and would place an unnecessary signaling burden on
+ the system. NetForecast, Inc., an independent consultant retained to
+ provide analysis and recommendations regarding Comcast's trials and
+ related congestion management work, suggested this approach, which
+ has worked well in our trials, lab testing, and subsequent national
+ deployment.
+
+ Simply put, there are four steps for determining whether the traffic
+ associated with a particular cable modem is designated as PBE or BE:
+
+ 1. Determine if the CMTS port is in a Near Congestion State.
+
+ 2. If yes, determine whether any users are in an Extended High
+ Consumption State.
+
+ 3. If yes, change those users' traffic to BE from PBE. If the answer
+ at either step one or step two is no, no action is taken.
+
+ 4. If a user's traffic has been designated BE, check user consumption
+ at the next interval. If user consumption has declined below the
+ predetermined threshold, reassign the user's traffic as PBE. If
+ not, recheck at the next interval.
+
+ In cases where a CMTS regularly enters a Near Congestion State, and
+ where congestion subsequently does occur, but where no users match
+ the criteria to be classified in an Extended High Consumption State,
+ this may indicate the congestion observed is regularly occurring,
+ rather than unpredictable congestion. As such, this may be an
+ additional data point in favor of considering whether and when to add
+ capacity.
+
+
+
+
+
+
+
+
+Bastian, et al. Informational [Page 17]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ Figure 2 graphically depicts how this congestion management process
+ works, using an example of a situation where upstream port
+ utilization may be reaching a Near Congestion State (the same
+ diagram, with different values in the appropriate places, could be
+ used to depict the management process for downstream ports, as well):
+
+ Figure 2: Upstream Congestion Management Decision Flowchart
+
+ /\
+ +------------+ / \ +---------+ +---------+
+ | Start | / \ | | / /
+ | Congestion | / \ | | / /
+ | Management +-->+ Question +--YES-->| Result |--THEN-->/ Action /
+ | Process | \ #1 / | #1 | / #1 /
+ | | \ / | | / /
+ +------------+ \ / +---------+ +---------+
+ \/ |
+ | THEN
+ NO |
+ | \/
+ \/ /\
+ +---------+ / \
+ | | / \
+ | | / \
+ | Result |<-------------NO------------+ Question +
+ | #2 | \ #2 /
+ | | \ /
+ +---------+ \ /
+ \/
+ |
+ YES
+ |
+ /\ \/
+ +---------+ / \ +---------+
+ | | / \ | |
+ | | / \ THEN, AT | |
+ | Result |<--YES--+ Question + <---NEXT ANALYSIS------+ Result |
+ | #4 | \ #3 / POINT /\ | #3 |
+ | | \ / | | |
+ +---------+ \ / | +---------+
+ \/ |
+ | |
+ +------------NO-------------+
+
+
+
+
+
+
+
+
+Bastian, et al. Informational [Page 18]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ KEY TO FIGURE 2 ABOVE:
+
+ Question #1: Is the CMTS Upstream Port Utilization at an average
+ of OVER 70% for OVER 15 minutes?
+
+ Result #1: CMTS marked in a Near Congestion State, indicating
+ congestion *may* occur soon.
+
+ Action #1: Search most recent analysis timeframe (approx. 15 mins.)
+ of IPDR usage data.
+
+ Question #2: Are any users consuming an average of OVER 70% of
+ provisioned upstream bandwidth for OVER 15 minutes?
+
+ Result #2: No action taken.
+
+ Result #3: Change user's upstream traffic from Priority Best Effort
+ (PBE) to Best Effort (BE).
+
+ Question #3: Is the user in Best Effort (BE) consuming an average
+ of LESS THAN 50% of provisioned upstream bandwidth
+ over a period of 15 minutes?
+
+ Result #4: Change user's upstream traffic back to Priority Best
+ Effort (PBE) from Best Effort (BE).
+
+ Figure 2
+
+7.3. Effect of BE Quality of Service on Users' Broadband Experience
+
+ When a CMTS port is in a Near Congestion State and a cable modem
+ connected to that port is in an Extended High Consumption State, that
+ cable modem's traffic is designated as BE. Depending upon the level
+ of utilization on the CMTS port, this designation may or may not
+ result in the user's traffic being delayed or, in extreme cases,
+ dropped before PBE traffic is dropped. This is because of the way
+ that the CMTS handles traffic. Specifically, CMTS ports have what is
+ commonly called a "scheduler" that puts all the packets coming from
+ or going to cable modems on that particular port in a queue and then
+ handles them in turn. A certain number of packets can be processed
+ by the scheduler in any given moment; for each time slot, PBE traffic
+ is given priority access to the available capacity, and BE traffic is
+ processed on a space-available basis.
+
+ A rough analogy would be to busses that empty and fill up at
+ incredibly fast speeds. As empty busses arrive at the figurative
+ "bus stop" -- every two milliseconds in this case -- they fill up
+ with as many packets as are waiting for "seats" on the bus, to the
+
+
+
+Bastian, et al. Informational [Page 19]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ limits of the bus' capacity. During non-congested periods, the bus
+ will usually have several empty seats, but during congested periods,
+ the bus will fill up and packets will have to wait for the next bus.
+ It is during the congested periods that BE packets will be affected.
+ If there is no congestion, packets from a user in a BE state should
+ have little trouble getting on the bus when they arrive at the bus
+ stop. If, on the other hand, there is congestion in a particular
+ instance, the bus may become filled by packets in a PBE state before
+ any BE packets can get on. In that situation, the BE packets would
+ have to wait for the next bus that is not filled by PBE packets. In
+ reality, this all takes place in two-millisecond increments, so even
+ if the packets miss 50 "busses", the delay will only be about one-
+ tenth of a second.
+
+ During times of actual network congestion, when packets from BE
+ traffic might be intermittently delayed, there is a variety of
+ effects that could be experienced by a user whose traffic is delayed,
+ depending upon what applications he or she is using. Typically, a
+ user whose traffic is in a BE state during actual congestion may find
+ that a webpage loads sluggishly, a peer-to-peer upload takes somewhat
+ longer to complete, or a VoIP call sounds choppy. Of course, the
+ same thing could happen to the customers on a port that is congested
+ in the absence of any congestion management; the difference here is
+ that the effects of any such delays are shifted toward those who have
+ been placing the greatest burden on the network, instead of being
+ distributed randomly among the users of that port without regard to
+ their consumption levels. As a matter of fact, our studies concluded
+ that the experience of the PBE subscribers improves when this
+ congestion management system is enabled. This conclusion is based on
+ network measurements, such as latency.
+
+ NetForecast explored the potential risk of a worst-case scenario for
+ users whose traffic is in a BE state: the possibility of "bandwidth
+ starvation" in the theoretical case where 100 percent of the CMTS
+ bandwidth is taken up by PBE traffic for an extended period of time.
+ In theory, such a condition could mean that a given user whose
+ traffic is designated BE would be unable to effectuate an upload or
+ download (as noted above, both are managed separately) for some
+ period of time. However, when these management techniques were
+ tested, first in company testbeds and then in our real-world trials
+ conducted in the five markets (further validated in our full
+ deployment), such a theoretical condition did not occur. In
+ addition, our experience with the system as fully deployed in our
+ production network demonstrates that these management practices have
+ very modest real-world impacts. In addition, Comcast did not receive
+ a single customer complaint, in any of the trial markets, that could
+ be traced to this congestion management system, despite having
+ broadly publicized these trials. In our subsequent national
+
+
+
+Bastian, et al. Informational [Page 20]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ deployment into our production network, we still have yet to find a
+ specific complaint that can be traced back to the effect of this
+ congestion management system.
+
+ Comcast continues to monitor how user traffic is affected by these
+ new congestion management techniques and will make the adjustments
+ necessary to ensure that all Comcast HSI customers have a high-
+ quality Internet experience.
+
+7.4. Equipment/Software Used and Location
+
+ The above-mentioned functions are carried out using three different
+ types of application servers, supplied by three different vendors.
+ As mentioned above, these servers are installed near Comcast's
+ regional network routers. The exact locations of these servers are
+ not particularly relevant to this document, as this information does
+ not change the fact that the servers manage individual CMTS ports.
+
+ The first application server is an IPDR server, which collects
+ relevant cable modem volume usage information from the CMTS, such as
+ how many aggregate upstream or downstream bytes a subscriber uses
+ over a particular period of time. IPDR has been adopted as a
+ standard by many industry organizations and initiatives, such as
+ CableLabs, the Alliance for Telecommunications Industry Solutions
+ (ATIS), the International Telecommunication Union (ITU), and the
+ Third Generation Partnership Project (3GPP), among others. The IPDR
+ software deployed was developed by Active Broadband Networks, and is
+ noted as the Statistics Collection Server in Figure 3.
+
+ The second application server is the Congestion Management server,
+ which uses the Simple Network Management Protocol (SNMP) [RFC3410] to
+ measure CMTS port utilization and detect when a port is in a Near
+ Congestion State. When this happens, the Congestion Management
+ server then queries the relevant IPDR data for a list of cable modems
+ meeting the criteria set forth above for being in an Extended High
+ Consumption State. The Congestion Management server software
+ deployed was developed by Sandvine.
+
+ If one or more users meet the criteria to be managed, then the
+ Congestion Management server notifies a third application server, the
+ PCMM application server, as to which users have been in an Extended
+ High Consumption State and whose traffic should be treated as BE.
+ The PCMM servers are responsible for signaling a given CMTS to set
+ the traffic for specific cable modems with a BE QoS, and for tracking
+ and managing the state of such CMTS actions. If no users meet the
+ criteria to be managed, no users will have their traffic managed.
+ The PCMM software deployed was developed by Camiant, and is noted as
+ the QoS Server in Figure 3.
+
+
+
+Bastian, et al. Informational [Page 21]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ Figure 3 graphically depicts the high-level management flows among
+ the congestion management components on Comcast's network, as
+ described above:
+
+ Figure 3: Simplified Diagram Showing High-Level Management Flows
+ Relevant to the System
+
+ +---------------+ +---------------+
+ | Congestion | Instruct QoS Server | QoS |
+ | Management |******to Change QoS for****>| Server |
+ | Server | a Device | |
+ +----+---+------+ +-------+-------+
+ /\ /\ *
+ | | Relay Selected *
+ X +---Statistics: Bytes---+ QoS Action:
+ | Up/Down by Device | Change from PBE
+ X +-------+-------+ to BE, or from
+ | | Statistics | BE to PBE
+ X | Collection | *
+ Periodic SNMP | Server | *
+ Requests to +---------------+ *
+ Check CMTS Port /\ *
+ Utilization | *
+ Levels Statistics Sent *
+ | Periodically From CMTS *
+ X | *
+ | +-----------+-----------+ *
+ +-X-X-X-X-X-X->| CMTS in Headend |<********
+ +-----------------------+
+ H /\ /\ H
+ H Internet Traffic H
+ H to/from User H
+ H \/ \/ H
+ /+---------------------+\
+ / | User's +---------+ |\
+ / | Home | Cable | | \
+ | | Modem | |
+ ============ | +---------+ |
+ = Notes: = +----------------------+
+ = ========================================================
+ = 1 - Statistics Collection Servers use IP Detail Records (IPDR). =
+ = 2 - QoS Servers use PacketCable Multimedia (PCMM) =
+ = to set QoS gates on the CMTS. =
+ = 3 - This figure is a simplification of the actual network and =
+ = servers, which included redundancies and other complexities =
+ = not necessary to depict the functional design. =
+ ===================================================================
+ Figure 3
+
+
+
+Bastian, et al. Informational [Page 22]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+8. Conclusion
+
+ Comcast started design and development of this new protocol-agnostic
+ congestion management system in March 2008. Comcast shared the
+ design with the IETF and others in the Internet community, as well as
+ with an independent consultant, incorporating feedback we received
+ into the final design. Following lab testing, the system was tested
+ in Comcast's production network in trial markets between June and
+ September 2008. Comcast's production network transition to this new
+ protocol-agnostic congestion management system began in October 2008
+ and was completed on December 31, 2008.
+
+ As described herein, the new approach does not manage congestion by
+ focusing on managing the use of specific protocols. Nor does this
+ approach use TCP "reset packets" [RFC3360]. Rather, the system acts
+ such that during periods when a CMTS port is in a Near Congestion
+ State, the system (1) identifies the subscribers on that port who
+ have consumed a disproportionate amount of bandwidth over the
+ preceding 15 minutes and (2) lowers the priority status of those
+ subscribers' traffic to BE status until those subscribers meet the
+ release criteria. During periods of actual congestion, the system
+ handles PBE traffic before BE traffic. Comcast's trials and
+ subsequent national deployment indicate that this new congestion
+ management system ensures a quality online experience for all of
+ Comcast's HSI customers.
+
+9. Exceptional Network Utilization Considerations
+
+ This system was developed to cope with somewhat "normal" occurrences
+ of congestion that could occur on virtually any IP network. It
+ should also be noted, however, that such a system could also prove
+ particularly useful in the case of "exceptional network utilization"
+ events that existing network usage models do not or cannot accurately
+ predict. Some network operators refer to these exceptional events as
+ "surges" in utilization, similar to sudden surges in demand in
+ electrical power grids, with which many people may be familiar.
+
+ For example, in the case of a severe global pandemic, it may be
+ expected that large swaths of the population may need to work
+ remotely, via their Internet connection. In such a case, a largely
+ unprecedented level of utilization may occur. In such cases, it may
+ be helpful to have a flexible congestion management system that could
+ adapt to this situation and help allocate network resources while
+ additional capacity is being brought online or while a temporary
+ condition persists.
+
+
+
+
+
+
+Bastian, et al. Informational [Page 23]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+10. Limitations of This Congestion Management System
+
+ The main limitations of the system include:
+
+ o The system is not an end-to-end congestion management system, nor
+ does it enable one.
+
+ o The system does not signal the presence of congestion to user
+ applications or to all devices on the network path.
+
+ o The system does not explicitly enable additional user and/or
+ application responses to congestion.
+
+ o The system does not enable distributed denial-of-service (DDoS)
+ mitigation or other capabilities.
+
+11. Low Extra Delay Background Transport and Other Possibilities
+
+ There are several new IETF working group efforts that are focused on
+ the question of congestion and its effects, avoiding congestion,
+ managing congestion, and communicating congestion information. This
+ includes the Congestion Exposure (CONEX) working group, the
+ Application Layer Transport Optimization (ALTO) working group, and
+ the Low Extra Delay Background Transport (LEDBAT) working group.
+ Should one or more of these working groups be successful in producing
+ useful work, it is possible that the design or configuration of the
+ system documented here may need to change. For example, this
+ congestion management system does not currently have a way to take
+ into account differing classes of data transfer, such as a class of
+ data transfer that LEDBAT may specify, which may better yield to
+ other traffic than existing transport protocols. In addition, CONEX
+ may specify methods for this or other systems to signal congestion
+ state or expected congestion to other parts of the network, and/or to
+ hosts on either end of a particular network flow. Furthermore, it is
+ conceivable that the result of current or future IETF work could
+ obviate the need for such a congestion management system entirely.
+
+12. Security Considerations
+
+ It is important that an ISP secure access to the Congestion
+ Management servers and the QoS Servers, as well as QoS signaling to
+ the CMTSs, so that unauthorized users and/or hosts cannot make
+ unauthorized changes to QoS settings in the network.
+
+ It is also important to secure access to the Statistics Collection
+ Server since this contains IPDR-based byte transfer data that is
+ considered private by end users on an individual basis. In addition,
+ this data is considered ISP-proprietary traffic data on an aggregate
+
+
+
+Bastian, et al. Informational [Page 24]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ basis. Access to the Statistics Collection Server should also be
+ secured so that false usage statistics cannot be fed into the system.
+ It is important to note that IPDR data contains a count of bytes sent
+ and bytes received, by cable modem MAC address, over a given interval
+ of time. This data does not contain things such as the source and/or
+ destination Internet address of that data, nor does it contain the
+ protocols used, ports used, etc.
+
+13. Acknowledgements
+
+ The authors wish to acknowledge the hard work of the many people who
+ helped to develop and/or review this document, as well as the people
+ who helped deploy the system in such a short period of time.
+
+ The authors also wish to acknowledge the following individuals for
+ performing a detailed review of this document and/or providing
+ comments and feedback that helped to improve and evolve this
+ document:
+
+ - Kris Bransom
+
+ - Bob Briscoe
+
+ - Lars Eggert
+
+ - Ari Keranen
+
+ - Tero Kivinen
+
+ - Matt Mathis
+
+ - Stanislav Shalunov
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Bastian, et al. Informational [Page 25]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+14. Informative References
+
+ [COMCAST_P2PI_PAPER]
+ Livingood, J. and R. Woundy, "Comcast's IETF P2P
+ Infrastructure Workshop Position Paper", FCC
+ Filings Comcast Network Management Proceedings, May 2008,
+ <http://trac.tools.ietf.org/area/rai/trac/raw-attachment/
+ wiki/PeerToPeerInfrastructure/
+ 16%20ietf-p2pi-comcast-20080509.pdf>.
+
+ [COMCAST_P2PI_PRES]
+ Livingood, J. and R. Woundy, "Comcast's IETF P2P
+ Infrastructure Workshop Presentation on May 28, 2008",
+ FCC Filings Comcast Network Management Proceedings,
+ May 2008,
+ <http://trac.tools.ietf.org/area/rai/trac/raw-attachment/
+ wiki/PeerToPeerInfrastructure/02-Comcast-IETF-P2Pi.pdf>.
+
+ [DOCSIS_CM2CPE]
+ CableLabs, "Data-Over-Cable Service Interface
+ Specifications - DOCSIS 3.0 - Cable Modem to Customer
+ Premise Equipment Interface Specification", DOCSIS
+ 3.0 CM-SP-CMCIv3-I01-080320, March 2008,
+ <http://www.cablelabs.com/cablemodem/specifications/
+ specifications30.html>.
+
+ [DOCSIS_IPDR]
+ Yassini, R., "Data-Over-Cable Service Interface
+ Specifications - DOCSIS 2.0 - Operations Support System
+ Interface Specification", DOCSIS 2.0 CM-SP-OSSIv2.0-C01-
+ 081104, November 2008, <http://www.cablelabs.com/
+ cablemodem/specifications/specifications30.html>.
+
+ [DOCSIS_MULPI]
+ CableLabs, "Data-Over-Cable Service Interface
+ Specifications - DOCSIS 3.0 - MAC and Upper Layer
+ Protocols Interface Specification", DOCSIS 3.0 CM-SP-
+ MULPIv3.0-I11-091002, October 2009, <http://
+ www.cablelabs.com/cablemodem/specifications/
+ specifications30.html>.
+
+ [DOCSIS_OSSI]
+ CableLabs, "Data-Over-Cable Service Interface
+ Specifications - DOCSIS 3.0 - Operations Support System
+ Interface Specification", DOCSIS 3.0 CM-SP-OSSIv3.0-I10-
+ 091002, October 2009, <http://www.cablelabs.com/
+ cablemodem/specifications/specifications30.html>.
+
+
+
+
+Bastian, et al. Informational [Page 26]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ [DOCSIS_PHY]
+ CableLabs, "Data-Over-Cable Service Interface
+ Specifications - DOCSIS 3.0 - Physical Layer
+ Specification", DOCSIS 3.0 CM-SP-PHYv3.0-I08-090121,
+ January 2009, <http://www.cablelabs.com/cablemodem/
+ specifications/specifications30.html>.
+
+ [DOCSIS_SEC]
+ CableLabs, "Data-Over-Cable Service Interface
+ Specifications - DOCSIS 3.0 - Security Specification",
+ DOCSIS 3.0 CM-SP-SECv3.0-I11-091002, March 2008, <http://
+ www.cablelabs.com/cablemodem/specifications/
+ specifications30.html>.
+
+ [FCC_Congest_Mgmt_Ltr]
+ Zachem, K., "Letter to the FCC Advising of Successful
+ Deployment of Comcast's New Congestion Management
+ System", FCC Filings Comcast Network Management
+ Proceedings, January 2009,
+ <http://fjallfoss.fcc.gov/ecfs/document/
+ view?id=6520192582>.
+
+ [FCC_Memo_Opinion]
+ Martin, K., Copps, M., Adelstein, J., Tate, D., and R.
+ McDowell, "FCC Memorandum and Opinion Regarding
+ Reasonable Network Management", File No. EB-08-IH-1518 WC
+ Docket No. 07-52, August 2008,
+ <http://hraunfoss.fcc.gov/
+ edocs_public/attachmatch/FCC-08-183A1.pdf>.
+
+ [FCC_Net_Mgmt_Response]
+ Zachem, K., "Letter to the FCC Regarding Comcast's
+ Network Management Practices", FCC Filings Comcast
+ Network Management Proceedings, September 2008, <http://
+ fjallfoss.fcc.gov/ecfs/document/view?id=6520169715>.
+
+ [IPDR_Standard]
+ Cotton, S., Cockrell, B., Walls, P., and T. Givoly,
+ "Network Data Management - Usage (NDM-U) For IP-Based
+ Services. Service Specification - Cable Labs DOCSIS 2.0
+ SAMIS", IPDR Service Specifications NDM-U, November 2004,
+ <http://www.ipdr.org/public/Service_Specifications/3.X/
+ DOCSIS(R)3.5-A.0.pdf>.
+
+
+
+
+
+
+
+
+Bastian, et al. Informational [Page 27]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+ [PowerBoost_Specification]
+ Comcast Cable Communications Management LLC, "Comcast
+ PowerBoost Specification", Website Comcast.com,
+ June 2010, <http://customer.comcast.com/Pages/
+ FAQListViewer.aspx?topic=Internet&
+ folder=8b2fc392-4cde-4750-ba34-051cd5feacf0>.
+
+ [RFC1633] Braden, B., Clark, D., and S. Shenker, "Integrated
+ Services in the Internet Architecture: an Overview",
+ RFC 1633, June 1994.
+
+ [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z.,
+ and W. Weiss, "An Architecture for Differentiated
+ Services", RFC 2475, December 1998.
+
+ [RFC3083] Woundy, R., "Baseline Privacy Interface Management
+ Information Base for DOCSIS Compliant Cable Modems and
+ Cable Modem Termination Systems", RFC 3083, March 2001.
+
+ [RFC3360] Floyd, S., "Inappropriate TCP Resets Considered Harmful",
+ BCP 60, RFC 3360, August 2002.
+
+ [RFC3410] Case, J., Mundy, R., Partain, D., and B. Stewart,
+ "Introduction and Applicability Statements for Internet-
+ Standard Management Framework", RFC 3410, December 2002.
+
+ [RFC5594] Peterson, J. and A. Cooper, "Report from the IETF
+ Workshop on Peer-to-Peer (P2P) Infrastructure, May 28,
+ 2008", RFC 5594, July 2009.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Bastian, et al. Informational [Page 28]
+
+RFC 6057 An ISP Congestion Management System December 2010
+
+
+Authors' Addresses
+
+ Chris Bastian
+ Comcast Cable Communications
+ One Comcast Center
+ 1701 John F. Kennedy Boulevard
+ Philadelphia, PA 19103
+ US
+ EMail: chris_bastian@cable.comcast.com
+ URI: http://www.comcast.com
+
+ Tom Klieber
+ Comcast Cable Communications
+ 1306 Goshen Parkway
+ West Chester, PA 19380
+ US
+ EMail: tom_klieber@cable.comcast.com
+ URI: http://www.comcast.com
+
+ Jason Livingood
+ Comcast Cable Communications
+ One Comcast Center
+ 1701 John F. Kennedy Boulevard
+ Philadelphia, PA 19103
+ US
+ EMail: jason_livingood@cable.comcast.com
+ URI: http://www.comcast.com
+
+ Jim Mills
+ Comcast Cable Communications
+ One Comcast Center
+ 1800 Bishops Gate Drive
+ Mount Laurel, NJ 08054
+ US
+ EMail: jim_mills@cable.comcast.com
+ URI: http://www.comcast.com
+
+ Richard Woundy
+ Comcast Cable Communications
+ 27 Industrial Avenue
+ Chelmsford, MA 01824
+ US
+ EMail: richard_woundy@cable.comcast.com
+ URI: http://www.comcast.com
+
+
+
+
+
+
+
+Bastian, et al. Informational [Page 29]
+