diff options
author | Thomas Voss <mail@thomasvoss.com> | 2024-11-27 20:54:24 +0100 |
---|---|---|
committer | Thomas Voss <mail@thomasvoss.com> | 2024-11-27 20:54:24 +0100 |
commit | 4bfd864f10b68b71482b35c818559068ef8d5797 (patch) | |
tree | e3989f47a7994642eb325063d46e8f08ffa681dc /doc/rfc/rfc5166.txt | |
parent | ea76e11061bda059ae9f9ad130a9895cc85607db (diff) |
doc: Add RFC documents
Diffstat (limited to 'doc/rfc/rfc5166.txt')
-rw-r--r-- | doc/rfc/rfc5166.txt | 1291 |
1 files changed, 1291 insertions, 0 deletions
diff --git a/doc/rfc/rfc5166.txt b/doc/rfc/rfc5166.txt new file mode 100644 index 0000000..0126f4f --- /dev/null +++ b/doc/rfc/rfc5166.txt @@ -0,0 +1,1291 @@ + + + + + + +Network Working Group S. Floyd, Ed. +Request for Comments: 5166 March 2008 +Category: Informational + + + Metrics for the Evaluation of Congestion Control Mechanisms + +Status of This Memo + + This memo provides information for the Internet community. It does + not specify an Internet standard of any kind. Distribution of this + memo is unlimited. + +IESG Note + + This document is not an IETF Internet Standard. It represents the + individual opinion(s) of one or more members of the TMRG Research + Group of the Internet Research Task Force. It may be considered for + standardization by the IETF or adoption as an IRTF Research Group + consensus document in the future. + +Abstract + + This document discusses the metrics to be considered in an evaluation + of new or modified congestion control mechanisms for the Internet. + These include metrics for the evaluation of new transport protocols, + of proposed modifications to TCP, of application-level congestion + control, and of Active Queue Management (AQM) mechanisms in the + router. This document is the first in a series of documents aimed at + improving the models that we use in the evaluation of transport + protocols. + + This document is a product of the Transport Modeling Research Group + (TMRG), and has received detailed feedback from many members of the + Research Group (RG). As the document tries to make clear, there is + not necessarily a consensus within the research community (or the + IETF community, the vendor community, the operations community, or + any other community) about the metrics that congestion control + mechanisms should be designed to optimize, in terms of trade-offs + between throughput and delay, fairness between competing flows, and + the like. However, we believe that there is a clear consensus that + congestion control mechanisms should be evaluated in terms of trade- + offs between a range of metrics, rather than in terms of optimizing + for a single metric. + + + + + + + +Floyd Informational [Page 1] + +RFC 5166 TMRG, METRICS March 2008 + + +Table of Contents + + 1. Introduction ....................................................2 + 2. Metrics .........................................................3 + 2.1. Throughput, Delay, and Loss Rates ..........................4 + 2.1.1. Throughput ..........................................5 + 2.1.2. Delay ...............................................6 + 2.1.3. Packet Loss Rates ...................................6 + 2.2. Response Times and Minimizing Oscillations .................7 + 2.2.1. Response to Changes .................................7 + 2.2.2. Minimizing Oscillations .............................8 + 2.3. Fairness and Convergence ...................................9 + 2.3.1. Metrics for Fairness between Flows .................10 + 2.3.2. Metrics for Fairness between Flows with + Different Resource Requirements ....................10 + 2.3.3. Comments on Fairness ...............................12 + 2.4. Robustness for Challenging Environments ...................13 + 2.5. Robustness to Failures and to Misbehaving Users ...........14 + 2.6. Deployability .............................................14 + 2.7. Metrics for Specific Types of Transport ...................15 + 2.8. User-Based Metrics ........................................15 + 3. Metrics in the IP Performance Metrics (IPPM) Working Group .....15 + 4. Comments on Methodology ........................................16 + 5. Security Considerations ........................................16 + 6. Acknowledgements ...............................................16 + 7. Informative References .........................................17 + +1. Introduction + + As a step towards improving our methodologies for evaluating + congestion control mechanisms, in this document we discuss some of + the metrics to be considered. We also consider the relationship + between metrics, e.g., the well-known trade-off between throughput + and delay. This document doesn't attempt to specify every metric + that a study might consider; for example, there are domain-specific + metrics that researchers might consider that are over and above the + metrics laid out here. + + We consider metrics for aggregate traffic (taking into account the + effect of flows on competing traffic in the network) as well as the + heterogeneous goals of different applications or transport protocols + (e.g., of high throughput for bulk data transfer, and of low delay + for interactive voice or video). Different transport protocols or + AQM mechanisms might have goals of optimizing different sets of + metrics, with one transport protocol optimized for per-flow + throughput and another optimized for robustness over wireless links, + and with different degrees of attention to fairness with competing + traffic. We hope this document will be used as a step in evaluating + + + +Floyd Informational [Page 2] + +RFC 5166 TMRG, METRICS March 2008 + + + proposed congestion control mechanisms for a wide range of metrics, + for example, noting that Mechanism X is good at optimizing Metric A, + but pays the price with poor performance for Metric B. The goal + would be to have a broad view of both the strengths and weaknesses of + newly proposed congestion control mechanisms. + + Subsequent documents are planned to present sets of simulation and + testbed scenarios for the evaluation of transport protocols and of + congestion control mechanisms, based on the best current practice of + the research community. These are not intended to be complete or + final benchmark test suites, but simply to be one step of many to be + used by researchers in evaluating congestion control mechanisms. + Subsequent documents are also planned on the methodologies in using + these sets of scenarios. + + This document is a product of the Transport Modeling Research Group + (TMRG), and has received detailed feedback from many members of the + Research Group (RG). As the document tries to make clear, there is + not necessarily a consensus within the research community (or the + IETF community, the vendor community, the operations community, or + any other community) about the metrics that congestion control + mechanisms should be designed to optimize, in terms of trade-offs + between throughput and delay, fairness between competing flows, and + the like. However, we believe that there is a clear consensus that + congestion control mechanisms should be evaluated in terms of + trade-offs between a range of metrics, rather than in terms of + optimizing for a single metric. + +2. Metrics + + The metrics that we discuss are the following: + + o Throughput; + + o Delay; + + o Packet loss rates; + + o Response to sudden changes or to transient events; + + o Minimizing oscillations in throughput or in delay; + + o Fairness and convergence times; + + o Robustness for challenging environments; + + o Robustness to failures and to misbehaving users; + + + + +Floyd Informational [Page 3] + +RFC 5166 TMRG, METRICS March 2008 + + + o Deployability; + + o Metrics for specific types of transport; + + o User-based metrics. + + We consider each of these below. Many of the metrics have + network-based, flow-based, and user-based interpretations. For + example, network-based metrics can consider aggregate bandwidth and + aggregate drop rates, flow-based metrics can consider end-to-end + transfer times for file transfers or end-to-end delay and packet drop + rates for interactive traffic, and user-based metrics can consider + user wait time or user satisfaction with the multimedia experience. + Our main goal in this document is to explain the set of metrics that + can be relevant, and not to legislate on the most appropriate + methodology for using each general metric. + + For some of the metrics, such as fairness, there is not a clear + agreement in the network community about the desired goals. In these + cases, the document attempts to present the range of approaches. + +2.1. Throughput, Delay, and Loss Rates + + Because of the clear trade-offs between throughput, delay, and loss + rates, it can be useful to consider these three metrics together. + The trade-offs are most clear in terms of queue management at the + router; is the queue configured to maximize aggregate throughput, to + minimize delay and loss rates, or some compromise between the two? + An alternative would be to consider a separate metric such as power, + defined in this context as throughput over delay, that combines + throughput and delay. However, we do not propose in this document a + clear target in terms of the trade-offs between throughput and delay; + we are simply proposing that the evaluation of transport protocols + include an exploration of the competing metrics. + + Using flow-based metrics instead of router-based metrics, the + relationship between per-flow throughput, delay, and loss rates can + often be given by the transport protocol itself. For example, in + TCP, the sending rate s in packets per second is given as: + + s = 1.2/(RTT*sqrt(p)), + + for the round-trip time RTT and loss rate p [FF99]. + + + + + + + + +Floyd Informational [Page 4] + +RFC 5166 TMRG, METRICS March 2008 + + +2.1.1. Throughput + + Throughput can be measured as a router-based metric of aggregate link + utilization, as a flow-based metric of per-connection transfer times, + and as user-based metrics of utility functions or user wait times. + It is a clear goal of most congestion control mechanisms to maximize + throughput, subject to application demand and to the constraints of + the other metrics. + + Throughput is sometimes distinguished from goodput, where throughput + is the link utilization or flow rate in bytes per second; goodput, + also measured in bytes per second, is the subset of throughput + consisting of useful traffic. That is, 'goodput' excludes duplicate + packets, packets that will be dropped downstream, packet fragments or + ATM cells that are dropped at the receiver because they can't be + re-assembled into complete packets, and the like. In general, this + document doesn't distinguish between throughput and goodput, and uses + the general term "throughput". + + We note that maximizing throughput is of concern in a wide range of + environments, from highly-congested networks to under-utilized ones, + and from long-lived flows to very short ones. As an example, + throughput has been used as one of the metrics for evaluating + Quick-Start, a proposal to allow flows to start up faster than + slow-start, where throughput has been evaluated in terms of the + transfer times for connections with a range of transfer sizes + [RFC4782] [SAF06]. + + In some contexts, it might be sufficient to consider the aggregate + throughput or the mean per-flow throughput [DM06], while in other + contexts it might be necessary to consider the distribution of + per-flow throughput. Some researchers evaluate transport protocols + in terms of maximizing the aggregate user utility, where a user's + utility is generally defined as a function of the user's throughput + [KMT98]. + + Individual applications can have application-specific needs in terms + of throughput. For example, real-time video traffic can have highly + variable bandwidth demands; Voice over IP (VoIP) traffic is sensitive + to the amount of bandwidth received immediately after idle periods. + Thus, user metrics for throughput can be more complex than simply the + per-connection transfer time. + + + + + + + + + +Floyd Informational [Page 5] + +RFC 5166 TMRG, METRICS March 2008 + + +2.1.2. Delay + + Like throughput, delay can be measured as a router-based metric of + queueing delay over time, or as a flow-based metric in terms of + per-packet transfer times. Per-packet delay can also include delay + at the sender waiting for the transport protocol to send the packet. + For reliable transfer, the per-packet transfer time seen by the + application includes the possible delay of retransmitting a lost + packet. + + Users of bulk data transfer applications might care about per-packet + transfer times only in so far as they affect the per-connection + transfer time. On the other end of the spectrum, for users of + streaming media, per-packet delay can be a significant concern. Note + that in some cases the average delay might not capture the metric of + interest to the users; for example, some users might care about the + worst-case delay, or about the tail of the delay distribution. + + Note that queueing delay at a router is shared by all flows at a FIFO + (First-In First-Out) queue. Thus, the router-based queueing delay + induced by bulk data transfers can be important even if the bulk data + transfers themselves are not concerned with per-packet transfer + times. + +2.1.3. Packet Loss Rates + + Packet loss rates can be measured as a network-based or as a + flow-based metric. + + When evaluating the effect of packet losses or ECN marks (Explicit + Congestion Notification) [RFC3168] on the performance of a congestion + control mechanism for an individual flow, researchers often use both + the packet loss/mark rate for that connection and the congestion + event rate (also called the loss event rate), where a congestion + event or loss event consists of one or more lost or marked packets in + one round-trip time [RFC3448]. + + Some users might care about the packet loss rate only in so far as it + affects per-connection transfer times, while other users might care + about packet loss rates directly. RFC 3611, RTP Control Protocol + Extended Reports, describes a VoIP performance-reporting standard + called RTP Control Protocol Extended Reports (RTCP XR), which + includes a set of burst metrics. In RFC 3611, a burst is defined as + the maximal sequence starting and ending with a lost packet, and not + including a sequence of Gmin or more packets that are not lost + [RFC3611]. The burst metrics in RFC 3611 consist of the burst + density (the fraction of packets in bursts), gap density (the + fraction of packets in the gaps between bursts), burst duration (the + + + +Floyd Informational [Page 6] + +RFC 5166 TMRG, METRICS March 2008 + + + mean duration of bursts in seconds), and gap duration (the mean + duration of gaps in seconds). RFC 3357 derives metrics for "loss + distance" and "loss period", along with statistics that capture loss + patterns experienced by packet streams on the Internet [RFC3357]. + + In some cases, it is useful to distinguish between packets dropped at + routers due to congestion and packets lost in the network due to + corruption. + + One network-related reason to avoid high steady-state packet loss + rates is to avoid congestion collapse in environments containing + paths with multiple congested links. In such environments, high + packet loss rates could result in congested links wasting scarce + bandwidth by carrying packets that will only be dropped downstream + before being delivered to the receiver [RFC2914]. We also note that + in some cases, the retransmit rate can be high, and the goodput + correspondingly low, even with a low packet drop rate [AEO03]. + +2.2. Response Times and Minimizing Oscillations + + In this section, we consider response times and oscillations + together, as there are well-known trade-offs between improving + response times and minimizing oscillations. In addition, the + scenarios that illustrate the dangers of poor response times are + often quite different from the scenarios that illustrate the dangers + of unnecessary oscillations. + +2.2.1. Response to Changes + + One of the key concerns in the design of congestion control + mechanisms has been the response times to sudden congestion in the + network. On the one hand, congestion control mechanisms should + respond reasonably promptly to sudden congestion from routing or + bandwidth changes or from a burst of competing traffic. At the same + time, congestion control mechanisms should not respond too severely + to transient changes, e.g., to a sudden increase in delay that will + dissipate in less than the connection's round-trip time. + + Congestion control mechanisms also have to contend with sudden + changes in the bandwidth-delay product due to mobility. Such + bandwidth-delay product changes are expected to become more frequent + and to have greater impact than path changes today. As a result of + both mobility and of the heterogeneity of wireless access types + (802.11b,a,g, WIMAX, WCDMA, HS-WCDMA, E-GPRS, Bluetooth, etc.), both + the bandwidth and the round-trip delay can change suddenly, sometimes + by several orders of magnitude. + + + + + +Floyd Informational [Page 7] + +RFC 5166 TMRG, METRICS March 2008 + + + Evaluating the response to sudden or transient changes can be of + particular concern for slowly responding congestion control + mechanisms such as equation-based congestion control [RFC3448] and + AIMD (Additive Increase Multiplicative Decrease) or for related + mechanisms using parameters that make them more slowly-responding + than TCP [BB01] [BBFS01]. + + In addition to the responsiveness and smoothness of aggregate + traffic, one can consider the trade-offs between responsiveness, + smoothness, and aggressiveness for an individual connection [FHP00] + [YKL01]. In this case, smoothness can be defined by the largest + reduction in the sending rate in one round-trip time, in a + deterministic environment with a packet drop exactly every 1/p + packets. The responsiveness is defined as the number of round-trip + times of sustained congestion required for the sender to halve the + sending rate; aggressiveness is defined as the maximum increase in + the sending rate in one round-trip time, in packets per second, in + the absence of congestion. This aggressiveness can be a function of + the mode of the transport protocol; for TCP, the aggressiveness of + slow-start is quite different from the aggressiveness of congestion + avoidance mode. + +2.2.2. Minimizing Oscillations + + One goal is that of stability, in terms of minimizing oscillations of + queueing delay or of throughput. In practice, stability is + frequently associated with rate fluctuations or variance. Rate + variations can result in fluctuations in router queue size and + therefore of queue overflows. These queue overflows can cause loss + synchronizations across coexisting flows and periodic + under-utilization of link capacity, both of which are considered to + be general signs of network instability. Thus, measuring the rate + variations of flows is often used to measure the stability of + transport protocols. To measure rate variations, [JWL04], [RX05], + and [FHPW00] use the coefficient of variation (CoV) of per-flow + transmission rates, and [WCL05] suggests the use of standard + deviations of per-flow rates. Since rate variations are a function + of time scales, it makes sense to measure these rate variations over + various time scales. + + Measuring per-flow rate variations, however, is only one aspect of + transport protocol stability. A realistic experimental setting + always involves multiple flows of the transport protocol being + observed, along with a significant amount of cross traffic, with + rates varying over time on both the forward and reverse paths. As a + congestion control protocol must adapt its rate to the varying rates + of competing traffic, just measuring the per-flow statistics of a + subset of the traffic could be misleading because it measures the + + + +Floyd Informational [Page 8] + +RFC 5166 TMRG, METRICS March 2008 + + + rate fluctuations due in part to the adaptation to competing traffic + on the path. Thus, per-flow statistics are most meaningful if they + are accompanied by the statistics measured at the network level. As + a complementary metric to the per-flow statistics, [HKLRX06] uses + measurements of the rate variations of the aggregate flows observed + in bottleneck routers over various time scales. + + Minimizing oscillations in queueing delay or throughput has related + per-flow metrics of minimizing jitter in round-trip times and loss + rates. + + An orthogonal goal for some congestion control mechanisms, e.g., for + equation-based congestion control, is to minimize the oscillations in + the sending rate for an individual connection, given an environment + with a fixed, steady-state packet drop rate. (As is well known, TCP + congestion control is characterized by a pronounced oscillation in + the sending rate, with the sender halving the sending rate in + response to congestion.) One metric for the level of oscillations is + the smoothness metric given in Section 2.2.1 above. + + As discussed in [FK07], the synchronization of loss events can also + affect convergence times, throughput, and delay. + +2.3. Fairness and Convergence + + Another set of metrics is that of fairness and convergence times. + Fairness can be considered between flows of the same protocol and + between flows using different protocols (e.g., TCP-friendliness, + referring to fairness between TCP and a new transport protocol). + Fairness can also be considered between sessions, between users, or + between other entities. + + There are a number of different fairness measures. These include + max-min fairness [HG86], proportional fairness [KMT98] [K01], the + fairness index proposed in [JCH84], and the product measure, a + variant of network power [BJ81]. + + + + + + + + + + + + + + + +Floyd Informational [Page 9] + +RFC 5166 TMRG, METRICS March 2008 + + +2.3.1. Metrics for Fairness between Flows + + This section discusses fairness metrics that consider the fairness + between flows, but that don't take into account different + characteristics of flows (e.g., the number of links in the path or + the round-trip time). For the discussion of fairness metrics, let + x_i be the throughput for the i-th connection. + + Jain's fairness index: The fairness index in [JCH84] is: + + (( sum_i x_i )^2) / (n * sum_i ( (x_i)^2 )), + + where there are n users. This fairness index ranges from 0 to 1, and + it is maximum when all users receive the same allocation. This index + is k/n when k users equally share the resource, and the other n-k + users receive zero allocation. + + The product measure: The product measure: + + product_i x_i , + + the product of the throughput of the individual connections, is also + used as a measure of fairness. (In some contexts x_i is taken as the + power of the i-th connection, and the product measure is referred to + as network power.) The product measure is particularly sensitive to + segregation; the product measure is zero if any connection receives + zero throughput. In [MS91], it is shown that for a network with many + connections and one shared gateway, the product measure is maximized + when all connections receive the same throughput. + + Epsilon-fairness: A rate allocation is defined as epsilon-fair if + + (min_i x_i) / (max_i x_i) >= 1 - epsilon. + + Epsilon-fairness measures the worst-case ratio between any two + throughput rates [ZKL04]. Epsilon-fairness is related to max-min + fairness, defined later in this document. + +2.3.2. Metrics for Fairness between Flows with Different Resource + Requirements + + This section discusses metrics for fairness between flows with + different resource requirements, that is, with different utility + functions, round-trip times, or number of links on the path. Many of + these metrics can be described as solutions to utility maximization + problems [K01]; the total utility quantifies both the fairness and + the throughput. + + + + +Floyd Informational [Page 10] + +RFC 5166 TMRG, METRICS March 2008 + + + Max-min fairness: In order to satisfy the max-min fairness criteria, + the smallest throughput rate must be as large as possible. Given + this condition, the next-smallest throughput rate must be as large as + possible, and so on. Thus, the max-min fairness gives absolute + priority to the smallest flows. (Max-min fairness can be explained + by the progressive filling algorithm, where all flow rates start at + zero, and the rates all grow at the same pace. Each flow rate stops + growing only when one or more links on the path reach link capacity.) + + Proportional fairness: In contrast, a feasible allocation, x, is + defined as proportionally fair if, for any other feasible allocation + x*, the aggregate of proportional changes is zero or negative: + + sum_i ( (x*_i - x_i)/x_i ) <= 0. + + "This criterion favours smaller flows, but less emphatically than + max-min fairness" [K01]. (Using the language of utility functions, + proportional fairness can be achieved by using logarithmic utility + functions, and maximizing the sum of the per-flow utility functions; + see [KMT98] for a fuller explanation.) + + Minimum potential delay fairness: Minimum potential delay fairness + has been shown to model TCP [KS03], and is a compromise between + max-min fairness and proportional fairness. An allocation, x, is + defined as having minimum potential delay fairness if: + + sum_i (1/x_i) + + is smaller than for any other feasible allocation. That is, it would + minimize the average download time if each flow was an equal-sized + file. + + In [CRM05], Colussi, et al. propose a new definition of TCP fairness, + that "a set of TCP fair flows do not cause more congestion than a set + of TCP flows would cause", where congestion is defined in terms of + queueing delay, queueing delay variation, the congestion event rate + [e.g., loss event rate], and the packet loss rate. + + Chiu and Tan in [CT06] argue for redefining the notion of fairness + when studying traffic controls for inelastic traffic, proposing that + inelastic flows adopt other traffic controls such as admission + control. + + The usefulness of flow-rate fairness has been challenged in [B07] by + Briscoe, and defended in [FA08] by Floyd and Allman. In [B07], + Briscoe argues that flow-rate fairness should not be a desired goal, + and that instead "we should judge fairness mechanisms on how they + share out the 'cost' of each user's actions on others". Floyd and + + + +Floyd Informational [Page 11] + +RFC 5166 TMRG, METRICS March 2008 + + + Allman in [FA08] argue that the current system based on TCP + congestion control and flow-rate fairness has been useful in the real + world, posing minimal demands on network and economic infrastructure + and enabling users to get a share of the network resources. + +2.3.3. Comments on Fairness + + Trade-offs between fairness and throughput: The fairness measures in + the section above generally measure both fairness and throughput, + giving different weights to each. Potential trade-offs between + fairness and throughput are also discussed by Tang, et al. in + [TWL06], for a framework where max-min fairness is defined as the + most fair. In particular, [TWL06] shows that in some topologies, + throughput is proportional to fairness, while in other topologies, + throughput is inversely proportional to fairness. + + Fairness and the number of congested links: Some of these fairness + metrics are discussed in more detail in [F91]. We note that there is + not a clear consensus for the fairness goals, in particular for + fairness between flows that traverse different numbers of congested + links [F91]. Utility maximization provides one framework for + describing this trade-off in fairness. + + Fairness and round-trip times: One goal cited in a number of new + transport protocols has been that of fairness between flows with + different round-trip times [KHR02] [XHR04]. We note that there is + not a consensus in the networking community about the desirability of + this goal, or about the implications and interactions between this + goal and other metrics [FJ92] (Section 3.3). One common argument + against the goal of fairness between flows with different round-trip + times has been that flows with long round-trip times consume more + resources; this aspect is covered by the previous paragraph. + Researchers have also noted the difference between the RTT-unfairness + of standard TCP, and the greater RTT-unfairness of some proposed + modifications to TCP [LLS05]. + + Fairness and packet size: One fairness issue is that of the relative + fairness for flows with different packet sizes. Many file transfer + applications will use the maximum packet size possible; in contrast, + low-bandwidth VoIP flows are likely to send small packets, sending a + new packet every 10 to 40 ms., to limit delay. Should a small-packet + VoIP connection receive the same sending rate in *bytes* per second + as a large-packet TCP connection in the same environment, or should + it receive the same sending rate in *packets* per second? This + fairness issue has been discussed in more detail in [RFC3714], with + [RFC4828] also describing the ways that packet size can affect the + packet drop rate experienced by a flow. + + + + +Floyd Informational [Page 12] + +RFC 5166 TMRG, METRICS March 2008 + + + Convergence times: Convergence times concern the time for convergence + to fairness between an existing flow and a newly starting one, and + are a special concern for environments with high-bandwidth long-delay + flows. Convergence times also concern the time for convergence to + fairness after a sudden change such as a change in the network path, + the competing cross-traffic, or the characteristics of a wireless + link. As with fairness, convergence times can matter both between + flows of the same protocol, and between flows using different + protocols [SLFK03]. One metric used for convergence times is the + delta-fair convergence time, defined as the time taken for two flows + with the same round-trip time to go from shares of 100/101-th and + 1/101-th of the link bandwidth, to having close to fair sharing with + shares of (1+delta)/2 and (1-delta)/2 of the link bandwidth [BBFS01]. + A similar metric for convergence times measures the convergence time + as the number of round-trip times for two flows to reach epsilon- + fairness, when starting from a maximally-unfair state [ZKL04]. + +2.4. Robustness for Challenging Environments + + While congestion control mechanisms are generally evaluated first + over environments with static routing in a network of two-way + point-to-point links, some environments bring up more challenging + problems (e.g., corrupted packets, reordering, variable bandwidth, + and mobility) as well as new metrics to be considered (e.g., energy + consumption). + + Robustness for challenging environments: Robustness needs to be + explored for paths with reordering, corruption, variable bandwidth, + asymmetric routing, router configuration changes, mobility, and the + like [GF04]. In general, the Internet architecture has valued + robustness over efficiency, e.g., when there are trade-offs between + robustness and the throughput, delay, and fairness metrics described + above. + + Energy consumption: In mobile environments, the energy consumption + for the mobile end-node can be a key metric that is affected by the + transport protocol [TM02]. + + The goodput ratio: For wireless networks, the goodput ratio can be a + useful metric, where the goodput ratio can be defined as the useful + data delivered to users as a fraction of the total amount of data + transmitted on the network. A high goodput ratio indicates an + efficient use of the radio spectrum and lower interference with other + users. + + + + + + + +Floyd Informational [Page 13] + +RFC 5166 TMRG, METRICS March 2008 + + +2.5. Robustness to Failures and to Misbehaving Users + + One goal is for congestion control mechanisms to be robust to + misbehaving users, such as receivers that 'lie' to data senders about + the congestion experienced along the path or otherwise attempt to + bypass the congestion control mechanisms of the sender [SCWA99]. + Another goal is for congestion control mechanisms to be as robust as + possible to failures, such as failures of routers in using explicit + feedback to end-nodes or failures of end-nodes to follow the + prescribed protocols. + +2.6. Deployability + + One metric for congestion control mechanisms is their deployability + in the current Internet. Metrics related to deployability include + the ease of failure diagnosis and the overhead in terms of packet + header size or added complexity at end-nodes or routers. + + One key aspect of deployability concerns the range of deployment + needed for a new congestion control mechanism. Consider the + following possible deployment requirements: + + * Only at the sender (e.g., NewReno in TCP [RFC3782]); + + * Only at the receiver (e.g., delayed acknowledgements in TCP); + + * Both the sender and receiver (e.g., Selective Acknowledgment + (SACK) TCP [RFC2018]); + + * At a single router (e.g., Random Early Detection (RED) [FJ93]); + + * All of the routers along the end-to-end path; + + * Both end-nodes and all routers along the path (e.g., Explicit + Control Protocol (XCP) [KHR02]). + + Some congestion control mechanisms (e.g., XCP [KHR02], Quick-Start + [RFC4782]) may also have deployment issues with IPsec, IP in IP, + MPLS, other tunnels, or with non-router queues such as those in + Ethernet switches. + + + + + + + + + + + +Floyd Informational [Page 14] + +RFC 5166 TMRG, METRICS March 2008 + + + Another deployability issue concerns the complexity of the code. How + complex is the code required to implement the mechanism in software? + Is floating point math required? How much new state must be kept to + implement the new mechanism, and who holds that state, the routers or + the end-nodes? We note that we don't suggest these questions as ways + to reduce the deployability metric to a single number; we suggest + them as issues that could be considered in evaluating the + deployability of a proposed congestion control mechanism. + +2.7. Metrics for Specific Types of Transport + + In some cases, modified metrics are needed for evaluating transport + protocols intended for quality-of-service (QoS)-enabled environments + or for below-best-effort traffic [VKD02] [KK03]. + +2.8. User-Based Metrics + + An alternate approach that has been proposed for the evaluation of + congestion control mechanisms would be to evaluate in terms of user + metrics, such as user satisfaction or in terms of + application-specific utility functions. Such an approach would + require the definition of a range of user metrics or of + application-specific utility functions for the range of applications + under consideration (e.g., FTP, HTTP, VoIP). + +3. Metrics in the IP Performance Metrics (IPPM) Working Group + + The IPPM Working Group [IPPM] was established to define performance + metrics to be used by network operators, end users, or independent + testing groups. The metrics include metrics for connectivity + [RFC2678], delay and loss [RFC2679], [RFC2680], and [RFC2681], delay + variation [RFC3393], loss patterns [RFC3357], packet reordering + [RFC4737], bulk transfer capacity [RFC3148], and link capacity + [RFC5136]. The IPPM documents give concrete, well-defined metrics, + along with a methodology for measuring the metric. The metrics + discussed in this document have a different purpose from the IPPM + metrics; in this document, we are discussing metrics as used in + analysis, simulations, and experiments for the evaluation of + congestion control mechanisms. Further, we are discussing these + metrics in a general sense, rather than looking for specific concrete + definitions for each metric. However, there are many cases where the + metric definitions from IPPM could be useful, for specific issues of + how to measure these metrics in simulations, or in testbeds, and for + providing common definitions for talking about these metrics. + + + + + + + +Floyd Informational [Page 15] + +RFC 5166 TMRG, METRICS March 2008 + + +4. Comments on Methodology + + The types of scenarios that are used to test specific metrics, and + the range of parameters that it is useful to consider, will be + discussed in separate documents, e.g., along with specific scenarios + for use in evaluating congestion control mechanisms. + + We note that it can be important to evaluate metrics over a wide + range of environments, with a range of link bandwidths, congestion + levels, and levels of statistical multiplexing. It is also important + to evaluate congestion control mechanisms in a range of scenarios, + including typical ranges of connection sizes and round-trip times + [FK02]. It is also useful to compare metrics for new or modified + transport protocols with those of the current standards for TCP. + + As an example from the literature on evaluating transport protocols, + Li, et al. in "Experimental Evaluation of TCP Protocols for High- + Speed Networks" [LLS05] focus on the performance of TCP in high- + speed networks, and consider metrics for aggregate throughput, loss + rates, fairness (including fairness between flows with different + round-trip times), response times (including convergence times), and + incremental deployment. More general references on methodology + include [J91]. Papers that discuss the range of metrics for + evaluating congestion control include [MTZ04]. + +5. Security Considerations + + Section 2.5 discusses the robustness of congestion control mechanisms + to failures and to misbehaving users. Transport protocols also have + other security concerns that are unrelated to congestion control + mechanisms; these are not discussed in this document. + +6. Acknowledgements + + Thanks to Lachlan Andrew, Mark Allman, Armando Caro, Dah Ming Chiu, + Eric Coe, Dado Colussi, Wesley Eddy, Aaron Falk, Nelson Fonseca, + Janardhan Iyengar, Doug Leith, Sara Landstrom, Tony Li, Saverio + Mascolo, Sean Moore, Injong Rhee, David Ros, Juergen Schoenwaelder, + Andras Veres, Michael Welzl, and Damon Wischik, and members of the + Transport Modeling Research Group for feedback and contributions. + + + + + + + + + + + +Floyd Informational [Page 16] + +RFC 5166 TMRG, METRICS March 2008 + + +7. Informative References + + [AEO03] M. Allman, W. Eddy, and S. Ostermann, Estimating Loss Rates + With TCP, ACM Performance Evaluation Review, 31(3), + December 2003. + + [BB01] D. Bansal and H. Balakrishnan, Binomial Congestion Control + Algorithms, IEEE Infocom, April 2001. + + [BBFS01] D. Bansal, H. Balakrishnan, S. Floyd, and S. Shenker, + Dynamic Behavior of Slowly-Responsive Congestion Control + Algorithms, SIGCOMM 2001. + + [BJ81] K. Bharath-Kumar and J. Jeffrey, A New Approach to + Performance-Oriented Flow Control, IEEE Transactions on + Communications, Vol.29 N.4, April 1981. + + [B07] B. Briscoe, "Flow Rate Fairness: Dismantling a Religion", + Computer Communications Review, V.37 N.2, April 2007. + + [CRM05] D. Colussi, A New Approach to TCP-Fairness, Report C-2005- + 49, University of Helsinki, Finland, 2005. + + [CT06] D. Chiu and A. Tam, Redefining Fairness in the Study of + TCP-friendly Traffic Controls, Technical Report, 2006. + + [DM06] N. Dukkipati and N. McKeown, Why Flow-Completion Time is + the Right Metric for Congestion Control, ACM SIGCOMM, + January 2006. + + [F91] S. Floyd, Connections with Multiple Congested Gateways in + Packet-Switched Networks Part 1: One-way Traffic, Computer + Communication Review, Vol.21 No.5, October 1991, p. 30-47. + + [FA08] S. Floyd and M. Allman, Comments on the Usefulness of + Simple Best-Effort Traffic, Work in Progress, January 2007. + + [FF99] Floyd, S., and Fall, K., "Promoting the Use of End-to-End + Congestion Control in the Internet", IEEE/ACM Transactions + on Networking, August 1999. + + [FHP00] S. Floyd, M. Handley, and J. Padhye, A Comparison of + Equation-Based and AIMD Congestion Control, May 2000. URL + http://www.icir.org/tfrc/. + + [FHPW00] S. Floyd, M. Handley, J. Padhye, and J. Widmer, Equation- + Based Congestion Control for Unicast Applications, SIGCOMM + 2000, August 2000. + + + +Floyd Informational [Page 17] + +RFC 5166 TMRG, METRICS March 2008 + + + [FJ92] S. Floyd and V. Jacobson, On Traffic Phase Effects in + Packet-Switched Gateways, Internetworking: Research and + Experience, V.3 N.3, September 1992, p.115-156. + + [FJ93] S. Floyd and V. Jacobson, Random Early Detection gateways + for Congestion Avoidance, IEEE/ACM Transactions on + Networking, V.1 N.4, August 1993, + + [FK02] S. Floyd and E. Kohler, Internet Research Needs Better + Models, Hotnets-I. October 2002. + + [FK07] S. Floyd and E. Kohler, "Tools for the Evaluation of + Simulation and Testbed Scenarios", Work in Progress, + February 2008. + + [GF04] A. Gurtov and S. Floyd, Modeling Wireless Links for + Transport Protocols, ACM CCR, 34(2):85-96, April 2004. + + [HKLRX06] S. Ha, Y. Kim, L. Le, I. Rhee, and L. Xu, A Step Toward + Realistic Evaluation of High-speed TCP Protocols, technical + report, North Carolina State University, January 2006. + + [HG86] E. Hahne and R. Gallager, Round Robin Scheduling for Fair + Flow Control in Data Communications Networks, IEEE + International Conference on Communications, June 1986. + + [IPPM] IP Performance Metrics (IPPM) Working Group, URL + http://www.ietf.org/html.charters/ippm-charter.html. + + [J91] R. Jain, The Art of Computer Systems Performance Analysis: + Techniques for Experimental Design, Measurement, + Simulation, and Modeling, John Wiley & Sons, 1991. + + [JCH84] R. Jain, D.M. Chiu, and W. Hawe, A Quantitative Measure of + Fairness and Discrimination for Resource Allocation in + Shared Systems, DEC TR-301, Littleton, MA: Digital + Equipment Corporation, 1984. + + [JWL04] C. Jin, D. Wei, and S. Low, FAST TCP: Motivation, + Architecture, Algorithms, Performance, IEEE INFOCOM, March + 2004. + + [K01] F. Kelly, Mathematical Modelling of the Internet, + "Mathematics Unlimited - 2001 and Beyond" (Editors B. + Engquist and W. Schmid), Springer-Verlag, Berlin, pp. + 685-702, 2001. + + + + + +Floyd Informational [Page 18] + +RFC 5166 TMRG, METRICS March 2008 + + + [KHR02] D. Katabi, M. Handley, and C. Rohrs, Congestion Control for + High Bandwidth-Delay Product Networks, ACM Sigcomm, 2002. + + [KK03] A. Kuzmanovic and E. W. Knightly, TCP-LP: A Distributed + Algorithm for Low Priority Data Transfer, IEEE INFOCOM + 2003, April 2003. + + [KMT98] F. Kelly, A. Maulloo and D. Tan, Rate Control in + Communication Networks: Shadow Prices, Proportional + Fairness and Stability. Journal of the Operational + Research Society 49, pp. 237-252, 1998. + + [KS03] S. Kunniyur and R. Srikant, End-to-end Congestion Control + Schemes: Utility Functions, Random Losses and ECN Marks, + IEEE/ACM Transactions on Networking, 11(5):689-702, October + 2003. + + [LLS05] Y-T. Li, D. Leith, and R. Shorten, Experimental Evaluation + of TCP Protocols for High-Speed Networks, Hamilton + Institute, 2005. URL + http://www.hamilton.ie/net/eval/results_HI2005.pdf. + + [MS91] D. Mitra and J. Seery, Dynamic Adaptive Windows for High + Speed Data Networks with Multiple Paths and Propagation + Delays, INFOCOM '91, pp 39-48. + + [MTZ04] L. Mamatas, V. Tsaoussidis, and C. Zhang, Approaches to + Congestion Control in Packet Networks, 2004. + + [RFC2018] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP + Selective Acknowledgment Options", RFC 2018, October 1996. + + [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way + Packet Loss Metric for IPPM", RFC 2680, September 1999. + + [RFC2678] Mahdavi, J. and V. Paxson, "IPPM Metrics for Measuring + Connectivity", RFC 2678, September 1999. + + [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way + Delay Metric for IPPM", RFC 2679, September 1999. + + [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip + Delay Metric for IPPM", RFC 2681, September 1999. + + [RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, RFC + 2914, September 2000. + + + + + +Floyd Informational [Page 19] + +RFC 5166 TMRG, METRICS March 2008 + + + [RFC3148] Mathis, M. and M. Allman, "A Framework for Defining + Empirical Bulk Transfer Capacity Metrics", RFC 3148, July + 2001. + + [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of + Explicit Congestion Notification (ECN) to IP", RFC 3168, + September 2001. + + [RFC3357] Koodli, R. and R. Ravikanth, "One-way Loss Pattern Sample + Metrics", RFC 3357, August 2002. + + [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation + Metric for IP Performance Metrics (IPPM)", RFC 3393, + November 2002. + + [RFC3448] Handley, M., Floyd, S., Padhye, J., and J. Widmer, "TCP + Friendly Rate Control (TFRC): Protocol Specification", RFC + 3448, January 2003. + + [RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed., + "RTP Control Protocol Extended Reports (RTCP XR)", RFC + 3611, November 2003. + + [RFC3714] Floyd, S., Ed., and J. Kempf, Ed., "IAB Concerns Regarding + Congestion Control for Voice Traffic in the Internet", RFC + 3714, March 2004. + + [RFC3782] Floyd, S., Henderson, T., and A. Gurtov, "The NewReno + Modification to TCP's Fast Recovery Algorithm", RFC 3782, + April 2004. + + [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, S., + and J. Perser, "Packet Reordering Metrics", RFC 4737, + November 2006. + + [RFC4782] Floyd, S., Allman, M., Jain, A., and P. Sarolahti, "Quick- + Start for TCP and IP", RFC 4782, January 2007. + + [RFC4828] Floyd, S. and E. Kohler, "TCP Friendly Rate Control (TFRC): + The Small-Packet (SP) Variant", RFC 4828, April 2007. + + [RFC5136] Chimento, P. and J. Ishac, "Defining Network Capacity", RFC + 5136, February 2008. + + [RX05] I. Rhee and L. Xu, CUBIC: A New TCP-Friendly High-Speed TCP + Variant, PFLDnet 2005. + + + + + +Floyd Informational [Page 20] + +RFC 5166 TMRG, METRICS March 2008 + + + [SAF06] P. Sarolahti, M. Allman, and S. Floyd, Determining an + Appropriate Sending Rate Over an Underutilized Network + Path, Computer Networks, September 2006. + + [SLFK03] R.N. Shorten, D.J. Leith, J. Foy, and R. Kilduff, Analysis + and Design of Congestion Control in Synchronised + Communication Networks. Proc. 12th Yale Workshop on + Adaptive & Learning Systems, May 2003. + + [SCWA99] S. Savage, N. Cardwell, D. Wetherall, and T. Anderson, TCP + Congestion Control with a Misbehaving Receiver, ACM + Computer Communications Review, October 1999. + + [TM02] V. Tsaoussidis and I. Matta, Open Issues of TCP for Mobile + Computing, Journal of Wireless Communications and Mobile + Computing: Special Issue on Reliable Transport Protocols + for Mobile Computing, February 2002. + + [TWL06] A. Tang, J. Wang and S. H. Low. Counter-Intuitive + Throughput Behaviors in Networks Under End-to-End Control, + IEEE/ACM Transactions on Networking, 14(2):355-368, April + 2006. + + [WCL05] D. X. Wei, P. Cao and S. H. Low, Time for a TCP Benchmark + Suite?, Technical Report, Caltech CS, Stanford EAS, + Caltech, 2005. + + [VKD02] A. Venkataramani, R. Kokku, and M. Dahlin, TCP Nice: A + Mechanism for Background Transfers, Fifth USENIX Symposium + on Operating System Design and Implementation (OSDI), 2002. + + [XHR04] L. Xu, K. Harfoush, and I. Rhee, Binary Increase Congestion + Control for Fast, Long Distance Networks, Infocom 2004. + + [YKL01] Y. Yang, M. Kim, and S. Lam, Transient Behaviors of TCP- + friendly Congestion Control Protocols, Infocom 2001. + + [ZKL04] Y. Zhang, S.-R. Kang, and D. Loguinov, Delayed Stability + and Performance of Distributed Congestion Control, ACM + SIGCOMM, August 2004. + + + + + + + + + + + +Floyd Informational [Page 21] + +RFC 5166 TMRG, METRICS March 2008 + + +Author's Address + + Sally Floyd + ICSI Center for Internet Research + 1947 Center Street, Suite 600 + Berkeley, CA 94704 + USA + + EMail: floyd@icir.org + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Floyd Informational [Page 22] + +RFC 5166 TMRG, METRICS March 2008 + + +Full Copyright Statement + + Copyright (C) The IETF Trust (2008). + + This document is subject to the rights, licenses and restrictions + contained in BCP 78 and at http://www.rfc-editor.org/copyright.html, + and except as set forth therein, the authors retain all their rights. + + This document and the information contained herein are provided on an + "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS + OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND + THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS + OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF + THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED + WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + +Intellectual Property + + The IETF takes no position regarding the validity or scope of any + Intellectual Property Rights or other rights that might be claimed to + pertain to the implementation or use of the technology described in + this document or the extent to which any license under such rights + might or might not be available; nor does it represent that it has + made any independent effort to identify any such rights. Information + on the procedures with respect to rights in RFC documents can be + found in BCP 78 and BCP 79. + + Copies of IPR disclosures made to the IETF Secretariat and any + assurances of licenses to be made available, or the result of an + attempt made to obtain a general license or permission for the use of + such proprietary rights by implementers or users of this + specification can be obtained from the IETF on-line IPR repository at + http://www.ietf.org/ipr. + + The IETF invites any interested party to bring to its attention any + copyrights, patents or patent applications, or other proprietary + rights that may cover technology that may be required to implement + this standard. Please address the information to the IETF at + ietf-ipr@ietf.org. + + + + + + + + + + + + +Floyd Informational [Page 23] + |