diff options
Diffstat (limited to 'doc/rfc/rfc7141.txt')
| -rw-r--r-- | doc/rfc/rfc7141.txt | 2299 |
1 files changed, 2299 insertions, 0 deletions
diff --git a/doc/rfc/rfc7141.txt b/doc/rfc/rfc7141.txt new file mode 100644 index 0000000..8751058 --- /dev/null +++ b/doc/rfc/rfc7141.txt @@ -0,0 +1,2299 @@ + + + + + + +Internet Engineering Task Force (IETF) B. Briscoe +Request for Comments: 7141 BT +BCP: 41 J. Manner +Updates: 2309, 2914 Aalto University +Category: Best Current Practice February 2014 +ISSN: 2070-1721 + + + Byte and Packet Congestion Notification + +Abstract + + This document provides recommendations of best current practice for + dropping or marking packets using any active queue management (AQM) + algorithm, including Random Early Detection (RED), BLUE, Pre- + Congestion Notification (PCN), and newer schemes such as CoDel + (Controlled Delay) and PIE (Proportional Integral controller + Enhanced). We give three strong recommendations: (1) packet size + should be taken into account when transports detect and respond to + congestion indications, (2) packet size should not be taken into + account when network equipment creates congestion signals (marking, + dropping), and therefore (3) in the specific case of RED, the byte- + mode packet drop variant that drops fewer small packets should not be + used. This memo updates RFC 2309 to deprecate deliberate + preferential treatment of small packets in AQM algorithms. + +Status of This Memo + + This memo documents an Internet Best Current Practice. + + This document is a product of the Internet Engineering Task Force + (IETF). It represents the consensus of the IETF community. It has + received public review and has been approved for publication by the + Internet Engineering Steering Group (IESG). Further information on + BCPs is available in Section 2 of RFC 5741. + + Information about the current status of this document, any errata, + and how to provide feedback on it may be obtained at + http://www.rfc-editor.org/info/rfc7141. + + + + + + + + + + + + +Briscoe & Manner Best Current Practice [Page 1] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + +Copyright Notice + + Copyright (c) 2014 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (http://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the Simplified BSD License. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Briscoe & Manner Best Current Practice [Page 2] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + +Table of Contents + + 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 + 1.1. Terminology and Scoping . . . . . . . . . . . . . . . . . 6 + 1.2. Example Comparing Packet-Mode Drop and Byte-Mode Drop . . 7 + 2. Recommendations . . . . . . . . . . . . . . . . . . . . . . . 9 + 2.1. Recommendation on Queue Measurement . . . . . . . . . . . 9 + 2.2. Recommendation on Encoding Congestion Notification . . . 10 + 2.3. Recommendation on Responding to Congestion . . . . . . . 11 + 2.4. Recommendation on Handling Congestion Indications When + Splitting or Merging Packets . . . . . . . . . . . . . . 12 + 3. Motivating Arguments . . . . . . . . . . . . . . . . . . . . 13 + 3.1. Avoiding Perverse Incentives to (Ab)use Smaller Packets . 13 + 3.2. Small != Control . . . . . . . . . . . . . . . . . . . . 14 + 3.3. Transport-Independent Network . . . . . . . . . . . . . . 14 + 3.4. Partial Deployment of AQM . . . . . . . . . . . . . . . . 16 + 3.5. Implementation Efficiency . . . . . . . . . . . . . . . . 17 + 4. A Survey and Critique of Past Advice . . . . . . . . . . . . 17 + 4.1. Congestion Measurement Advice . . . . . . . . . . . . . . 18 + 4.1.1. Fixed-Size Packet Buffers . . . . . . . . . . . . . . 18 + 4.1.2. Congestion Measurement without a Queue . . . . . . . 19 + 4.2. Congestion Notification Advice . . . . . . . . . . . . . 20 + 4.2.1. Network Bias When Encoding . . . . . . . . . . . . . 20 + 4.2.2. Transport Bias When Decoding . . . . . . . . . . . . 22 + 4.2.3. Making Transports Robust against Control Packet + Losses . . . . . . . . . . . . . . . . . . . . . . . 23 + 4.2.4. Congestion Notification: Summary of Conflicting + Advice . . . . . . . . . . . . . . . . . . . . . . . 24 + 5. Outstanding Issues and Next Steps . . . . . . . . . . . . . . 25 + 5.1. Bit-congestible Network . . . . . . . . . . . . . . . . . 25 + 5.2. Bit- and Packet-Congestible Network . . . . . . . . . . . 26 + 6. Security Considerations . . . . . . . . . . . . . . . . . . . 26 + 7. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 27 + 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 28 + 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 28 + 9.1. Normative References . . . . . . . . . . . . . . . . . . 28 + 9.2. Informative References . . . . . . . . . . . . . . . . . 29 + Appendix A. Survey of RED Implementation Status . . . . . . . . 33 + Appendix B. Sufficiency of Packet-Mode Drop . . . . . . . . . . 34 + B.1. Packet-Size (In)Dependence in Transports . . . . . . . . 35 + B.2. Bit-Congestible and Packet-Congestible Indications . . . 38 + Appendix C. Byte-Mode Drop Complicates Policing Congestion + Response . . . . . . . . . . . . . . . . . . . . . . 39 + + + + + + + + +Briscoe & Manner Best Current Practice [Page 3] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + +1. Introduction + + This document provides recommendations of best current practice for + how we should correctly scale congestion control functions with + respect to packet size for the long term. It also recognises that + expediency may be necessary to deal with existing widely deployed + protocols that don't live up to the long-term goal. + + When signalling congestion, the problem of how (and whether) to take + packet sizes into account has exercised the minds of researchers and + practitioners for as long as active queue management (AQM) has been + discussed. Indeed, one reason AQM was originally introduced was to + reduce the lock-out effects that small packets can have on large + packets in tail-drop queues. This memo aims to state the principles + we should be using and to outline how these principles will affect + future protocol design, taking into account pre-existing deployments. + + The question of whether to take into account packet size arises at + three stages in the congestion notification process: + + Measuring congestion: When a congested resource measures locally how + congested it is, should it measure its queue length in time, + bytes, or packets? + + Encoding congestion notification into the wire protocol: When a + congested network resource signals its level of congestion, should + the probability that it drops/marks each packet depend on the size + of the particular packet in question? + + Decoding congestion notification from the wire protocol: When a + transport interprets the notification in order to decide how much + to respond to congestion, should it take into account the size of + each missing or marked packet? + + Consensus has emerged over the years concerning the first stage, + which Section 2.1 records in the RFC Series. In summary: If + possible, it is best to measure congestion by time in the queue; + otherwise, the choice between bytes and packets solely depends on + whether the resource is congested by bytes or packets. + + The controversy is mainly around the last two stages: whether to + allow for the size of the specific packet notifying congestion i) + when the network encodes or ii) when the transport decodes the + congestion notification. + + Currently, the RFC series is silent on this matter other than a paper + trail of advice referenced from [RFC2309], which conditionally + recommends byte-mode (packet-size dependent) drop [pktByteEmail]. + + + +Briscoe & Manner Best Current Practice [Page 4] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + Reducing the number of small packets dropped certainly has some + tempting advantages: i) it drops fewer control packets, which tend to + be small and ii) it makes TCP's bit rate less dependent on packet + size. However, there are ways of addressing these issues at the + transport layer, rather than reverse engineering network forwarding + to fix the problems. + + This memo updates [RFC2309] to deprecate deliberate preferential + treatment of packets in AQM algorithms solely because of their size. + It recommends that (1) packet size should be taken into account when + transports detect and respond to congestion indications, (2) not when + network equipment creates them. This memo also adds to the + congestion control principles enumerated in BCP 41 [RFC2914]. + + In the particular case of Random Early Detection (RED), this means + that the byte-mode packet drop variant should not be used to drop + fewer small packets, because that creates a perverse incentive for + transports to use tiny segments, consequently also opening up a DoS + vulnerability. Fortunately, all the RED implementers who responded + to our admittedly limited survey (Section 4.2.4) have not followed + the earlier advice to use byte-mode drop, so the position this memo + argues for seems to already exist in implementations. + + However, at the transport layer, TCP congestion control is a widely + deployed protocol that doesn't scale with packet size (i.e., its + reduction in rate does not take into account the size of a lost + packet). To date, this hasn't been a significant problem because + most TCP implementations have been used with similar packet sizes. + But, as we design new congestion control mechanisms, this memo + recommends that we build in scaling with packet size rather than + assuming that we should follow TCP's example. + + This memo continues as follows. First, it discusses terminology and + scoping. Section 2 gives concrete formal recommendations, followed + by motivating arguments in Section 3. We then critically survey the + advice given previously in the RFC Series and the research literature + (Section 4), referring to an assessment of whether or not this advice + has been followed in production networks (Appendix A). To wrap up, + outstanding issues are discussed that will need resolution both to + inform future protocol designs and to handle legacy AQM deployments + (Section 5). Then security issues are collected together in + Section 6 before conclusions are drawn in Section 7. The interested + reader can find discussion of more detailed issues on the theme of + byte vs. packet in the appendices. + + This memo intentionally includes a non-negligible amount of material + on the subject. For the busy reader, Section 2 summarises the + recommendations for the Internet community. + + + +Briscoe & Manner Best Current Practice [Page 5] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + +1.1. Terminology and Scoping + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in [RFC2119]. + + This memo applies to the design of all AQM algorithms, for example, + Random Early Detection (RED) [RFC2309], BLUE [BLUE02], Pre-Congestion + Notification (PCN) [RFC5670], Controlled Delay (CoDel) [CoDel], and + the Proportional Integral controller Enhanced (PIE) [PIE]. + Throughout, RED is used as a concrete example because it is a widely + known and deployed AQM algorithm. There is no intention to imply + that the advice is any less applicable to the other algorithms, nor + that RED is preferred. + + Congestion Notification: Congestion notification is a changing + signal that aims to communicate the probability that the network + resource(s) will not be able to forward the level of traffic load + offered (or that there is an impending risk that they will not be + able to). + + The 'impending risk' qualifier is added, because AQM systems set a + virtual limit smaller than the actual limit to the resource, then + notify the transport when this virtual limit is exceeded in order + to avoid uncontrolled congestion of the actual capacity. + + Congestion notification communicates a real number bounded by the + range [ 0 , 1 ]. This ties in with the most well-understood + measure of congestion notification: drop probability. + + Explicit and Implicit Notification: The byte vs. packet dilemma + concerns congestion notification irrespective of whether it is + signalled implicitly by drop or explicitly using ECN [RFC3168] or + PCN [RFC5670]. Throughout this document, unless clear from the + context, the term 'marking' will be used to mean notifying + congestion explicitly, while 'congestion notification' will be + used to mean notifying congestion either implicitly by drop or + explicitly by marking. + + Bit-congestible vs. Packet-congestible: If the load on a resource + depends on the rate at which packets arrive, it is called 'packet- + congestible'. If the load depends on the rate at which bits + arrive, it is called 'bit-congestible'. + + + + + + + + +Briscoe & Manner Best Current Practice [Page 6] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + Examples of packet-congestible resources are route look-up engines + and firewalls, because load depends on how many packet headers + they have to process. Examples of bit-congestible resources are + transmission links, radio power, and most buffer memory, because + the load depends on how many bits they have to transmit or store. + Some machine architectures use fixed-size packet buffers, so + buffer memory in these cases is packet-congestible (see + Section 4.1.1). + + The path through a machine will typically encounter both packet- + congestible and bit-congestible resources. However, currently, a + design goal of network processing equipment such as routers and + firewalls is to size the packet-processing engine(s) relative to + the lines in order to keep packet processing uncongested, even + under worst-case packet rates with runs of minimum-size packets. + Therefore, packet congestion is currently rare (see Section 3.3 of + [RFC6077]), but there is no guarantee that it will not become more + common in the future. + + Note that information is generally processed or transmitted with a + minimum granularity greater than a bit (e.g., octets). The + appropriate granularity for the resource in question should be + used, but for the sake of brevity we will talk in terms of bytes + in this memo. + + Coarser Granularity: Resources may be congestible at higher levels + of granularity than bits or packets, for instance stateful + firewalls are flow-congestible and call-servers are session- + congestible. This memo focuses on congestion of connectionless + resources, but the same principles may be applicable for + congestion notification protocols controlling per-flow and per- + session processing or state. + + RED Terminology: In RED, whether to use packets or bytes when + measuring queues is called, respectively, 'packet-mode queue + measurement' or 'byte-mode queue measurement'. And whether the + probability of dropping a particular packet is independent or + dependent on its size is called, respectively, 'packet-mode drop' + or 'byte-mode drop'. The terms 'byte-mode' and 'packet-mode' + should not be used without specifying whether they apply to queue + measurement or to drop. + +1.2. Example Comparing Packet-Mode Drop and Byte-Mode Drop + + Taking RED as a well-known example algorithm, a central question + addressed by this document is whether to recommend RED's packet-mode + drop variant and to deprecate byte-mode drop. Table 1 compares how + packet-mode and byte-mode drop affect two flows of different size + + + +Briscoe & Manner Best Current Practice [Page 7] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + packets. For each it gives the expected number of packets and of + bits dropped in one second. Each example flow runs at the same bit + rate of 48 Mbps, but one is broken up into small 60 byte packets and + the other into large 1,500 byte packets. + + To keep up the same bit rate, in one second there are about 25 times + more small packets because they are 25 times smaller. As can be seen + from the table, the packet rate is 100,000 small packets versus 4,000 + large packets per second (pps). + + Parameter Formula Small packets Large packets + -------------------- --------------- ------------- ------------- + Packet size s/8 60 B 1,500 B + Packet size s 480 b 12,000 b + Bit rate x 48 Mbps 48 Mbps + Packet rate u = x/s 100 kpps 4 kpps + + Packet-mode Drop + Pkt-loss probability p 0.1% 0.1% + Pkt-loss rate p*u 100 pps 4 pps + Bit-loss rate p*u*s 48 kbps 48 kbps + + Byte-mode Drop MTU, M=12,000 b + Pkt-loss probability b = p*s/M 0.004% 0.1% + Pkt-loss rate b*u 4 pps 4 pps + Bit-loss rate b*u*s 1.92 kbps 48 kbps + + Table 1: Example Comparing Packet-Mode and Byte-Mode Drop + + For packet-mode drop, we illustrate the effect of a drop probability + of 0.1%, which the algorithm applies to all packets irrespective of + size. Because there are 25 times more small packets in one second, + it naturally drops 25 times more small packets, that is, 100 small + packets but only 4 large packets. But if we count how many bits it + drops, there are 48,000 bits in 100 small packets and 48,000 bits in + 4 large packets -- the same number of bits of small packets as large. + + The packet-mode drop algorithm drops any bit with the same + probability whether the bit is in a small or a large packet. + + For byte-mode drop, again we use an example drop probability of 0.1%, + but only for maximum size packets (assuming the link maximum + transmission unit (MTU) is 1,500 B or 12,000 b). The byte-mode + algorithm reduces the drop probability of smaller packets + proportional to their size, making the probability that it drops a + small packet 25 times smaller at 0.004%. But there are 25 times more + small packets, so dropping them with 25 times lower probability + results in dropping the same number of packets: 4 drops in both + + + +Briscoe & Manner Best Current Practice [Page 8] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + cases. The 4 small dropped packets contain 25 times less bits than + the 4 large dropped packets: 1,920 compared to 48,000. + + The byte-mode drop algorithm drops any bit with a probability + proportionate to the size of the packet it is in. + +2. Recommendations + + This section gives recommendations related to network equipment in + Sections 2.1 and 2.2, and we discuss the implications on transport + protocols in Sections 2.3 and 2.4. + +2.1. Recommendation on Queue Measurement + + Ideally, an AQM would measure the service time of the queue to + measure congestion of a resource. However service time can only be + measured as packets leave the queue, where it is not always expedient + to implement a full AQM algorithm. To predict the service time as + packets join the queue, an AQM algorithm needs to measure the length + of the queue. + + In this case, if the resource is bit-congestible, the AQM + implementation SHOULD measure the length of the queue in bytes and, + if the resource is packet-congestible, the implementation SHOULD + measure the length of the queue in packets. Subject to the + exceptions below, no other choice makes sense, because the number of + packets waiting in the queue isn't relevant if the resource gets + congested by bytes and vice versa. For example, the length of the + queue into a transmission line would be measured in bytes, while the + length of the queue into a firewall would be measured in packets. + + To avoid the pathological effects of tail drop, the AQM can then + transform this service time or queue length into the probability of + dropping or marking a packet (e.g., RED's piecewise linear function + between thresholds). + + What this advice means for RED as a specific example: + + 1. A RED implementation SHOULD use byte-mode queue measurement for + measuring the congestion of bit-congestible resources and packet- + mode queue measurement for packet-congestible resources. + + 2. An implementation SHOULD NOT make it possible to configure the + way a queue measures itself, because whether a queue is bit- + congestible or packet-congestible is an inherent property of the + queue. + + + + + +Briscoe & Manner Best Current Practice [Page 9] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + Exceptions to these recommendations might be necessary, for instance + where a packet-congestible resource has to be configured as a proxy + bottleneck for a bit-congestible resource in an adjacent box that + does not support AQM. + + The recommended approach in less straightforward scenarios, such as + fixed-size packet buffers, resources without a queue, and buffers + comprising a mix of packet and bit-congestible resources, is + discussed in Section 4.1. For instance, Section 4.1.1 explains that + the queue into a line should be measured in bytes even if the queue + consists of fixed-size packet buffers, because the root cause of any + congestion is bytes arriving too fast for the line -- packets filling + buffers are merely a symptom of the underlying congestion of the + line. + +2.2. Recommendation on Encoding Congestion Notification + + When encoding congestion notification (e.g., by drop, ECN, or PCN), + the probability that network equipment drops or marks a particular + packet to notify congestion SHOULD NOT depend on the size of the + packet in question. As the example in Section 1.2 illustrates, to + drop any bit with probability 0.1%, it is only necessary to drop + every packet with probability 0.1% without regard to the size of each + packet. + + This approach ensures the network layer offers sufficient congestion + information for all known and future transport protocols and also + ensures no perverse incentives are created that would encourage + transports to use inappropriately small packet sizes. + + What this advice means for RED as a specific example: + + 1. The RED AQM algorithm SHOULD NOT use byte-mode drop, i.e., it + ought to use packet-mode drop. Byte-mode drop is more complex, + it creates the perverse incentive to fragment segments into tiny + pieces and it is vulnerable to floods of small packets. + + 2. If a vendor has implemented byte-mode drop, and an operator has + turned it on, it is RECOMMENDED that the operator use packet-mode + drop instead, after establishing if there are any implications on + the relative performance of applications using different packet + sizes. The unlikely possibility of some application-specific + legacy use of byte-mode drop is the only reason that all the + above recommendations on encoding congestion notification are not + phrased more strongly. + + + + + + +Briscoe & Manner Best Current Practice [Page 10] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + RED as a whole SHOULD NOT be switched off. Without RED, a tail- + drop queue biases against large packets and is vulnerable to + floods of small packets. + + Note well that RED's byte-mode queue drop is completely orthogonal to + byte-mode queue measurement and should not be confused with it. If a + RED implementation has a byte-mode but does not specify what sort of + byte-mode, it is most probably byte-mode queue measurement, which is + fine. However, if in doubt, the vendor should be consulted. + + A survey (Appendix A) showed that there appears to be little, if any, + installed base of the byte-mode drop variant of RED. This suggests + that deprecating byte-mode drop will have little, if any, incremental + deployment impact. + +2.3. Recommendation on Responding to Congestion + + When a transport detects that a packet has been lost or congestion + marked, it SHOULD consider the strength of the congestion indication + as proportionate to the size in octets (bytes) of the missing or + marked packet. + + In other words, when a packet indicates congestion (by being lost or + marked), it can be considered conceptually as if there is a + congestion indication on every octet of the packet, not just one + indication per packet. + + To be clear, the above recommendation solely describes how a + transport should interpret the meaning of a congestion indication, as + a long term goal. It makes no recommendation on whether a transport + should act differently based on this interpretation. It merely aids + interoperability between transports, if they choose to make their + actions depend on the strength of congestion indications. + + This definition will be useful as the IETF transport area continues + its programme of: + + o updating host-based congestion control protocols to take packet + size into account, and + + o making transports less sensitive to losing control packets like + SYNs and pure ACKs. + + + + + + + + + +Briscoe & Manner Best Current Practice [Page 11] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + What this advice means for the case of TCP: + + 1. If two TCP flows with different packet sizes are required to run + at equal bit rates under the same path conditions, this SHOULD be + done by altering TCP (Section 4.2.2), not network equipment (the + latter affects other transports besides TCP). + + 2. If it is desired to improve TCP performance by reducing the + chance that a SYN or a pure ACK will be dropped, this SHOULD be + done by modifying TCP (Section 4.2.3), not network equipment. + + To be clear, we are not recommending at all that TCPs under + equivalent conditions should aim for equal bit rates. We are merely + saying that anyone trying to do such a thing should modify their TCP + algorithm, not the network. + + These recommendations are phrased as 'SHOULD' rather than 'MUST', + because there may be cases where expediency dictates that + compatibility with pre-existing versions of a transport protocol make + the recommendations impractical. + +2.4. Recommendation on Handling Congestion Indications When Splitting + or Merging Packets + + Packets carrying congestion indications may be split or merged in + some circumstances (e.g., at an RTP / RTP Control Protocol (RTCP) + transcoder or during IP fragment reassembly). Splitting and merging + only make sense in the context of ECN, not loss. + + The general rule to follow is that the number of octets in packets + with congestion indications SHOULD be equivalent before and after + merging or splitting. This is based on the principle used above; + that an indication of congestion on a packet can be considered as an + indication of congestion on each octet of the packet. + + The above rule is not phrased with the word 'MUST' to allow the + following exception. There are cases in which pre-existing protocols + were not designed to conserve congestion-marked octets (e.g., IP + fragment reassembly [RFC3168] or loss statistics in RTCP receiver + reports [RFC3550] before ECN was added [RFC6679]). When any such + protocol is updated, it SHOULD comply with the above rule to conserve + marked octets. However, the rule may be relaxed if it would + otherwise become too complex to interoperate with pre-existing + implementations of the protocol. + + One can think of a splitting or merging process as if all the + incoming congestion-marked octets increment a counter and all the + outgoing marked octets decrement the same counter. In order to + + + +Briscoe & Manner Best Current Practice [Page 12] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + ensure that congestion indications remain timely, even the smallest + positive remainder in the conceptual counter should trigger the next + outgoing packet to be marked (causing the counter to go negative). + +3. Motivating Arguments + + This section is informative. It justifies the recommendations made + in the previous section. + +3.1. Avoiding Perverse Incentives to (Ab)use Smaller Packets + + Increasingly, it is being recognised that a protocol design must take + care not to cause unintended consequences by giving the parties in + the protocol exchange perverse incentives [Evol_cc] [RFC3426]. Given + there are many good reasons why larger path maximum transmission + units (PMTUs) would help solve a number of scaling issues, we do not + want to create any bias against large packets that is greater than + their true cost. + + Imagine a scenario where the same bit rate of packets will contribute + the same to bit congestion of a link irrespective of whether it is + sent as fewer larger packets or more smaller packets. A protocol + design that caused larger packets to be more likely to be dropped + than smaller ones would be dangerous in both of the following cases: + + Malicious transports: A queue that gives an advantage to small + packets can be used to amplify the force of a flooding attack. By + sending a flood of small packets, the attacker can get the queue + to discard more large-packet traffic, allowing more attack traffic + to get through to cause further damage. Such a queue allows + attack traffic to have a disproportionately large effect on + regular traffic without the attacker having to do much work. + + Non-malicious transports: Even if an application designer is not + actually malicious, if over time it is noticed that small packets + tend to go faster, designers will act in their own interest and + use smaller packets. Queues that give advantage to small packets + create an evolutionary pressure for applications or transports to + send at the same bit rate but break their data stream down into + tiny segments to reduce their drop rate. Encouraging a high + volume of tiny packets might in turn unnecessarily overload a + completely unrelated part of the system, perhaps more limited by + header processing than bandwidth. + + Imagine that two unresponsive flows arrive at a bit-congestible + transmission link each with the same bit rate, say 1 Mbps, but one + consists of 1,500 B and the other 60 B packets, which are 25x + smaller. Consider a scenario where gentle RED [gentle_RED] is used, + + + +Briscoe & Manner Best Current Practice [Page 13] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + along with the variant of RED we advise against, i.e., where the RED + algorithm is configured to adjust the drop probability of packets in + proportion to each packet's size (byte-mode packet drop). In this + case, RED aims to drop 25x more of the larger packets than the + smaller ones. Thus, for example, if RED drops 25% of the larger + packets, it will aim to drop 1% of the smaller packets (but, in + practice, it may drop more as congestion increases; see Appendix B.4 + of [RFC4828]). Even though both flows arrive with the same bit rate, + the bit rate the RED queue aims to pass to the line will be 750 kbps + for the flow of larger packets but 990 kbps for the smaller packets + (because of rate variations, it will actually be a little less than + this target). + + Note that, although the byte-mode drop variant of RED amplifies + small-packet attacks, tail-drop queues amplify small-packet attacks + even more (see Security Considerations in Section 6). Wherever + possible, neither should be used. + +3.2. Small != Control + + Dropping fewer control packets considerably improves performance. It + is tempting to drop small packets with lower probability in order to + improve performance, because many control packets tend to be smaller + (TCP SYNs and ACKs, DNS queries and responses, SIP messages, HTTP + GETs, etc). However, we must not give control packets preference + purely by virtue of their smallness, otherwise it is too easy for any + data source to get the same preferential treatment simply by sending + data in smaller packets. Again, we should not create perverse + incentives to favour small packets rather than to favour control + packets, which is what we intend. + + Just because many control packets are small does not mean all small + packets are control packets. + + So, rather than fix these problems in the network, we argue that the + transport should be made more robust against losses of control + packets (see Section 4.2.3). + +3.3. Transport-Independent Network + + TCP congestion control ensures that flows competing for the same + resource each maintain the same number of segments in flight, + irrespective of segment size. So under similar conditions, flows + with different segment sizes will get different bit rates. + + To counter this effect, it seems tempting not to follow our + recommendation, and instead for the network to bias congestion + notification by packet size in order to equalise the bit rates of + + + +Briscoe & Manner Best Current Practice [Page 14] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + flows with different packet sizes. However, in order to do this, the + queuing algorithm has to make assumptions about the transport, which + become embedded in the network. Specifically: + + o The queuing algorithm has to assume how aggressively the transport + will respond to congestion (see Section 4.2.4). If the network + assumes the transport responds as aggressively as TCP NewReno, it + will be wrong for Compound TCP and differently wrong for Cubic + TCP, etc. To achieve equal bit rates, each transport then has to + guess what assumption the network made, and work out how to + replace this assumed aggressiveness with its own aggressiveness. + + o Also, if the network biases congestion notification by packet + size, it has to assume a baseline packet size -- all proposed + algorithms use the local MTU (for example, see the byte-mode loss + probability formula in Table 1). Then if the non-Reno transports + mentioned above are trying to reverse engineer what the network + assumed, they also have to guess the MTU of the congested link. + + Even though reducing the drop probability of small packets (e.g., + RED's byte-mode drop) helps ensure TCP flows with different packet + sizes will achieve similar bit rates, we argue that this correction + should be made to any future transport protocols based on TCP, not to + the network in order to fix one transport, no matter how predominant + it is. Effectively, favouring small packets is reverse engineering + of network equipment around one particular transport protocol (TCP), + contrary to the excellent advice in [RFC3426], which asks designers + to question "Why are you proposing a solution at this layer of the + protocol stack, rather than at another layer?" + + In contrast, if the network never takes packet size into account, the + transport can be certain it will never need to guess any assumptions + that the network has made. And the network passes two pieces of + information to the transport that are sufficient in all cases: i) + congestion notification on the packet and ii) the size of the packet. + Both are available for the transport to combine (by taking packet + size into account when responding to congestion) or not. Appendix B + checks that these two pieces of information are sufficient for all + relevant scenarios. + + When the network does not take packet size into account, it allows + transport protocols to choose whether or not to take packet size into + account. However, if the network were to bias congestion + notification by packet size, transport protocols would have no + choice; those that did not take into account packet size themselves + would unwittingly become dependent on packet size, and those that + already took packet size into account would end up taking it into + account twice. + + + +Briscoe & Manner Best Current Practice [Page 15] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + +3.4. Partial Deployment of AQM + + In overview, the argument in this section runs as follows: + + o Because the network does not and cannot always drop packets in + proportion to their size, it shouldn't be given the task of making + drop signals depend on packet size at all. + + o Transports on the other hand don't always want to make their rate + response proportional to the size of dropped packets, but if they + want to, they always can. + + The argument is similar to the end-to-end argument that says "Don't + do X in the network if end systems can do X by themselves, and they + want to be able to choose whether to do X anyway". Actually the + following argument is stronger; in addition it says "Don't give the + network task X that could be done by the end systems, if X is not + deployed on all network nodes, and end systems won't be able to tell + whether their network is doing X, or whether they need to do X + themselves." In this case, the X in question is "making the response + to congestion depend on packet size". + + We will now re-run this argument reviewing each step in more depth. + The argument applies solely to drop, not to ECN marking. + + A queue drops packets for either of two reasons: a) to signal to host + congestion controls that they should reduce the load and b) because + there is no buffer left to store the packets. Active queue + management tries to use drops as a signal for hosts to slow down + (case a) so that drops due to buffer exhaustion (case b) should not + be necessary. + + AQM is not universally deployed in every queue in the Internet; many + cheap Ethernet bridges, software firewalls, NATs on consumer devices, + etc implement simple tail-drop buffers. Even if AQM were universal, + it has to be able to cope with buffer exhaustion (by switching to a + behaviour like tail drop), in order to cope with unresponsive or + excessive transports. For these reasons networks will sometimes be + dropping packets as a last resort (case b) rather than under AQM + control (case a). + + When buffers are exhausted (case b), they don't naturally drop + packets in proportion to their size. The network can only reduce the + probability of dropping smaller packets if it has enough space to + store them somewhere while it waits for a larger packet that it can + drop. If the buffer is exhausted, it does not have this choice. + Admittedly tail drop does naturally drop somewhat fewer small + packets, but exactly how few depends more on the mix of sizes than + + + +Briscoe & Manner Best Current Practice [Page 16] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + the size of the packet in question. Nonetheless, in general, if we + wanted networks to do size-dependent drop, we would need universal + deployment of (packet-size dependent) AQM code, which is currently + unrealistic. + + A host transport cannot know whether any particular drop was a + deliberate signal from an AQM or a sign of a queue shedding packets + due to buffer exhaustion. Therefore, because the network cannot + universally do size-dependent drop, it should not do it all. + + Whereas universality is desirable in the network, diversity is + desirable between different transport-layer protocols -- some, like + standards track TCP congestion control [RFC5681], may not choose to + make their rate response proportionate to the size of each dropped + packet, while others will (e.g., TCP-Friendly Rate Control for Small + Packets (TFRC-SP) [RFC4828]). + +3.5. Implementation Efficiency + + Biasing against large packets typically requires an extra multiply + and divide in the network (see the example byte-mode drop formula in + Table 1). Taking packet size into account at the transport rather + than in the network ensures that neither the network nor the + transport needs to do a multiply operation -- multiplication by + packet size is effectively achieved as a repeated add when the + transport adds to its count of marked bytes as each congestion event + is fed to it. Also, the work to do the biasing is spread over many + hosts, rather than concentrated in just the congested network + element. These aren't principled reasons in themselves, but they are + a happy consequence of the other principled reasons. + +4. A Survey and Critique of Past Advice + + This section is informative, not normative. + + The original 1993 paper on RED [RED93] proposed two options for the + RED active queue management algorithm: packet mode and byte mode. + Packet mode measured the queue length in packets and dropped (or + marked) individual packets with a probability independent of their + size. Byte mode measured the queue length in bytes and marked an + individual packet with probability in proportion to its size + (relative to the maximum packet size). In the paper's outline of + further work, it was stated that no recommendation had been made on + whether the queue size should be measured in bytes or packets, but + noted that the difference could be significant. + + + + + + +Briscoe & Manner Best Current Practice [Page 17] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + When RED was recommended for general deployment in 1998 [RFC2309], + the two modes were mentioned implying the choice between them was a + question of performance, referring to a 1997 email [pktByteEmail] for + advice on tuning. A later addendum to this email introduced the + insight that there are in fact two orthogonal choices: + + o whether to measure queue length in bytes or packets (Section 4.1), + and + + o whether the drop probability of an individual packet should depend + on its own size (Section 4.2). + + The rest of this section is structured accordingly. + +4.1. Congestion Measurement Advice + + The choice of which metric to use to measure queue length was left + open in RFC 2309. It is now well understood that queues for bit- + congestible resources should be measured in bytes, and queues for + packet-congestible resources should be measured in packets + [pktByteEmail]. + + Congestion in some legacy bit-congestible buffers is only measured in + packets not bytes. In such cases, the operator has to take into + account a typical mix of packet sizes when setting the thresholds. + Any AQM algorithm on such a buffer will be oversensitive to high + proportions of small packets, e.g., a DoS attack, and under-sensitive + to high proportions of large packets. However, there is no need to + make allowances for the possibility of such a legacy in future + protocol design. This is safe because any under-sensitivity during + unusual traffic mixes cannot lead to congestion collapse given that + the buffer will eventually revert to tail drop, which discards + proportionately more large packets. + +4.1.1. Fixed-Size Packet Buffers + + The question of whether to measure queues in bytes or packets seems + to be well understood. However, measuring congestion is confusing + when the resource is bit-congestible but the queue into the resource + is packet-congestible. This section outlines the approach to take. + + Some, mostly older, queuing hardware allocates fixed-size buffers in + which to store each packet in the queue. This hardware forwards + packets to the line in one of two ways: + + o With some hardware, any fixed-size buffers not completely filled + by a packet are padded when transmitted to the wire. This case + should clearly be treated as packet-congestible, because both + + + +Briscoe & Manner Best Current Practice [Page 18] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + queuing and transmission are in fixed MTU-size units. Therefore, + the queue length in packets is a good model of congestion of the + link. + + o More commonly, hardware with fixed-size packet buffers transmits + packets to the line without padding. This implies a hybrid + forwarding system with transmission congestion dependent on the + size of packets but queue congestion dependent on the number of + packets, irrespective of their size. + + Nonetheless, there would be no queue at all unless the line had + become congested -- the root cause of any congestion is too many + bytes arriving for the line. Therefore, the AQM should measure + the queue length as the sum of all the packet sizes in bytes that + are queued up waiting to be serviced by the line, irrespective of + whether each packet is held in a fixed-size buffer. + + In the (unlikely) first case where use of padding means the queue + should be measured in packets, further confusion is likely because + the fixed buffers are rarely all one size. Typically, pools of + different-sized buffers are provided (Cisco uses the term 'buffer + carving' for the process of dividing up memory into these pools + [IOSArch]). Usually, if the pool of small buffers is exhausted, + arriving small packets can borrow space in the pool of large buffers, + but not vice versa. However, there is no need to consider all this + complexity, because the root cause of any congestion is still line + overload -- buffer consumption is only the symptom. Therefore, the + length of the queue should be measured as the sum of the bytes in the + queue that will be transmitted to the line, including any padding. + In the (unusual) case of transmission with padding, this means the + sum of the sizes of the small buffers queued plus the sum of the + sizes of the large buffers queued. + + We will return to borrowing of fixed-size buffers when we discuss + biasing the drop/marking probability of a specific packet because of + its size in Section 4.2.1. But here, we can repeat the simple rule + for how to measure the length of queues of fixed buffers: no matter + how complicated the buffering scheme is, ultimately a transmission + line is nearly always bit-congestible so the number of bytes queued + up waiting for the line measures how congested the line is, and it is + rarely important to measure how congested the buffering system is. + +4.1.2. Congestion Measurement without a Queue + + AQM algorithms are nearly always described assuming there is a queue + for a congested resource and the algorithm can use the queue length + to determine the probability that it will drop or mark each packet. + But not all congested resources lead to queues. For instance, power- + + + +Briscoe & Manner Best Current Practice [Page 19] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + limited resources are usually bit-congestible if energy is primarily + required for transmission rather than header processing, but it is + rare for a link protocol to build a queue as it approaches maximum + power. + + Nonetheless, AQM algorithms do not require a queue in order to work. + For instance, spectrum congestion can be modelled by signal quality + using the target bit-energy-to-noise-density ratio. And, to model + radio power exhaustion, transmission-power levels can be measured and + compared to the maximum power available. [ECNFixedWireless] proposes + a practical and theoretically sound way to combine congestion + notification for different bit-congestible resources at different + layers along an end-to-end path, whether wireless or wired, and + whether with or without queues. + + In wireless protocols that use request to send / clear to send + (RTS / CTS) control, such as some variants of IEEE802.11, it is + reasonable to base an AQM on the time spent waiting for transmission + opportunities (TXOPs) even though the wireless spectrum is usually + regarded as congested by bits (for a given coding scheme). This is + because requests for TXOPs queue up as the spectrum gets congested by + all the bits being transferred. So the time that TXOPs are queued + directly reflects bit congestion of the spectrum. + +4.2. Congestion Notification Advice + +4.2.1. Network Bias When Encoding + +4.2.1.1. Advice on Packet-Size Bias in RED + + The previously mentioned email [pktByteEmail] referred to by + [RFC2309] advised that most scarce resources in the Internet were + bit-congestible, which is still believed to be true (Section 1.1). + But it went on to offer advice that is updated by this memo. It said + that drop probability should depend on the size of the packet being + considered for drop if the resource is bit-congestible, but not if it + is packet-congestible. The argument continued that if packet drops + were inflated by packet size (byte-mode dropping), "a flow's fraction + of the packet drops is then a good indication of that flow's fraction + of the link bandwidth in bits per second". This was consistent with + a referenced policing mechanism being worked on at the time for + detecting unusually high bandwidth flows, eventually published in + 1999 [pBox]. However, the problem could and should have been solved + by making the policing mechanism count the volume of bytes randomly + dropped, not the number of packets. + + + + + + +Briscoe & Manner Best Current Practice [Page 20] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + A few months before RFC 2309 was published, an addendum was added to + the above archived email referenced from the RFC, in which the final + paragraph seemed to partially retract what had previously been said. + It clarified that the question of whether the probability of + dropping/marking a packet should depend on its size was not related + to whether the resource itself was bit-congestible, but a completely + orthogonal question. However, the only example given had the queue + measured in packets but packet drop depended on the size of the + packet in question. No example was given the other way round. + + In 2000, Cnodder et al. [REDbyte] pointed out that there was an error + in the part of the original 1993 RED algorithm that aimed to + distribute drops uniformly, because it didn't correctly take into + account the adjustment for packet size. They recommended an + algorithm called RED_4 to fix this. But they also recommended a + further change, RED_5, to adjust the drop rate dependent on the + square of the relative packet size. This was indeed consistent with + one implied motivation behind RED's byte-mode drop -- that we should + reverse engineer the network to improve the performance of dominant + end-to-end congestion control mechanisms. This memo makes a + different recommendations in Section 2. + + By 2003, a further change had been made to the adjustment for packet + size, this time in the RED algorithm of the ns2 simulator. Instead + of taking each packet's size relative to a 'maximum packet size', it + was taken relative to a 'mean packet size', intended to be a static + value representative of the 'typical' packet size on the link. We + have not been able to find a justification in the literature for this + change; however, Eddy and Allman conducted experiments [REDbias] that + assessed how sensitive RED was to this parameter, amongst other + things. This changed algorithm can often lead to drop probabilities + of greater than 1 (which gives a hint that there is probably a + mistake in the theory somewhere). + + On 10-Nov-2004, this variant of byte-mode packet drop was made the + default in the ns2 simulator. It seems unlikely that byte-mode drop + has ever been implemented in production networks (Appendix A); + therefore, any conclusions based on ns2 simulations that use RED + without disabling byte-mode drop are likely to behave very + differently from RED in production networks. + +4.2.1.2. Packet-Size Bias Regardless of AQM + + The byte-mode drop variant of RED (or a similar variant of other AQM + algorithms) is not the only possible bias towards small packets in + queuing systems. We have already mentioned that tail-drop queues + naturally tend to lock out large packets once they are full. + + + + +Briscoe & Manner Best Current Practice [Page 21] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + But also, queues with fixed-size buffers reduce the probability that + small packets will be dropped if (and only if) they allow small + packets to borrow buffers from the pools for larger packets (see + Section 4.1.1). Borrowing effectively makes the maximum queue size + for small packets greater than that for large packets, because more + buffers can be used by small packets while less will fit large + packets. Incidentally, the bias towards small packets from buffer + borrowing is nothing like as large as that of RED's byte-mode drop. + + Nonetheless, fixed-buffer memory with tail drop is still prone to + lock out large packets, purely because of the tail-drop aspect. So, + fixed-size packet buffers should be augmented with a good AQM + algorithm and packet-mode drop. If an AQM is too complicated to + implement with multiple fixed buffer pools, the minimum necessary to + prevent large-packet lockout is to ensure that smaller packets never + use the last available buffer in any of the pools for larger packets. + +4.2.2. Transport Bias When Decoding + + The above proposals to alter the network equipment to bias towards + smaller packets have largely carried on outside the IETF process. + Whereas, within the IETF, there are many different proposals to alter + transport protocols to achieve the same goals, i.e., either to make + the flow bit rate take into account packet size, or to protect + control packets from loss. This memo argues that altering transport + protocols is the more principled approach. + + A recently approved experimental RFC adapts its transport-layer + protocol to take into account packet sizes relative to typical TCP + packet sizes. This proposes a new small-packet variant of TCP- + friendly rate control (TFRC [RFC5348]), which is called TFRC-SP + [RFC4828]. Essentially, it proposes a rate equation that inflates + the flow rate by the ratio of a typical TCP segment size (1,500 B + including TCP header) over the actual segment size [PktSizeEquCC]. + (There are also other important differences of detail relative to + TFRC, such as using virtual packets [CCvarPktSize] to avoid + responding to multiple losses per round trip and using a minimum + inter-packet interval.) + + Section 4.5.1 of the TFRC-SP specification discusses the implications + of operating in an environment where queues have been configured to + drop smaller packets with proportionately lower probability than + larger ones. But it only discusses TCP operating in such an + environment, only mentioning TFRC-SP briefly when discussing how to + define fairness with TCP. And it only discusses the byte-mode + dropping version of RED as it was before Cnodder et al. pointed out + that it didn't sufficiently bias towards small packets to make TCP + independent of packet size. + + + +Briscoe & Manner Best Current Practice [Page 22] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + So the TFRC-SP specification doesn't address the issue of whether the + network or the transport _should_ handle fairness between different + packet sizes. In Appendix B.4 of RFC 4828, it discusses the + possibility of both TFRC-SP and some network buffers duplicating each + other's attempts to deliberately bias towards small packets. But the + discussion is not conclusive, instead reporting simulations of many + of the possibilities in order to assess performance but not + recommending any particular course of action. + + The paper originally proposing TFRC with virtual packets (VP-TFRC) + [CCvarPktSize] proposed that there should perhaps be two variants to + cater for the different variants of RED. However, as the TFRC-SP + authors point out, there is no way for a transport to know whether + some queues on its path have deployed RED with byte-mode packet drop + (except if an exhaustive survey found that no one has deployed it! -- + see Appendix A). Incidentally, VP-TFRC also proposed that byte-mode + RED dropping should really square the packet-size compensation factor + (like that of Cnodder's RED_5, but apparently unaware of it). + + Pre-congestion notification [RFC5670] is an IETF technology to use a + virtual queue for AQM marking for packets within one Diffserv class + in order to give early warning prior to any real queuing. The PCN- + marking algorithms have been designed not to take into account packet + size when forwarding through queues. Instead, the general principle + has been to take the sizes of marked packets into account when + monitoring the fraction of marking at the edge of the network, as + recommended here. + +4.2.3. Making Transports Robust against Control Packet Losses + + Recently, two RFCs have defined changes to TCP that make it more + robust against losing small control packets [RFC5562] [RFC5690]. In + both cases, they note that the case for these two TCP changes would + be weaker if RED were biased against dropping small packets. We + argue here that these two proposals are a safer and more principled + way to achieve TCP performance improvements than reverse engineering + RED to benefit TCP. + + Although there are no known proposals, it would also be possible and + perfectly valid to make control packets robust against drop by + requesting a scheduling class with lower drop probability, which + would be achieved by re-marking to a Diffserv code point [RFC2474] + within the same behaviour aggregate. + + Although not brought to the IETF, a simple proposal from Wischik + [DupTCP] suggests that the first three packets of every TCP flow + should be routinely duplicated after a short delay. It shows that + this would greatly improve the chances of short flows completing + + + +Briscoe & Manner Best Current Practice [Page 23] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + quickly, but it would hardly increase traffic levels on the Internet, + because Internet bytes have always been concentrated in the large + flows. It further shows that the performance of many typical + applications depends on completion of long serial chains of short + messages. It argues that, given most of the value people get from + the Internet is concentrated within short flows, this simple + expedient would greatly increase the value of the best-effort + Internet at minimal cost. A similar but more extensive approach has + been evaluated on Google servers [GentleAggro]. + + The proposals discussed in this sub-section are experimental + approaches that are not yet in wide operational use, but they are + existence proofs that transports can make themselves robust against + loss of control packets. The examples are all TCP-based, but + applications over non-TCP transports could mitigate loss of control + packets by making similar use of Diffserv, data duplication, FEC, + etc. + +4.2.4. Congestion Notification: Summary of Conflicting Advice + + +-----------+-----------------+-----------------+-------------------+ + | transport | RED_1 (packet- | RED_4 (linear | RED_5 (square | + | cc | mode drop) | byte-mode drop) | byte-mode drop) | + +-----------+-----------------+-----------------+-------------------+ + | TCP or | s/sqrt(p) | sqrt(s/p) | 1/sqrt(p) | + | TFRC | | | | + | TFRC-SP | 1/sqrt(p) | 1/sqrt(s*p) | 1/(s*sqrt(p)) | + +-----------+-----------------+-----------------+-------------------+ + + Table 2: Dependence of flow bit rate per RTT on packet size, s, and + drop probability, p, when there is network and/or transport bias + towards small packets to varying degrees + + Table 2 aims to summarise the potential effects of all the advice + from different sources. Each column shows a different possible AQM + behaviour in different queues in the network, using the terminology + of Cnodder et al. outlined earlier (RED_1 is basic RED with packet- + mode drop). Each row shows a different transport behaviour: TCP + [RFC5681] and TFRC [RFC5348] on the top row with TFRC-SP [RFC4828] + below. Each cell shows how the bits per round trip of a flow depends + on packet size, s, and drop probability, p. In order to declutter + the formulae to focus on packet-size dependence, they are all given + per round trip, which removes any RTT term. + + Let us assume that the goal is for the bit rate of a flow to be + independent of packet size. Suppressing all inessential details, the + table shows that this should either be achievable by not altering the + TCP transport in a RED_5 network, or using the small packet TFRC-SP + + + +Briscoe & Manner Best Current Practice [Page 24] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + transport (or similar) in a network without any byte-mode dropping + RED (top right and bottom left). Top left is the 'do nothing' + scenario, while bottom right is the 'do both' scenario in which the + bit rate would become far too biased towards small packets. Of + course, if any form of byte-mode dropping RED has been deployed on a + subset of queues that congest, each path through the network will + present a different hybrid scenario to its transport. + + Whatever the case, we can see that the linear byte-mode drop column + in the middle would considerably complicate the Internet. Even if + one believes the network should be doing the biasing, linear byte- + mode drop is a half-way house that doesn't bias enough towards small + packets. Section 2 recommends that _all_ bias in network equipment + towards small packets should be turned off -- if indeed any equipment + vendors have implemented it -- leaving packet-size bias solely as the + preserve of the transport layer (solely the leftmost, packet-mode + drop column). + + In practice, it seems that no deliberate bias towards small packets + has been implemented for production networks. Of the 19% of vendors + who responded to a survey of 84 equipment vendors, none had + implemented byte-mode drop in RED (see Appendix A for details). + +5. Outstanding Issues and Next Steps + +5.1. Bit-congestible Network + + For a connectionless network with nearly all resources being bit- + congestible, the recommended position is clear -- the network should + not make allowance for packet sizes and the transport should. This + leaves two outstanding issues: + + o The question of how to handle any legacy AQM deployments using + byte-mode drop; + + o The need to start a programme to update transport congestion + control protocol standards to take packet size into account. + + A survey of equipment vendors (Section 4.2.4) found no evidence that + byte-mode packet drop had been implemented, so deployment will be + sparse at best. A migration strategy is not really needed to remove + an algorithm that may not even be deployed. + + A programme of experimental updates to take packet size into account + in transport congestion control protocols has already started with + TFRC-SP [RFC4828]. + + + + + +Briscoe & Manner Best Current Practice [Page 25] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + +5.2. Bit- and Packet-Congestible Network + + The position is much less clear-cut if the Internet becomes populated + by a more even mix of both packet-congestible and bit-congestible + resources (see Appendix B.2). This problem is not pressing, because + most Internet resources are designed to be bit-congestible before + packet processing starts to congest (see Section 1.1). + + The IRTF's Internet Congestion Control Research Group (ICCRG) has set + itself the task of reaching consensus on generic forwarding + mechanisms that are necessary and sufficient to support the + Internet's future congestion control requirements (the first + challenge in [RFC6077]). The research question of whether packet + congestion might become common and what to do if it does may in the + future be explored in the IRTF (the "Challenge 3: Packet Size" in + [RFC6077]). + + Note that sometimes it seems that resources might be congested by + neither bits nor packets, e.g., where the queue for access to a + wireless medium is in units of transmission opportunities. However, + the root cause of congestion of the underlying spectrum is overload + of bits (see Section 4.1.2). + +6. Security Considerations + + This memo recommends that queues do not bias drop probability due to + packets size. For instance, dropping small packets less often than + large ones creates a perverse incentive for transports to break down + their flows into tiny segments. One of the benefits of implementing + AQM was meant to be to remove this perverse incentive that tail-drop + queues gave to small packets. + + In practice, transports cannot all be trusted to respond to + congestion. So another reason for recommending that queues not bias + drop probability towards small packets is to avoid the vulnerability + to small-packet DDoS attacks that would otherwise result. One of the + benefits of implementing AQM was meant to be to remove tail drop's + DoS vulnerability to small packets, so we shouldn't add it back + again. + + If most queues implemented AQM with byte-mode drop, the resulting + network would amplify the potency of a small-packet DDoS attack. At + the first queue, the stream of packets would push aside a greater + proportion of large packets, so more of the small packets would + survive to attack the next queue. Thus a flood of small packets + would continue on towards the destination, pushing regular traffic + with large packets out of the way in one queue after the next, but + suffering much less drop itself. + + + +Briscoe & Manner Best Current Practice [Page 26] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + Appendix C explains why the ability of networks to police the + response of _any_ transport to congestion depends on bit-congestible + network resources only doing packet-mode drop, not byte-mode drop. + In summary, it says that making drop probability depend on the size + of the packets that bits happen to be divided into simply encourages + the bits to be divided into smaller packets. Byte-mode drop would + therefore irreversibly complicate any attempt to fix the Internet's + incentive structures. + +7. Conclusions + + This memo identifies the three distinct stages of the congestion + notification process where implementations need to decide whether to + take packet size into account. The recommendations provided in + Section 2 of this memo are different in each case: + + o When network equipment measures the length of a queue, if it is + not feasible to use time; it is recommended to count in bytes if + the network resource is congested by bytes, or to count in packets + if is congested by packets. + + o When network equipment decides whether to drop (or mark) a packet, + it is recommended that the size of the particular packet should + not be taken into account. + + o However, when a transport algorithm responds to a dropped or + marked packet, the size of the rate reduction should be + proportionate to the size of the packet. + + In summary, the answers are 'it depends', 'no', and 'yes', + respectively. + + For the specific case of RED, this means that byte-mode queue + measurement will often be appropriate, but the use of byte-mode drop + is very strongly discouraged. + + At the transport layer, the IETF should continue updating congestion + control protocols to take into account the size of each packet that + indicates congestion. Also, the IETF should continue to make + protocols less sensitive to losing control packets like SYNs, pure + ACKs, and DNS exchanges. Although many control packets happen to be + small, the alternative of network equipment favouring all small + packets would be dangerous. That would create perverse incentives to + split data transfers into smaller packets. + + The memo develops these recommendations from principled arguments + concerning scaling, layering, incentives, inherent efficiency, + security, and 'policeability'. It also addresses practical issues + + + +Briscoe & Manner Best Current Practice [Page 27] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + such as specific buffer architectures and incremental deployment. + Indeed, a limited survey of RED implementations is discussed, which + shows there appears to be little, if any, installed base of RED's + byte-mode drop. Therefore, it can be deprecated with little, if any, + incremental deployment complications. + + The recommendations have been developed on the well-founded basis + that most Internet resources are bit-congestible, not packet- + congestible. We need to know the likelihood that this assumption + will prevail in the longer term and, if it might not, what protocol + changes will be needed to cater for a mix of the two. The IRTF + Internet Congestion Control Research Group (ICCRG) is currently + working on these problems [RFC6077]. + +8. Acknowledgements + + Thank you to Sally Floyd, who gave extensive and useful review + comments. Also thanks for the reviews from Philip Eardley, David + Black, Fred Baker, David Taht, Toby Moncaster, Arnaud Jacquet, and + Mirja Kuehlewind, as well as helpful explanations of different + hardware approaches from Larry Dunn and Fred Baker. We are grateful + to Bruce Davie and his colleagues for providing a timely and + efficient survey of RED implementation in Cisco's product range. + Also, grateful thanks to Toby Moncaster, Will Dormann, John Regnault, + Simon Carter, and Stefaan De Cnodder who further helped survey the + current status of RED implementation and deployment, and, finally, + thanks to the anonymous individuals who responded. + + Bob Briscoe and Jukka Manner were partly funded by Trilogy and + Trilogy 2, research projects (ICT-216372, ICT-317756) supported by + the European Community under its Seventh Framework Programme. The + views expressed here are those of the authors only. + +9. References + +9.1. Normative References + + [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate + Requirement Levels", BCP 14, RFC 2119, March 1997. + + [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, + S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., + Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, + S., Wroclawski, J., and L. Zhang, "Recommendations on + Queue Management and Congestion Avoidance in the + Internet", RFC 2309, April 1998. + + + + + +Briscoe & Manner Best Current Practice [Page 28] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + [RFC2914] Floyd, S., "Congestion Control Principles", BCP 41, RFC + 2914, September 2000. + + [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition + of Explicit Congestion Notification (ECN) to IP", RFC + 3168, September 2001. + +9.2. Informative References + + [BLUE02] Feng, W-c., Shin, K., Kandlur, D., and D. Saha, "The BLUE + active queue management algorithms", IEEE/ACM Transactions + on Networking 10(4) 513-528, August 2002, + <http://dx.doi.org/10.1109/TNET.2002.801399>. + + [CCvarPktSize] + Widmer, J., Boutremans, C., and J-Y. Le Boudec, "End-to- + end congestion control for TCP-friendly flows with + variable packet size", ACM CCR 34(2) 137-151, April 2004, + <http://doi.acm.org/10.1145/997150.997162>. + + [CHOKe_Var_Pkt] + Psounis, K., Pan, R., and B. Prabhaker, "Approximate Fair + Dropping for Variable-Length Packets", IEEE Micro + 21(1):48-56, January-February 2001, + <http://ieeexplore.ieee.org/xpl/ + articleDetails.jsp?arnumber=903061>. + + [CoDel] Nichols, K. and V. Jacobson, "Controlled Delay Active + Queue Management", Work in Progress, February 2013. + + [DRQ] Shin, M., Chong, S., and I. Rhee, "Dual-Resource TCP/AQM + for Processing-Constrained Networks", IEEE/ACM + Transactions on Networking Vol 16, issue 2, April 2008, + <http://dx.doi.org/10.1109/TNET.2007.900415>. + + [DupTCP] Wischik, D., "Short messages", Philosophical Transactions + of the Royal Society A 366(1872):1941-1953, June 2008, + <http://rsta.royalsocietypublishing.org/content/366/1872/ + 1941.full.pdf+html>. + + [ECNFixedWireless] + Siris, V., "Resource Control for Elastic Traffic in CDMA + Networks", Proc. ACM MOBICOM'02 , September 2002, + <http://www.ics.forth.gr/netlab/publications/ + resource_control_elastic_cdma.html>. + + + + + + +Briscoe & Manner Best Current Practice [Page 29] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + [Evol_cc] Gibbens, R. and F. Kelly, "Resource pricing and the + evolution of congestion control", Automatica + 35(12)1969-1985, December 1999, + <http://www.sciencedirect.com/science/article/pii/ + S0005109899001351>. + + [GentleAggro] + Flach, T., Dukkipati, N., Terzis, A., Raghavan, B., + Cardwell, N., Cheng, Y., Jain, A., Hao, S., Katz-Bassett, + E., and R. Govindan, "Reducing web latency: the virtue of + gentle aggression", ACM SIGCOMM CCR 43(4)159-170, August + 2013, <http://doi.acm.org/10.1145/2486001.2486014>. + + [IOSArch] Bollapragada, V., White, R., and C. Murphy, "Inside Cisco + IOS Software Architecture", Cisco Press: CCIE Professional + Development ISBN13: 978-1-57870-181-0, July 2000. + + [PIE] Pan, R., Natarajan, P., Piglione, C., Prabhu, M., + Subramanian, V., Baker, F., and B. Steeg, "PIE: A + Lightweight Control Scheme To Address the Bufferbloat + Problem", Work in Progress, February 2014. + + [PktSizeEquCC] + Vasallo, P., "Variable Packet Size Equation-Based + Congestion Control", ICSI Technical Report tr-00-008, + 2000, <http://http.icsi.berkeley.edu/ftp/global/pub/ + techreports/2000/tr-00-008.pdf>. + + [RED93] Floyd, S. and V. Jacobson, "Random Early Detection (RED) + gateways for Congestion Avoidance", IEEE/ACM Transactions + on Networking 1(4) 397--413, August 1993, + <http://ieeexplore.ieee.org/xpls/ + abs_all.jsp?arnumber=251892>. + + [REDbias] Eddy, W. and M. Allman, "A Comparison of RED's Byte and + Packet Modes", Computer Networks 42(3) 261--280, June + 2003, + <http://www.ir.bbn.com/documents/articles/redbias.ps>. + + [REDbyte] De Cnodder, S., Elloumi, O., and K. Pauwels, "Effect of + different packet sizes on RED performance", Proc. 5th IEEE + Symposium on Computers and Communications (ISCC) 793-799, + July 2000, <http://ieeexplore.ieee.org/xpls/ + abs_all.jsp?arnumber=860741>. + + + + + + + +Briscoe & Manner Best Current Practice [Page 30] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, + "Definition of the Differentiated Services Field (DS + Field) in the IPv4 and IPv6 Headers", RFC 2474, December + 1998. + + [RFC3426] Floyd, S., "General Architectural and Policy + Considerations", RFC 3426, November 2002. + + [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. + Jacobson, "RTP: A Transport Protocol for Real-Time + Applications", STD 64, RFC 3550, July 2003. + + [RFC3714] Floyd, S. and J. Kempf, "IAB Concerns Regarding Congestion + Control for Voice Traffic in the Internet", RFC 3714, + March 2004. + + [RFC4828] Floyd, S. and E. Kohler, "TCP Friendly Rate Control + (TFRC): The Small-Packet (SP) Variant", RFC 4828, April + 2007. + + [RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP + Friendly Rate Control (TFRC): Protocol Specification", RFC + 5348, September 2008. + + [RFC5562] Kuzmanovic, A., Mondal, A., Floyd, S., and K. + Ramakrishnan, "Adding Explicit Congestion Notification + (ECN) Capability to TCP's SYN/ACK Packets", RFC 5562, June + 2009. + + [RFC5670] Eardley, P., "Metering and Marking Behaviour of PCN- + Nodes", RFC 5670, November 2009. + + [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion + Control", RFC 5681, September 2009. + + [RFC5690] Floyd, S., Arcia, A., Ros, D., and J. Iyengar, "Adding + Acknowledgement Congestion Control to TCP", RFC 5690, + February 2010. + + [RFC6077] Papadimitriou, D., Welzl, M., Scharf, M., and B. Briscoe, + "Open Research Issues in Internet Congestion Control", RFC + 6077, February 2011. + + [RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P., + and K. Carlberg, "Explicit Congestion Notification (ECN) + for RTP over UDP", RFC 6679, August 2012. + + + + + +Briscoe & Manner Best Current Practice [Page 31] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + [RFC6789] Briscoe, B., Woundy, R., and A. Cooper, "Congestion + Exposure (ConEx) Concepts and Use Cases", RFC 6789, + December 2012. + + [Rate_fair_Dis] + Briscoe, B., "Flow Rate Fairness: Dismantling a Religion", + ACM CCR 37(2)63-74, April 2007, + <http://portal.acm.org/citation.cfm?id=1232926>. + + [gentle_RED] + Floyd, S., "Recommendation on using the "gentle_" variant + of RED", Web page , March 2000, + <http://www.icir.org/floyd/red/gentle.html>. + + [pBox] Floyd, S. and K. Fall, "Promoting the Use of End-to-End + Congestion Control", IEEE/ACM Transactions on Networking + 7(4) 458--472, August 1999, <http://ieeexplore.ieee.org/ + xpls/abs_all.jsp?arnumber=793002>. + + [pktByteEmail] + Floyd, S., "RED: Discussions of Byte and Packet Modes", + email, March 1997, + <http://ee.lbl.gov/floyd/REDaveraging.txt>. + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Briscoe & Manner Best Current Practice [Page 32] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + +Appendix A. Survey of RED Implementation Status + + This Appendix is informative, not normative. + + In May 2007 a survey was conducted of 84 vendors to assess how widely + drop probability based on packet size has been implemented in RED + Table 3. About 19% of those surveyed replied, giving a sample size + of 16. Although in most cases we do not have permission to identify + the respondents, we can say that those that have responded include + most of the larger equipment vendors, covering a large fraction of + the market. The two who gave permission to be identified were Cisco + and Alcatel-Lucent. The others range across the large network + equipment vendors at L3 & L2, firewall vendors, wireless equipment + vendors, as well as large software businesses with a small selection + of networking products. All those who responded confirmed that they + have not implemented the variant of RED with drop dependent on packet + size (2 were fairly sure they had not but needed to check more + thoroughly). At the time the survey was conducted, Linux did not + implement RED with packet-size bias of drop, although we have not + investigated a wider range of open source code. + + +-------------------------------+----------------+--------------+ + | Response | No. of vendors | % of vendors | + +-------------------------------+----------------+--------------+ + | Not implemented | 14 | 17% | + | Not implemented (probably) | 2 | 2% | + | Implemented | 0 | 0% | + | No response | 68 | 81% | + | Total companies/orgs surveyed | 84 | 100% | + +-------------------------------+----------------+--------------+ + + Table 3: Vendor Survey on byte-mode drop variant of RED (lower drop + probability for small packets) + + Where reasons were given for why the byte-mode drop variant had not + been implemented, the extra complexity of packet-bias code was most + prevalent, though one vendor had a more principled reason for + avoiding it -- similar to the argument of this document. + + Our survey was of vendor implementations, so we cannot be certain + about operator deployment. But we believe many queues in the + Internet are still tail drop. The company of one of the co-authors + (BT) has widely deployed RED; however, many tail-drop queues are + bound to still exist, particularly in access network equipment and on + middleboxes like firewalls, where RED is not always available. + + + + + + +Briscoe & Manner Best Current Practice [Page 33] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + Routers using a memory architecture based on fixed-size buffers with + borrowing may also still be prevalent in the Internet. As explained + in Section 4.2.1, these also provide a marginal (but legitimate) bias + towards small packets. So even though RED byte-mode drop is not + prevalent, it is likely there is still some bias towards small + packets in the Internet due to tail-drop and fixed-buffer borrowing. + +Appendix B. Sufficiency of Packet-Mode Drop + + This Appendix is informative, not normative. + + Here we check that packet-mode drop (or marking) in the network gives + sufficiently generic information for the transport layer to use. We + check against a 2x2 matrix of four scenarios that may occur now or in + the future (Table 4). Checking the two scenarios in each of the + horizontal and vertical dimensions tests the extremes of sensitivity + to packet size in the transport and in the network respectively. + + Note that this section does not consider byte-mode drop at all. + Having deprecated byte-mode drop, the goal here is to check that + packet-mode drop will be sufficient in all cases. + + +-------------------------------+-----------------+-----------------+ + | Transport -> | a) Independent | b) Dependent on | + | ----------------------------- | of packet size | packet size of | + | Network | of congestion | congestion | + | | notifications | notifications | + +-------------------------------+-----------------+-----------------+ + | 1) Predominantly bit- | Scenario a1) | Scenario b1) | + | congestible network | | | + | 2) Mix of bit-congestible and | Scenario a2) | Scenario b2) | + | pkt-congestible network | | | + +-------------------------------+-----------------+-----------------+ + + Table 4: Four Possible Congestion Scenarios + + Appendix B.1 focuses on the horizontal dimension of Table 4 checking + that packet-mode drop (or marking) gives sufficient information, + whether or not the transport uses it -- scenarios b) and a) + respectively. + + Appendix B.2 focuses on the vertical dimension of Table 4, checking + that packet-mode drop gives sufficient information to the transport + whether resources in the network are bit-congestible or packet- + congestible (these terms are defined in Section 1.1). + + + + + + +Briscoe & Manner Best Current Practice [Page 34] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + Notation: To be concrete, we will compare two flows with different + packet sizes, s_1 and s_2. As an example, we will take + s_1 = 60 B = 480 b and s_2 = 1,500 B = 12,000 b. + + A flow's bit rate, x [bps], is related to its packet rate, u + [pps], by + + x(t) = s*u(t). + + In the bit-congestible case, path congestion will be denoted by + p_b, and in the packet-congestible case by p_p. When either case + is implied, the letter p alone will denote path congestion. + +B.1. Packet-Size (In)Dependence in Transports + + In all cases, we consider a packet-mode drop queue that indicates + congestion by dropping (or marking) packets with probability p + irrespective of packet size. We use an example value of loss + (marking) probability, p=0.1%. + + A transport like TCP as specified in RFC 5681 treats a congestion + notification on any packet whatever its size as one event. However, + a network with just the packet-mode drop algorithm gives more + information if the transport chooses to use it. We will use Table 5 + to illustrate this. + + We will set aside the last column until later. The columns labelled + 'Flow 1' and 'Flow 2' compare two flows consisting of 60 B and + 1,500 B packets respectively. The body of the table considers two + separate cases, one where the flows have an equal bit rate and the + other with equal packet rates. In both cases, the two flows fill a + 96 Mbps link. Therefore, in the equal bit rate case, they each have + half the bit rate (48Mbps). Whereas, with equal packet rates, Flow 1 + uses 25 times smaller packets so it gets 25 times less bit rate -- it + only gets 1/(1+25) of the link capacity (96 Mbps / 26 = 4 Mbps after + rounding). In contrast Flow 2 gets 25 times more bit rate (92 Mbps) + in the equal packet rate case because its packets are 25 times + larger. The packet rate shown for each flow could easily be derived + once the bit rate was known by dividing the bit rate by packet size, + as shown in the column labelled 'Formula'. + + + + + + + + + + + +Briscoe & Manner Best Current Practice [Page 35] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + Parameter Formula Flow 1 Flow 2 Combined + ----------------------- ----------- -------- -------- -------- + Packet size s/8 60 B 1,500 B (Mix) + Packet size s 480 b 12,000 b (Mix) + Pkt loss probability p 0.1% 0.1% 0.1% + + EQUAL BIT RATE CASE + Bit rate x 48 Mbps 48 Mbps 96 Mbps + Packet rate u = x/s 100 kpps 4 kpps 104 kpps + Absolute pkt-loss rate p*u 100 pps 4 pps 104 pps + Absolute bit-loss rate p*u*s 48 kbps 48 kbps 96 kbps + Ratio of lost/sent pkts p*u/u 0.1% 0.1% 0.1% + Ratio of lost/sent bits p*u*s/(u*s) 0.1% 0.1% 0.1% + + EQUAL PACKET RATE CASE + Bit rate x 4 Mbps 92 Mbps 96 Mbps + Packet rate u = x/s 8 kpps 8 kpps 15 kpps + Absolute pkt-loss rate p*u 8 pps 8 pps 15 pps + Absolute bit-loss rate p*u*s 4 kbps 92 kbps 96 kbps + Ratio of lost/sent pkts p*u/u 0.1% 0.1% 0.1% + Ratio of lost/sent bits p*u*s/(u*s) 0.1% 0.1% 0.1% + + Table 5: Absolute Loss Rates and Loss Ratios for Flows of Small and + Large Packets and Both Combined + + So far, we have merely set up the scenarios. We now consider + congestion notification in the scenario. Two TCP flows with the same + round-trip time aim to equalise their packet-loss rates over time; + that is, the number of packets lost in a second, which is the packets + per second (u) multiplied by the probability that each one is dropped + (p). Thus, TCP converges on the case labelled 'Equal packet rate' in + the table, where both flows aim for the same absolute packet-loss + rate (both 8 pps in the table). + + Packet-mode drop actually gives flows sufficient information to + measure their loss rate in bits per second, if they choose, not just + packets per second. Each flow can count the size of a lost or marked + packet and scale its rate response in proportion (as TFRC-SP does). + The result is shown in the row entitled 'Absolute bit-loss rate', + where the bits lost in a second is the packets per second (u) + multiplied by the probability of losing a packet (p) multiplied by + the packet size (s). Such an algorithm would try to remove any + imbalance in the bit-loss rate such as the wide disparity in the case + labelled 'Equal packet rate' (4k bps vs. 92 kbps). Instead, a + packet-size-dependent algorithm would aim for equal bit-loss rates, + which would drive both flows towards the case labelled 'Equal bit + rate', by driving them to equal bit-loss rates (both 48 kbps in this + example). + + + +Briscoe & Manner Best Current Practice [Page 36] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + The explanation so far has assumed that each flow consists of packets + of only one constant size. Nonetheless, it extends naturally to + flows with mixed packet sizes. In the right-most column of Table 5, + a flow of mixed-size packets is created simply by considering Flow 1 + and Flow 2 as a single aggregated flow. There is no need for a flow + to maintain an average packet size. It is only necessary for the + transport to scale its response to each congestion indication by the + size of each individual lost (or marked) packet. Taking, for + example, the case labelled 'Equal packet rate', in one second about 8 + small packets and 8 large packets are lost (making closer to 15 than + 16 losses per second due to rounding). If the transport multiplies + each loss by its size, in one second it responds to 8*480 and + 8*12,000 lost bits, adding up to 96,000 lost bits in a second. This + double checks correctly, being the same as 0.1% of the total bit rate + of 96 Mbps. For completeness, the formula for absolute bit-loss rate + is p(u1*s1+u2*s2). + + Incidentally, a transport will always measure the loss probability + the same, irrespective of whether it measures in packets or in bytes. + In other words, the ratio of lost packets to sent packets will be the + same as the ratio of lost bytes to sent bytes. (This is why TCP's + bit rate is still proportional to packet size, even when byte + counting is used, as recommended for TCP in [RFC5681], mainly for + orthogonal security reasons.) This is intuitively obvious by + comparing two example flows; one with 60 B packets, the other with + 1,500 B packets. If both flows pass through a queue with drop + probability 0.1%, each flow will lose 1 in 1,000 packets. In the + stream of 60 B packets, the ratio of lost bytes to sent bytes will be + 60 B in every 60,000 B; and in the stream of 1,500 B packets, the + loss ratio will be 1,500 B out of 1,500,000 B. When the transport + responds to the ratio of lost to sent packets, it will measure the + same ratio whether it measures in packets or bytes: 0.1% in both + cases. The fact that this ratio is the same whether measured in + packets or bytes can be seen in Table 5, where the ratio of lost + packets to sent packets and the ratio of lost bytes to sent bytes is + always 0.1% in all cases (recall that the scenario was set up with + p=0.1%). + + This discussion of how the ratio can be measured in packets or bytes + is only raised here to highlight that it is irrelevant to this memo! + Whether or not a transport depends on packet size depends on how this + ratio is used within the congestion control algorithm. + + So far, we have shown that packet-mode drop passes sufficient + information to the transport layer so that the transport can take bit + congestion into account, by using the sizes of the packets that + indicate congestion. We have also shown that the transport can + + + + +Briscoe & Manner Best Current Practice [Page 37] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + choose not to take packet size into account if it wishes. We will + now consider whether the transport can know which to do. + +B.2. Bit-Congestible and Packet-Congestible Indications + + As a thought-experiment, imagine an idealised congestion notification + protocol that supports both bit-congestible and packet-congestible + resources. It would require at least two ECN flags, one for each of + the bit-congestible and packet-congestible resources. + + 1. A packet-congestible resource trying to code congestion level p_p + into a packet stream should mark the idealised 'packet + congestion' field in each packet with probability p_p + irrespective of the packet's size. The transport should then + take a packet with the packet congestion field marked to mean + just one mark, irrespective of the packet size. + + 2. A bit-congestible resource trying to code time-varying byte- + congestion level p_b into a packet stream should mark the 'byte + congestion' field in each packet with probability p_b, again + irrespective of the packet's size. Unlike before, the transport + should take a packet with the byte congestion field marked to + count as a mark on each byte in the packet. + + This hides a fundamental problem -- much more fundamental than + whether we can magically create header space for yet another ECN + flag, or whether it would work while being deployed incrementally. + Distinguishing drop from delivery naturally provides just one + implicit bit of congestion indication information -- the packet is + either dropped or not. It is hard to drop a packet in two ways that + are distinguishable remotely. This is a similar problem to that of + distinguishing wireless transmission losses from congestive losses. + + This problem would not be solved, even if ECN were universally + deployed. A congestion notification protocol must survive a + transition from low levels of congestion to high. Marking two states + is feasible with explicit marking, but it is much harder if packets + are dropped. Also, it will not always be cost-effective to implement + AQM at every low-level resource, so drop will often have to suffice. + + We are not saying two ECN fields will be needed (and we are not + saying that somehow a resource should be able to drop a packet in one + of two different ways so that the transport can distinguish which + sort of drop it was!). These two congestion notification channels + are a conceptual device to illustrate a dilemma we could face in the + future. Section 3 gives four good reasons why it would be a bad idea + to allow for packet size by biasing drop probability in favour of + small packets within the network. The impracticality of our thought + + + +Briscoe & Manner Best Current Practice [Page 38] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + experiment shows that it will be hard to give transports a practical + way to know whether or not to take into account the size of + congestion indication packets. + + Fortunately, this dilemma is not pressing because by design most + equipment becomes bit-congested before its packet processing becomes + congested (as already outlined in Section 1.1). Therefore, + transports can be designed on the relatively sound assumption that a + congestion indication will usually imply bit congestion. + + Nonetheless, although the above idealised protocol isn't intended for + implementation, we do want to emphasise that research is needed to + predict whether there are good reasons to believe that packet + congestion might become more common, and if so, to find a way to + somehow distinguish between bit and packet congestion [RFC3714]. + + Recently, the dual resource queue (DRQ) proposal [DRQ] has been made + on the premise that, as network processors become more cost- + effective, per-packet operations will become more complex + (irrespective of whether more function in the network is desirable). + Consequently the premise is that CPU congestion will become more + common. DRQ is a proposed modification to the RED algorithm that + folds both bit congestion and packet congestion into one signal + (either loss or ECN). + + Finally, we note one further complication. Strictly, packet- + congestible resources are often cycle-congestible. For instance, for + routing lookups, load depends on the complexity of each lookup and + whether or not the pattern of arrivals is amenable to caching. This + also reminds us that any solution must not require a forwarding + engine to use excessive processor cycles in order to decide how to + say it has no spare processor cycles. + +Appendix C. Byte-Mode Drop Complicates Policing Congestion Response + + This section is informative, not normative. + + There are two main classes of approach to policing congestion + response: (i) policing at each bottleneck link or (ii) policing at + the edges of networks. Packet-mode drop in RED is compatible with + either, while byte-mode drop precludes edge policing. + + The simplicity of an edge policer relies on one dropped or marked + packet being equivalent to another of the same size without having to + know which link the drop or mark occurred at. However, the byte-mode + drop algorithm has to depend on the local MTU of the line -- it needs + to use some concept of a 'normal' packet size. Therefore, one + dropped or marked packet from a byte-mode drop algorithm is not + + + +Briscoe & Manner Best Current Practice [Page 39] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + + necessarily equivalent to another from a different link. A policing + function local to the link can know the local MTU where the + congestion occurred. However, a policer at the edge of the network + cannot, at least not without a lot of complexity. + + The early research proposals for type (i) policing at a bottleneck + link [pBox] used byte-mode drop, then detected flows that contributed + disproportionately to the number of packets dropped. However, with + no extra complexity, later proposals used packet-mode drop and looked + for flows that contributed a disproportionate amount of dropped bytes + [CHOKe_Var_Pkt]. + + Work is progressing on the Congestion Exposure (ConEx) protocol + [RFC6789], which enables a type (ii) edge policer located at a user's + attachment point. The idea is to be able to take an integrated view + of the effect of all a user's traffic on any link in the + internetwork. However, byte-mode drop would effectively preclude + such edge policing because of the MTU issue above. + + Indeed, making drop probability depend on the size of the packets + that bits happen to be divided into would simply encourage the bits + to be divided into smaller packets in order to confuse policing. In + contrast, as long as a dropped/marked packet is taken to mean that + all the bytes in the packet are dropped/marked, a policer can remain + robust against sequences of bits being re-divided into different size + packets or across different size flows [Rate_fair_Dis]. + + + + + + + + + + + + + + + + + + + + + + + + + +Briscoe & Manner Best Current Practice [Page 40] + +RFC 7141 Byte and Packet Congestion Notification February 2014 + + +Authors' Addresses + + Bob Briscoe + BT + B54/77, Adastral Park + Martlesham Heath + Ipswich IP5 3RE + UK + + Phone: +44 1473 645196 + EMail: bob.briscoe@bt.com + URI: http://bobbriscoe.net/ + + Jukka Manner + Aalto University + Department of Communications and Networking (Comnet) + P.O. Box 13000 + FIN-00076 Aalto + Finland + + Phone: +358 9 470 22481 + EMail: jukka.manner@aalto.fi + URI: http://www.netlab.tkk.fi/~jmanner/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Briscoe & Manner Best Current Practice [Page 41] + |