path: root/doc/rfc/rfc9331.txt

Internet Engineering Task Force (IETF)                    K. De Schepper
Request for Comments: 9331                               Nokia Bell Labs
Category: Experimental                                   B. Briscoe, Ed.
ISSN: 2070-1721                                              Independent
                                                            January 2023


The Explicit Congestion Notification (ECN) Protocol for Low Latency, Low
                  Loss, and Scalable Throughput (L4S)

Abstract

   This specification defines the protocol to be used for a new network
   service called Low Latency, Low Loss, and Scalable throughput (L4S).
   L4S uses an Explicit Congestion Notification (ECN) scheme at the IP
   layer that is similar to the original (or 'Classic') ECN approach,
   except as specified within.  L4S uses 'Scalable' congestion control,
   which induces much more frequent control signals from the network,
   and it responds to them with much more fine-grained adjustments so
   that very low (typically sub-millisecond on average) and consistently
   low queuing delay becomes possible for L4S traffic without
   compromising link utilization.  Thus, even capacity-seeking (TCP-
   like) traffic can have high bandwidth and very low delay at the same
   time, even during periods of high traffic load.

   The L4S identifier defined in this document distinguishes L4S from
   'Classic' (e.g., TCP-Reno-friendly) traffic.  Then, network
   bottlenecks can be incrementally modified to distinguish and isolate
   existing traffic that still follows the Classic behaviour, to prevent
   it from degrading the low queuing delay and low loss of L4S traffic.
   This Experimental specification defines the rules that L4S transports
   and network elements need to follow, with the intention that L4S
   flows neither harm each other's performance nor that of Classic
   traffic.  It also suggests open questions to be investigated during
   experimentation.  Examples of new Active Queue Management (AQM)
   marking algorithms and new transports (whether TCP-like or real time)
   are specified separately.

Status of This Memo

   This document is not an Internet Standards Track specification; it is
   published for examination, experimental implementation, and
   evaluation.

   This document defines an Experimental Protocol for the Internet
   community.  This document is a product of the Internet Engineering
   Task Force (IETF).  It represents the consensus of the IETF
   community.  It has received public review and has been approved for
   publication by the Internet Engineering Steering Group (IESG).  Not
   all documents approved by the IESG are candidates for any level of
   Internet Standard; see Section 2 of RFC 7841.

   Information about the current status of this document, any errata,
   and how to provide feedback on it may be obtained at
   https://www.rfc-editor.org/info/rfc9331.

Copyright Notice

   Copyright (c) 2023 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Revised BSD License text as described in Section 4.e of the
   Trust Legal Provisions and are provided without warranty as described
   in the Revised BSD License.

Table of Contents

   1.  Introduction
     1.1.  Latency, Loss, and Scaling Problems
     1.2.  Terminology
     1.3.  Scope
   2.  L4S Packet Identification: Document Roadmap
   3.  Choice of L4S Packet Identifier: Requirements
   4.  Transport-Layer Behaviour (the 'Prague L4S Requirements')
     4.1.  Codepoint Setting
     4.2.  Prerequisite Transport Feedback
     4.3.  Prerequisite Congestion Response
       4.3.1.  Guidance on Congestion Response in the RFC Series
     4.4.  Filtering or Smoothing of ECN Feedback
   5.  Network Node Behaviour
     5.1.  Classification and Re-Marking Behaviour
     5.2.  The Strength of L4S CE Marking Relative to Drop
     5.3.  Exception for L4S Packet Identification by Network Nodes
           with Transport-Layer Awareness
     5.4.  Interaction of the L4S Identifier with Other Identifiers
       5.4.1.  DualQ Examples of Other Identifiers Complementing L4S
               Identifiers
         5.4.1.1.  Inclusion of Additional Traffic with L4S
         5.4.1.2.  Exclusion of Traffic from L4S Treatment
         5.4.1.3.  Generalized Combination of L4S and Other
                 Identifiers
       5.4.2.  Per-flow Queuing Examples of Other Identifiers
               Complementing L4S Identifiers
     5.5.  Limiting Packet Bursts from Links
       5.5.1.  Limiting Packet Bursts from Links Fed by an L4S AQM
       5.5.2.  Limiting Packet Bursts from Links Upstream of an L4S
               AQM
   6.  Behaviour of Tunnels and Encapsulations
     6.1.  No Change to ECN Tunnels and Encapsulations in General
     6.2.  VPN Behaviour to Avoid Limitations of Anti-Replay
   7.  L4S Experiments
     7.1.  Open Questions
     7.2.  Open Issues
     7.3.  Future Potential
   8.  IANA Considerations
   9.  Security Considerations
   10. References
     10.1.  Normative References
     10.2.  Informative References
   Appendix A.  Rationale for the 'Prague L4S Requirements'
     A.1.  Rationale for the Requirements for Scalable Transport
           Protocols
       A.1.1.  Use of L4S Packet Identifier
       A.1.2.  Accurate ECN Feedback
       A.1.3.  Capable of Replacement by Classic Congestion Control
       A.1.4.  Fall Back to Classic Congestion Control on Packet Loss
       A.1.5.  Coexistence with Classic Congestion Control at Classic
               ECN Bottlenecks
       A.1.6.  Reduce RTT Dependence
       A.1.7.  Scaling Down to Fractional Congestion Windows
       A.1.8.  Measuring Reordering Tolerance in Time Units
     A.2.  Scalable Transport Protocol Optimizations
       A.2.1.  Setting ECT in Control Packets and Retransmissions
       A.2.2.  Faster than Additive Increase
       A.2.3.  Faster Convergence at Flow Start
   Appendix B.  Compromises in the Choice of L4S Identifier
   Appendix C.  Potential Competing Uses for the ECT(1) Codepoint
     C.1.  Integrity of Congestion Feedback
     C.2.  Notification of Less Severe Congestion than CE
   Acknowledgements
   Authors' Addresses

1.  Introduction

   This Experimental specification defines the protocol to be used for a
   new network service called Low Latency, Low Loss, and Scalable
   throughput (L4S).  L4S uses an Explicit Congestion Notification (ECN)
   scheme at the IP layer with the same set of codepoint transitions as
   the original (or 'Classic') ECN [RFC3168].  [RFC3168] requires an ECN
   mark to be equivalent to a drop, both when applied in the network and
   when responded to by a transport.  Unlike Classic ECN marking, i) the
   network applies L4S marking more immediately and more frequently than
   drop and ii) the transport response to each mark is reduced and
   smoothed relative to that for drop.  The two changes counterbalance
   each other so that the throughput of an L4S flow will be roughly the
   same as a comparable non-L4S flow under the same conditions.
   Nonetheless, the much more frequent ECN control signals and the finer
   responses to these signals result in very low queuing delay without
   compromising link utilization, and this low delay can be maintained
   during high load.  For instance, queuing delay under heavy and highly
   varying load with the example DCTCP/DualQ solution described below on
   a DSL or Ethernet link is sub-millisecond on average and roughly 1 to
   2 milliseconds at the 99th percentile without losing link utilization
   [L4Seval22] [DualPI2Linux].  Note that the queuing delay while
   waiting to acquire a shared medium such as wireless has to be added
   to the above.  It is a different issue that needs to be addressed,
   but separately (see Section 6.3 of the L4S architecture [RFC9330]).

   L4S relies on 'Scalable' congestion controls for these delay
   properties and for preserving low delay as flow rate scales, hence
   the name.  The congestion control used in Data Center TCP (DCTCP) is
   an example of a Scalable congestion control, but DCTCP is applicable
   solely to controlled environments like data centres [RFC8257],
   because it is too aggressive to coexist with existing TCP-Reno-
   friendly traffic.  Dual-Queue Coupled AQM, which is defined in a
   complementary Experimental specification [RFC9332], is an AQM
   framework that enables Scalable congestion controls derived from
   DCTCP to coexist with existing traffic, each getting roughly the same
   flow rate when they compete under similar conditions.  Note that a
   Scalable congestion control is still not safe to deploy on the
   Internet unless it satisfies the requirements listed in Section 4.

   L4S is not only for elastic (TCP-like) traffic -- there are Scalable
   congestion controls for real-time media, such as the L4S variant
   [SCReAM-L4S] of the SCReAM [RFC8298] RTP Media Congestion Avoidance
   Techniques (RMCAT).  The factor that distinguishes L4S from Classic
   traffic is its behaviour in response to congestion.  The transport
   wire protocol, e.g., TCP, QUIC, the Stream Control Transmission
   Protocol (SCTP), the Datagram Congestion Control Protocol (DCCP), or
   RTP/RTCP, is orthogonal (and therefore not suitable for
   distinguishing L4S from Classic packets).

   The L4S identifier defined in this document is the key piece that
   distinguishes L4S from 'Classic' (e.g., Reno-friendly) traffic.
   Then, network bottlenecks can be incrementally modified to
   distinguish and isolate existing Classic traffic from L4S traffic, to
   prevent the former from degrading the very low queuing delay and loss
   of the new Scalable transports, without harming Classic performance
   at these bottlenecks.  Although both sender and network deployment
   are required before any benefit, initial implementations of the
   separate parts of the system have been motivated by the potential
   performance benefits.

1.1.  Latency, Loss, and Scaling Problems

   Latency is becoming the critical performance factor for many (perhaps
   most) Internet applications, e.g., interactive web, web services,
   voice, conversational video, interactive video, interactive remote
   presence, instant messaging, online gaming, remote desktop, cloud-
   based applications & services, and remote control of machinery and
   industrial processes.  In many parts of the world, further increases
   in access network bitrate offer diminishing returns [Dukkipati06],
   whereas latency is still a multi-faceted problem.  As a result, much
   has been done to reduce propagation time by placing caches or servers
   closer to users.  However, queuing remains a major, albeit
   intermittent, component of latency.

   The Diffserv architecture provides Expedited Forwarding (EF)
   [RFC3246] so that low-latency traffic can jump the queue of other
   traffic.  If growth in latency-sensitive applications continues,
   periods with solely latency-sensitive traffic will become
   increasingly common on links where traffic aggregation is low.
   During these periods, if all the traffic were marked for the same
   treatment, Diffserv would make no difference.  The links with low
   aggregation also tend to become the path bottleneck under load, for
   instance, the access links dedicated to individual sites (homes,
   small enterprises, or mobile devices).  So, instead of
   differentiation, it becomes imperative to remove the underlying
   causes of any unnecessary delay.

   The Bufferbloat project has shown that excessively large buffering
   ('bufferbloat') has been introducing significantly more delay than
   the underlying propagation time [Bufferbloat].  These delays appear
   only intermittently -- only when a capacity-seeking (e.g., TCP) flow
   is long enough for the queue to fill the buffer, causing every packet
   in other flows sharing the buffer to have to work its way through the
   queue.

   AQM was originally developed to solve this problem (and others).
   Unlike Diffserv, which gives low latency to some traffic at the
   expense of others, AQM controls latency for _all_ traffic in a class.
   In general, AQM methods introduce an increasing level of discard from
   the buffer, the longer the queue persists above a shallow threshold.
   This gives sufficient signals to capacity-seeking (a.k.a. greedy)
   flows to keep the buffer empty for its intended purpose: absorbing
   bursts.  However, Random Early Detection (RED) and other algorithms
   from the 1990s were sensitive to their configuration and hard to set
   correctly [RFC7567].  So this form of AQM was not widely deployed.

   More recent state-of-the-art AQM methods, such as Flow Queue CoDel
   [RFC8290], Proportional Integral controller Enhanced (PIE) [RFC8033],
   or Adaptive RED [ARED01], are easier to configure, because they
   define the queuing threshold in time not bytes, so configuration is
   invariant whatever the link rate.  However, the sawtoothing window of
   a Classic congestion control creates a dilemma for the operator:
   either i) configure a shallow AQM operating point so the tips of the
   sawteeth cause minimal queue delay, but then the troughs underutilize
   the link, or ii) configure the operating point deeper into the buffer
   so the troughs utilize the link better, but then the tips cause more
   delay variation.  Even with a perfectly tuned AQM, the additional
   queuing delay at the tips of the sawteeth will be of the same order
   as the underlying base round-trip time (RTT), thereby roughly
   doubling the total RTT.

   If a sender's own behaviour is introducing queuing delay variation,
   no AQM in the network can 'un-vary' the delay without significantly
   compromising link utilization.  Even flow queuing (e.g., [RFC8290]),
   which isolates one flow from another, cannot isolate a flow from the
   delay variations it inflicts on itself.  Therefore, those
   applications that need to seek out high bandwidth but also need low
   latency will have to migrate to Scalable congestion control, which
   uses much smaller sawtooth variations.

   Altering host behaviour is not enough on its own though.  Even if
   hosts adopt low-latency Scalable congestion controls, they need to be
   isolated from the large queue variations induced by existing Classic
   congestion controls.  L4S AQMs provide that latency isolation in the
   network, and the L4S identifier enables the AQMs to distinguish the
   two types of packets that need to be isolated: L4S and Classic.  L4S
   isolation can be achieved with a queue per flow (e.g., [RFC8290]),
   but a DualQ [RFC9332] is sufficient and actually gives better tail
   latency [DCttH19].  Both approaches are addressed in this document.

   The DualQ solution was developed to make very low latency available
   without requiring per-flow queues at every bottleneck.  This was
   useful because per-flow queuing (FQ) has well-known downsides -- not
   least the need to inspect transport-layer headers in the network,
   which makes it incompatible with privacy approaches such as IPsec
   Virtual Private Network (VPN) tunnels and incompatible with link-
   layer queue management, where transport-layer headers can be hidden,
   e.g., 5G.

   Latency is not the only concern addressed by L4S.  It was known when
   TCP congestion avoidance was first developed that it would not scale
   to high bandwidth-delay products (see footnote 6 of Jacobson and
   Karels [TCP-CA]).  Given that Reno congestion control is already
   beyond its scaling range at regular broadband bitrates over WAN
   distances [RFC3649], 'less unscalable' CUBIC [RFC8312] and Compound
   [CTCP] variants of TCP have been successfully deployed.  However,
   these are now approaching their scaling limits.  Unfortunately, fully
   Scalable congestion controls such as DCTCP [RFC8257] outcompete
   Classic ECN congestion controls sharing the same queue, which is why
   they have been confined to private data centres or research testbeds.

   It turns out that these Scalable congestion control algorithms that
   solve the latency problem can also solve the scalability problem of
   Classic congestion controls.  The finer sawteeth in the congestion
   window (cwnd) have low amplitude, so they cause very little queuing
   delay variation, and the average time to recover from one congestion
   signal to the next (the average duration of each sawtooth) remains
   invariant, which maintains constant tight control as flow rate
   scales.  A background paper [L4Seval22] gives the full explanation of
   why the design solves both the latency and the scaling problems, both
   in plain English and in more precise mathematical form.  The
   explanation is summarized without the mathematics in Section 4 of the
   L4S architecture [RFC9330].

1.2.  Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

   Classic Congestion Control:  A congestion control behaviour that can
      coexist with standard Reno [RFC5681] without causing significantly
      negative impact on its flow rate [RFC5033].  With Classic
      congestion controls, such as Reno or CUBIC, because flow rate has
      scaled since TCP congestion control was first designed in 1988, it
      now takes hundreds of round trips (and growing) to recover after a
      congestion signal (whether a loss or an ECN mark) as shown in the
      examples in Section 5.1 of the L4S architecture [RFC9330] and in
      [RFC3649].  Therefore, control of queuing and utilization becomes
      very slack, and the slightest disturbances (e.g., from new flows
      starting) prevent a high rate from being attained.

   Scalable Congestion Control:  A congestion control where the average
      time from one congestion signal to the next (the recovery time)
      remains invariant as flow rate scales, all other factors being
      equal.  This maintains the same degree of control over queuing and
      utilization whatever the flow rate, as well as ensuring that high
      throughput is robust to disturbances.  For instance, DCTCP
      averages 2 congestion signals per round trip, whatever the flow
      rate, as do other recently developed Scalable congestion controls,
      e.g., Relentless TCP [RELENTLESS], Prague for TCP and QUIC
      [PRAGUE-CC] [PragueLinux], the L4S ECN part of Bottleneck
      Bandwidth and Round-trip propagation time (BBRv2) [BBRv2]
      [BBR-CC], and the L4S variant of SCReAM for real-time media
      [SCReAM-L4S] [RFC8298].  See Section 4.3 for more explanation.

   Classic Service:  The Classic service is intended for all the
      congestion control behaviours that coexist with Reno [RFC5681]
      (e.g., Reno itself, CUBIC [RFC8312], Compound [CTCP], and TFRC
      [RFC5348]).  The term 'Classic queue' means a queue providing the
      Classic service.

   Low Latency, Low Loss, and Scalable throughput (L4S) service:  The
      'L4S' service is intended for traffic from Scalable congestion
      control algorithms, such as the Prague congestion control
      [PRAGUE-CC], which was derived from DCTCP [RFC8257].  The L4S
      service is for more general traffic than just Prague -- it allows
      the set of congestion controls with similar scaling properties to
      Prague to evolve, such as the examples listed above (Relentless,
      SCReAM, etc.).  The term 'L4S queue' means a queue providing the
      L4S service.

      The terms Classic or L4S can also qualify other nouns, such as
      'queue', 'codepoint', 'identifier', 'classification', 'packet',
      and 'flow'.  For example, an L4S packet means a packet with an L4S
      identifier sent from an L4S congestion control.

      Both Classic and L4S services can cope with a proportion of
      unresponsive or less-responsive traffic as well but, in the L4S
      case, its rate has to be smooth enough or low enough to not build
      a queue (e.g., DNS, Voice over IP (VoIP), game sync datagrams,
      etc.).

   Reno-friendly:  The subset of Classic traffic that is friendly to the
      standard Reno congestion control defined for TCP in [RFC5681].
      The TFRC spec [RFC5348] indirectly implies that 'friendly' is
      defined as "generally within a factor of two of the sending rate
      of a TCP flow under the same conditions".  Reno-friendly is used
      here in place of 'TCP-friendly', given the latter has become
      imprecise, because the TCP protocol is now used with so many
      different congestion control behaviours, and Reno is used in non-
      TCP transports, such as QUIC [RFC9000].

   Classic ECN:  The original Explicit Congestion Notification (ECN)
      protocol [RFC3168] that requires ECN signals to be treated as
      equivalent to drops, both when generated in the network and when
      responded to by the sender.

      For L4S, the names used for the four codepoints of the 2-bit IP-
      ECN field are unchanged from those defined in the ECN spec
      [RFC3168], i.e., Not-ECT, ECT(0), ECT(1), and CE, where ECT stands
      for ECN-Capable Transport and CE stands for Congestion
      Experienced.  A packet marked with the CE codepoint is termed
      'ECN-marked' or sometimes just 'marked' where the context makes
      ECN obvious.

   Site:  A home, mobile device, small enterprise, or campus where the
      network bottleneck is typically the access link to the site.  Not
      all network arrangements fit this model, but it is a useful,
      widely applicable generalization.

1.3.  Scope

   The new L4S identifier defined in this specification is applicable
   for IPv4 and IPv6 packets (as is Classic ECN [RFC3168]).  It is
   applicable for the unicast, multicast, and anycast forwarding modes.

   The L4S identifier is an orthogonal packet classification to the
   Differentiated Services Code Point (DSCP) [RFC2474].  Section 5.4
   explains what this means in practice.

   This document is Experimental, so it does not update any Standards
   Track RFCs.  Therefore, it depends on [RFC8311], which is a Standards
   Track specification that:

   *  updates the ECN Proposed Standard [RFC3168] to allow Experimental
      RFCs to relax the requirement that an ECN mark must be equivalent
      to a drop (when the network applies markings and/or when the
      sender responds to them).  For instance, in the Alternative
      Backoff with ECN (ABE) experiment [RFC8511], this relaxation
      permits a sender to respond less to ECN marks than to drops;

   *  changes the status of the Experimental ECN nonce spec [RFC3540] to
      Historic; and

   *  makes consequent updates to the following additional Proposed
      Standard RFCs to reflect the above two bullets:

      -  ECN for RTP [RFC6679] and

      -  the congestion control specifications of various DCCP
         Congestion Control Identifier (CCID) profiles [RFC4341]
         [RFC4342] [RFC5622].

   This document is about identifiers that are used for interoperation
   between hosts and networks.  So the audience is broad, covering
   developers of host transports and network AQMs, as well as covering
   how operators might wish to combine various identifiers, which would
   require flexibility from equipment developers.

2.  L4S Packet Identification: Document Roadmap

   The L4S ECN marking treatment is an experimental alternative to the
   Classic ECN treatment in [RFC3168], which has been updated by
   [RFC8311] to allow experiments such as the one defined in the present
   specification.  [RFC4774] discusses some of the issues and evaluation
   criteria when defining alternative ECN semantics, which are further
   discussed in Section 4.3.1.

   The L4S architecture [RFC9330] describes the three main components of
   L4S: the sending host behaviour, the marking behaviour in the
   network, and the L4S ECN protocol that identifies L4S packets as they
   flow between the two.

   Section 3 of this document records the requirements that informed the
   choice of L4S identifier.  Then, subsequent sections specify the L4S
   ECN protocol, which i) identifies packets that have been sent from
   hosts that are expected to comply with a broad type of sending
   behaviour and ii) identifies the marking treatment that network nodes
   are expected to apply to L4S packets.

   For a packet to receive L4S treatment as it is forwarded, the sender
   sets the ECN field in the IP header to the ECT(1) codepoint.  See
   Section 4 for full transport-layer behaviour requirements, including
   feedback and congestion response.

   A network node that implements the L4S service always classifies
   arriving ECT(1) packets for L4S treatment and by default classifies
   CE packets for L4S treatment unless the heuristics described in
   Section 5.3 are employed.  See Section 5 for full network element
   behaviour requirements, including classification, ECN marking, and
   interaction of the L4S identifier with other identifiers and per-hop
   behaviours.

   L4S ECN works with ECN tunnelling and encapsulation behaviour as is,
   except there is one known case where careful attention to
   configuration is required, which is detailed in Section 6.

   This specification of L4S ECN currently has Experimental status.  So
   Section 7 collects the general questions and issues that remain open
   for investigation during L4S experimentation.  Open issues or
   questions specific to particular components are called out in the
   specifications of each component part, such as the DualQ [RFC9332].

   The IANA assignment of the L4S identifier is specified in Section 8.
   And Section 9 covers security considerations specific to the L4S
   identifier.  System security aspects, such as policing and privacy,
   are covered in the L4S architecture [RFC9330].

3.  Choice of L4S Packet Identifier: Requirements

   This subsection briefly records the process that led to the chosen
   L4S identifier.

   The identifier for packets using the L4S service needs to meet the
   following requirements:

   *  it SHOULD survive end to end between source and destination
      endpoints: across the boundary between host and network, between
      interconnected networks, and through middleboxes;

   *  it SHOULD be visible at the IP layer;

   *  it SHOULD be common to IPv4 and IPv6 and be transport-agnostic;

   *  it SHOULD be incrementally deployable;

   *  it SHOULD enable an AQM to classify packets encapsulated by outer
      IP or lower-layer headers;

   *  it SHOULD consume minimal extra codepoints; and

   *  it SHOULD be consistent on all the packets of a transport-layer
      flow, so that some packets of a flow are not served by a different
      queue to others.

   Whether the identifier would be recoverable if the experiment failed
   is a factor that could be taken into account.  However, this has not
   been made a requirement, because that would favour schemes that would
   be easier to fail rather than more likely to succeed.

   It is recognized that any choice of identifier is unlikely to satisfy
   all these requirements, particularly given the limited space left in
   the IP header.  Therefore, a compromise will always be necessary,
   which is why all the above requirements are expressed with the word
   "SHOULD" not "MUST".

   After extensive assessment of alternative schemes, "ECT(1) and CE
   codepoints" was chosen as the best compromise.  Therefore, this
   scheme is defined in detail in the following sections, while
   Appendix B records its pros and cons against the above requirements.

4.  Transport-Layer Behaviour (the 'Prague L4S Requirements')

   This section defines L4S behaviour at the transport layer, also known
   as the 'Prague L4S Requirements' (see Appendix A for the origin of
   the name).

4.1.  Codepoint Setting

   A sender that wishes a packet to receive L4S treatment as it is
   forwarded MUST set the ECN field in the IP header (v4 or v6) to the
   ECT(1) codepoint.

4.2.  Prerequisite Transport Feedback

   For a transport protocol to provide Scalable congestion control
   (Section 4.3), it MUST provide feedback of the extent of CE marking
   on the forward path.  When ECN was added to TCP [RFC3168], the
   feedback method reported no more than one CE mark per round trip.
   Some transport protocols derived from TCP mimic this behaviour while
   others report the accurate extent of ECN marking.  This means that
   some transport protocols will need to be updated as a prerequisite
   for Scalable congestion control.  The position for a few well-known
   transport protocols is given below.

   TCP:  Support for the accurate ECN feedback requirements [RFC7560]
      (such as that provided by AccECN [ACCECN]) by both ends is a
      prerequisite for Scalable congestion control in TCP.  Therefore,
      the presence of ECT(1) in the IP headers even in one direction of
      a TCP connection will imply that both ends support accurate ECN
      feedback.  However, the converse does not apply.  So even if both
      ends support AccECN, either of the two ends can choose not to use
      a Scalable congestion control, whatever the other end's choice is.

   SCTP:  A suitable ECN feedback mechanism for SCTP could add a chunk
      to report the number of received CE marks (as described in a long-
      expired document [SCTP-ECN] or as sketched out in Appendix A of
      the now-obsolete second Standards Track specification of SCTP
      [RFC4960]).

   RTP over UDP:  A prerequisite for Scalable congestion control is for
      both (all) ends of one media-level hop to signal ECN support
      [RFC6679] and use the new generic RTCP feedback format of
      [RFC8888].  The presence of ECT(1) implies that both (all) ends of
      that media-level hop support ECN.  However, the converse does not
      apply.  So each end of a media-level hop can independently choose
      not to use a Scalable congestion control, even if both ends
      support ECN.

   QUIC:  Support for sufficiently fine-grained ECN feedback is provided
      by IETF QUIC transport v1 [RFC9000].

   DCCP:  The Acknowledgement (ACK) vector in DCCP [RFC4340] is already
      sufficient to report the extent of CE marking as needed by a
      Scalable congestion control.

4.3.  Prerequisite Congestion Response

   As a condition for a host to send packets with the L4S identifier
   (ECT(1)), it SHOULD implement a congestion control behaviour that
   ensures that, in steady state, the average duration between induced
   ECN marks does not increase as flow rate scales up, all other factors
   being equal.  This is termed a Scalable congestion control.  This
   invariant duration ensures that, as flow rate scales, the average
   period with no feedback information about capacity does not become
   excessive.  It also ensures that queue variations remain small,
   without having to sacrifice utilization.

   With a congestion control that sawtooths to probe capacity, this
   duration is called the recovery time, because each time the sawtooth
   yields, on average it takes this time to recover to its previous high
   point.  A Scalable congestion control does not have to sawtooth, but
   it has to coexist with Scalable congestion controls that do.

   For instance, for DCTCP [RFC8257], TCP Prague [PRAGUE-CC]
   [PragueLinux], and the L4S variant of SCReAM [SCReAM-L4S] [RFC8298],
   the average recovery time is always half a round trip (or half a
   reference round trip), whatever the flow rate.

   As with all transport behaviours, a detailed specification (probably
   an Experimental RFC) is expected for each congestion control,
   following the guidelines for specifying new congestion control
   algorithms in [RFC5033].  In addition, it is expected that these L4S-
   specific matters will be documented, specifically the timescale over
   which the proportionality is averaged and the control of burstiness.
   The recovery time requirement above is worded as a "SHOULD" rather
   than a "MUST" to allow reasonable flexibility for such
   implementations.

   The condition 'all other factors being equal' allows the recovery
   time to be different for different round-trip times, as long as it
   does not increase with flow rate for any particular RTT.

   Saying that the recovery time remains roughly invariant is equivalent
   to saying that the number of ECN CE marks per round trip remains
   invariant as flow rate scales, all other factors being equal.  For
   instance, an average recovery time of half of 1 RTT is equivalent to
   2 ECN marks per round trip.  For those familiar with steady-state
   congestion response functions, it is also equivalent to say that the
   congestion window is inversely proportional to the proportion of
   bytes in packets marked with the CE codepoint (see Section 2 of
   [PI2]).

   In order to coexist safely with other Internet traffic, a Scalable
   congestion control is not allowed to tag its packets with the ECT(1)
   codepoint unless it complies with the following numbered requirements
   and recommendations:

   1.  A Scalable congestion control MUST be capable of being replaced
       by a Classic congestion control (by application and/or by
       administrative control).  If a Classic congestion control is
       activated, it will not tag its packets with the ECT(1) codepoint
       (see Appendix A.1.3 for rationale).

   2.  As well as responding to ECN markings, a Scalable congestion
       control MUST react to packet loss in a way that will coexist
       safely with Classic congestion controls such as standard Reno
       [RFC5681], as required by [RFC5033] (see Appendix A.1.4 for
       rationale).

   3.  In uncontrolled environments, monitoring MUST be implemented to
       support detection of problems with an ECN-capable AQM at the path
       bottleneck that appears not to support L4S and that might be in a
       shared queue.  Such monitoring SHOULD be applied to live traffic
       that is using Scalable congestion control.  Alternatively,
       monitoring need not be applied to live traffic, if monitoring
       with test traffic has been arranged to cover the paths that live
       traffic takes through uncontrolled environments.

       A function to detect the above problems with an ECN-capable AQM
       MUST also be implemented and used.  The detection function SHOULD
       be capable of making the congestion control adapt its ECN-marking
       response in real time to coexist safely with Classic congestion
       controls such as standard Reno [RFC5681], as required by
       [RFC5033].  This could be complemented by more detailed offline
       detection of potential problems.  If only offline detection is
       used and potential problems with such an AQM are detected on
       certain paths, the Scalable congestion control MUST be replaced
       by a Classic congestion control, at least for the problem paths.

       See Section 4.3.1, Appendix A.1.5, and the L4S operational
       guidance [L4SOPS] for rationale and explanation.

          Note that a Scalable congestion control is not expected to
          change to setting ECT(0) while it transiently adapts to
          coexist with Classic congestion controls, whereas a
          replacement congestion control that solely behaves in the
          Classic way will set ECT(0).

   4.  In the range between the minimum likely RTT and typical RTTs
       expected in the intended deployment scenario, a Scalable
       congestion control MUST converge towards a rate that is as
       independent of RTT as is possible without compromising stability
       or utilization (see Appendix A.1.6 for rationale).

   5.  A Scalable congestion control SHOULD remain responsive to
       congestion when typical RTTs over the public Internet are
       significantly smaller because they are no longer inflated by
       queuing delay.  It would be preferable for the minimum window of
       a Scalable congestion control to be lower than 1 segment rather
       than use the timeout approach described for TCP in Section 6.1.2
       of the ECN spec [RFC3168] (or an equivalent for other
       transports).  However, a lower minimum is not set as a formal
       requirement for L4S experiments (see Appendix A.1.7 for
       rationale).

   6.  A Scalable congestion control's loss detection SHOULD be
       resilient to reordering over an adaptive time interval that
       scales with throughput and adapts to reordering (as in Recent ACK
       (RACK) [RFC8985]), as opposed to counting only in fixed units of
       packets (as in the 3 Duplicate ACK (DupACK) rule of NewReno
       [RFC5681] [RFC6675], which is not scalable).  As data rates
       increase (e.g., due to new and/or improved technology),
       congestion controls that detect loss by counting in units of
       packets become more likely to incorrectly treat reordering events
       as congestion-caused loss events (see Appendix A.1.8 for further
       rationale).  This requirement does not apply to congestion
       controls that are solely used in controlled environments where
       the network introduces hardly any reordering.

   7.  A Scalable congestion control is expected to limit the queue
       caused by bursts of packets.  It would not seem necessary to set
       the limit any lower than 10% of the minimum RTT expected in a
       typical deployment (e.g., additional queuing of roughly 250 us
       for the public Internet).  This would be converted to a number of
       packets by multiplying by the current average packet rate.  Then,
       the queue caused by each burst at the bottleneck link would not
       exceed 250 us (under the worst-case assumption that the flow is
       filling the capacity).  No normative requirement to limit bursts
       is given here, and until there is more industry experience from
       the L4S experiment, it is not even known whether one is needed --
       it seems to be in an L4S sender's self-interest to limit bursts.

   Each sender in a session can use a Scalable congestion control
   independently of the congestion control used by the receiver(s) when
   they send data.  Therefore, there might be ECT(1) packets in one
   direction and ECT(0) or Not-ECT in the other.

   Later, this document discusses the conditions for mixing other
   "'Safe' Unresponsive Traffic" (e.g., DNS, Lightweight Directory
   Access Protocol (LDAP), NTP, voice, and game sync packets) with L4S
   traffic; see Section 5.4.1.1.  To be clear, although such traffic can
   share the same queue as L4S traffic, it is not appropriate for the
   sender to tag it as ECT(1), except in the (unlikely) case that it
   satisfies the above conditions.

4.3.1.  Guidance on Congestion Response in the RFC Series

   [RFC3168] requires the congestion responses to a CE-marked packet and
   a dropped packet to be the same.  [RFC8311] is a Standards Track
   update to [RFC3168] that is intended to enable experimentation with
   ECN, including the L4S experiment.  [RFC8311] allows an experimental
   congestion control's response to a CE-marked packet to differ from
   the response to a dropped packet, provided that the differences are
   documented in an Experimental RFC, such as the present document.

   BCP 124 [RFC4774] gives guidance to protocol designers, when
   specifying alternative semantics for the IP-ECN field.  [RFC8311]
   explained that it did not need to update the best current practice in
   BCP 124 in order to relax the 'equivalence with drop' requirement
   because, although BCP 124 quotes the same requirement from [RFC3168],
   the BCP does not impose requirements based on it.  BCP 124 [RFC4774]
   describes three options for incremental deployment, with Option 3 (in
   Section 4.3 of BCP 124 [RFC4774]) best matching the L4S case.  Option
   3's requirement for end-nodes is that they respond to CE marks "in a
   way that is friendly to flows using IETF-conformant congestion
   control."  This echoes other general congestion control requirements
   in the RFC Series, for example, [RFC5033] states that "...congestion
   controllers that have a significantly negative impact on traffic
   using standard congestion control may be suspect" and [RFC8085],
   which concerns UDP congestion control, states that "Bulk-transfer
   applications that choose not to implement TFRC or TCP-like windowing
   SHOULD implement a congestion control scheme that results in
   bandwidth (capacity) use that competes fairly with TCP within an
   order of magnitude."

   The normative Item 3 in Section 4.3 above (which concerns L4S
   response to congestion from a Classic ECN AQM) aims to ensure that
   these 'coexistence' requirements are satisfied, but it makes some
   compromises.  This subsection highlights and justifies those
   compromises, and Appendix A.1.5 and the L4S operational guidance
   [L4SOPS] give detailed analysis, examples, and references (the
   normative text in that bullet takes precedence if any informative
   elaboration leads to ambiguity).  The approach is based on an
   assessment of the risk of harm, which is a combination of the
   prevalence of the conditions necessary for harm to occur, and the
   potential severity of the harm if they do.

   Prevalence:  There are three cases:

      *  Drop Tail: Coexistence between L4S and Classic flows is not in
         doubt where the bottleneck does not support any form of ECN,
         which has remained by far the most prevalent case since the ECN
         spec [RFC3168] was published in 2001.

      *  L4S: Coexistence is not in doubt if the bottleneck supports
         L4S.

      *  Classic ECN: The compromises centre around cases where the
         bottleneck supports Classic ECN [RFC3168] but not L4S.  But it
         depends on which sub-case:

         -  Shared Queue with Classic ECN: At the time of writing, the
            members of the Transport Working Group are not aware of any
            current deployments of single-queue Classic ECN bottlenecks
            in the Internet.  Nonetheless, at the scale of the Internet,
            rarity need not imply small numbers nor that there will be
            rarity in the future.

         -  Per-Flow Queues with Classic ECN: Most AQMs with per-flow
            queuing deployed from 2012 onwards had Classic ECN enabled
            by default, specifically FQ-CoDel [RFC8290] and COBALT
            [COBALT].  But the compromises only apply to the second of
            two further sub-cases:

            o  With per-flow queuing, coexistence between Classic and
               L4S flows is not normally a problem, because different
               flows are not meant to be in the same queue (BCP 124
               [RFC4774] did not foresee the introduction of per-flow
               queuing, which appeared as a potential isolation
               technique some eight years after the BCP was published).

            o  However, the isolation between L4S and Classic flows is
               not perfect in cases where the hashes of flow identifiers
               (IDs) collide or where multiple flows within a Layer 3
               VPN are encapsulated within one flow ID.

      To summarize, the coexistence problem is confined to cases of
      imperfect flow isolation in an FQ or in potential cases where a
      Classic ECN AQM has been deployed in a shared queue (see the L4S
      operational guidance [L4SOPS] for further details including recent
      surveys attempting to quantify prevalence).  Further, if one of
      these cases does occur, the coexistence problem does not arise
      unless sources of Classic and L4S flows are simultaneously sharing
      the same bottleneck queue (e.g., different applications in the
      same household), and flows of each type have to be large enough to
      coincide for long enough for any throughput imbalance to have
      developed.  Therefore, how often the coexistence problem arises in
      practice is listed in Section 7 as an open question that L4S
      experiments will need to answer.

   Severity:  Where long-running L4S and Classic flows coincide in a
      shared queue, testing of one L4S congestion control (TCP Prague)
      has found that the imbalance in average throughput between an L4S
      and a Classic flow can reach 25:1 in favour of L4S in the worst
      case [ecn-fallback].  However, when capacity is most scarce, the
      Classic flow gets a higher proportion of the link, for instance,
      over a 4 Mb/s link the throughput ratio is below ~10:1 over paths
      with a base RTT below 100 ms, and it falls below ~5:1 for base
      RTTs below 20 ms.

   These throughput ratios can clearly fall well outside current RFC
   guidance on coexistence.  However, the tendency towards leaving a
   greater share for Classic flows at lower link rate and the very
   limited prevalence of the conditions necessary for harm to occur led
   to the possibility of allowing the RFC requirements to be
   compromised, albeit briefly:

   *  The recommended approach is still to detect and adapt to a Classic
      ECN AQM in real time, which is fully consistent with all the RFCs
      on coexistence.  In other words, the "SHOULD"s in Item 3 of
      Section 4.3 above expect the sender to implement something similar
      to the proof-of-concept code that detects the presence of a
      Classic ECN AQM and falls back to a Classic congestion response
      within a few round trips [ecn-fallback].  However, although this
      code reliably detects a Classic ECN AQM, the current code can also
      wrongly categorize an L4S AQM as Classic, most often in cases when
      link rate is low or RTT is high.  Although this is the safe way
      round, and although implementers are expected to be able to
      improve on this proof of concept, concerns have been raised that
      implementers might lose faith in such detection and disable it.

   *  Item 3 in Section 4.3 above therefore allows a compromise where
      coexistence could briefly diverge from the requirements in the RFC
      Series, but mandatory monitoring is required in order to detect
      such cases and trigger remedial action.  This approach tolerates a
      brief divergence from the RFCs given the likely low prevalence and
      given harm here means a flow progresses more slowly than it would
      otherwise, but it does progress.  The L4S operational guidance
      [L4SOPS] outlines a range of example remedial actions that include
      alterations to either the sender or the network.  However, the
      final normative requirement in Item 3 of Section 4.3 above places
      ultimate responsibility for remedial action on the sender.  If
      coexistence problems with a Classic ECN AQM are detected (implying
      they have not been resolved by the network), it states that the
      sender "MUST" revert to a Classic congestion control.

   [L4SOPS] also gives example ways in which L4S congestion controls can
   be rolled out initially in lower-risk scenarios.

4.4.  Filtering or Smoothing of ECN Feedback

   Section 5.2 below specifies that an L4S AQM is expected to signal L4S
   ECN immediately, to avoid introducing delay due to filtering or
   smoothing.  This contrasts with a Classic AQM, which filters out
   variations in the queue before signalling ECN marking or drop.  In
   the L4S architecture [RFC9330], responsibility for smoothing out
   these variations shifts to the sender's congestion control.

   This shift of responsibility has the advantage that each sender can
   smooth variations over a timescale proportionate to its own RTT.
   Whereas, in the Classic approach, the network doesn't know the RTTs
   of any of the flows, so it has to smooth out variations for a worst-
   case RTT to ensure stability.  For all the typical flows with shorter
   RTTs than the worst-case, this makes congestion control unnecessarily
   sluggish.

   This also gives an L4S sender the choice not to smooth, depending on
   its context (start-up, congestion avoidance, etc.).  Therefore, this
   document places no requirement on an L4S congestion control to smooth
   out variations in any particular way.  Implementers are encouraged to
   openly publish the approach they take to smoothing as well as results
   and experience they gain during the L4S experiment.

5.  Network Node Behaviour

5.1.  Classification and Re-Marking Behaviour

   A network node that implements the L4S service:

   *  MUST classify arriving ECT(1) packets for L4S treatment, unless
      overridden by another classifier (e.g., see Section 5.4.1.2).

   *  MUST classify arriving CE packets for L4S treatment as well,
      unless overridden by another classifier or unless the exception
      referred to next applies.

      CE packets might have originated as ECT(1) or ECT(0), but the
      above rule to classify them as if they originated as ECT(1) is the
      safe choice (see Appendix B for rationale).  The exception is
      where some flow-aware in-network mechanism happens to be available
      for distinguishing CE packets that originated as ECT(0), as
      described in Section 5.3, but there is no implication that such a
      mechanism is necessary.

   An L4S AQM treatment follows similar codepoint transition rules to
   those in [RFC3168].  Specifically, the ECT(1) codepoint MUST NOT be
   changed to any codepoint other than CE, and CE MUST NOT be changed to
   any other codepoint.  An ECT(1) packet is classified as 'ECN-
   capable', and if congestion increases, an L4S AQM algorithm will
   increasingly mark the IP-ECN field as CE, otherwise forwarding
   packets unchanged as ECT(1).  Necessary conditions for an L4S marking
   treatment are defined in Section 5.2.

   Under persistent overload, an L4S marking treatment MUST begin
   applying drop to L4S traffic until the overload episode has subsided,
   as recommended for all AQM methods in Section 4.2.1 of [RFC7567],
   which follows the similar advice in Section 7 of [RFC3168].  During
   overload, it MUST apply the same drop probability to L4S traffic as
   it would to Classic traffic.

   Where an L4S AQM is transport-aware, this requirement could be
   satisfied by using drop in only the most overloaded individual per-
   flow AQMs.  In a DualQ with flow-aware queue protection (e.g.,
   [DOCSIS-QPROT]), this could be achieved by redirecting packets in
   those flows contributing most to the overload out of the L4S queue so
   that they are subjected to drop in the Classic queue.

   For backward compatibility in uncontrolled environments, a network
   node that implements the L4S treatment MUST also implement an AQM
   treatment for the Classic service as defined in Section 1.2.  This
   Classic AQM treatment need not mark ECT(0) packets, but if it does,
   see Section 5.2 for the strengths of the markings relative to drop.
   It MUST classify arriving ECT(0) and Not-ECT packets for treatment by
   this Classic AQM (for the DualQ Coupled AQM; see the extensive
   discussion on classification in Sections 2.3 and 2.5.1.1 of
   [RFC9332]).

   In case unforeseen problems arise with the L4S experiment, it MUST be
   possible to configure an L4S implementation to disable the L4S
   treatment.  Once disabled, ECT(1) packets MUST be treated as if they
   were Not-ECT.

5.2.  The Strength of L4S CE Marking Relative to Drop

   The relative strengths of L4S CE and drop are irrelevant where AQMs
   are implemented in separate queues per application-flow, which are
   then explicitly scheduled (e.g., with an FQ scheduler as in FQ-CoDel
   [RFC8290]).  Nonetheless, the relationship between them needs to be
   defined for the coupling between L4S and Classic congestion signals
   in a DualQ Coupled AQM [RFC9332], as indicated below.

   Unless an AQM node schedules application flows explicitly, the
   likelihood that the AQM drops a Not-ECT Classic packet (p_C) MUST be
   roughly proportional to the square of the likelihood that it would
   have marked it if it had been an L4S packet (p_L).  That is:

      p_C ~= (p_L / k)^2

   The constant of proportionality (k) does not have to be standardized
   for interoperability, but a value of 2 is RECOMMENDED.  The term
   'likelihood' is used above to allow for marking and dropping to be
   either probabilistic or deterministic.

   This formula ensures that Scalable and Classic flows will converge to
   roughly equal congestion windows, for the worst case of Reno
   congestion control.  This is because the congestion windows of
   Scalable and Classic congestion controls are inversely proportional
   to p_L and sqrt(p_C), respectively.  So squaring p_C in the above
   formula counterbalances the square root that characterizes Reno-
   friendly flows.

      |  Note that, contrary to [RFC3168], an AQM implementing the L4S
      |  and Classic treatments does not mark an ECT(1) packet under the
      |  same conditions that it would have dropped a Not-ECT packet, as
      |  allowed by [RFC8311], which updates [RFC3168].  However, if it
      |  marks ECT(0) packets, it does so under the same conditions that
      |  it would have dropped a Not-ECT packet [RFC3168].

   Also, in the L4S architecture [RFC9330], the sender, not the network,
   is responsible for smoothing out variations in the queue.  So an L4S
   AQM MUST signal congestion as soon as possible.  Then, an L4S sender
   generally interprets CE marking as an unsmoothed signal.

   This requirement does not prevent an L4S AQM from mixing in
   additional congestion signals that are smoothed, such as the signals
   from a Classic smoothed AQM that are coupled with unsmoothed L4S
   signals in the coupled DualQ [RFC9332], but only as long as the onset
   of congestion can be signalled immediately and can be interpreted by
   the sender as if it has been signalled immediately, which is
   important for interoperability

5.3.  Exception for L4S Packet Identification by Network Nodes with
      Transport-Layer Awareness

   To implement L4S packet classification, a network node does not need
   to identify transport-layer flows.  Nonetheless, if an L4S network
   node classifies packets by their transport-layer flow ID and their
   ECN field, and if all the ECT packets in a flow have been ECT(0), the
   node MAY classify any CE packets in the same flow as if they were
   Classic ECT(0) packets.  In all other cases, a network node MUST
   classify all CE packets as if they were ECT(1) packets.  Examples of
   such other cases are: i) if no ECT packets have yet been identified
   in a flow; ii) if it is not desirable for a network node to identify
   transport-layer flows; or iii) if some ECT packets in a flow have
   been ECT(1) (this advice will need to be verified as part of L4S
   experiments).

5.4.  Interaction of the L4S Identifier with Other Identifiers

   The examples in this section concern how additional identifiers might
   complement the L4S identifier to classify packets between class-based
   queues.  Firstly, Section 5.4.1 considers two queues, L4S and
   Classic, as in the DualQ Coupled AQM [RFC9332], either alone
   (Section 5.4.1.1) or within a larger queuing hierarchy
   (Section 5.4.1.2).  Then, Section 5.4.2 considers schemes that might
   combine per-flow 5-tuples with other identifiers.

5.4.1.  DualQ Examples of Other Identifiers Complementing L4S
        Identifiers

5.4.1.1.  Inclusion of Additional Traffic with L4S

   In a typical case for the public Internet, a network element that
   implements L4S in a shared queue might want to classify some low-rate
   but unresponsive traffic (e.g., DNS, LDAP, NTP, voice, and game sync
   packets) into the low-latency queue to mix with L4S traffic.  In this
   case, it would not be appropriate to call the queue an L4S queue,
   because it is shared by L4S and non-L4S traffic.  Instead, it will be
   called the low-latency or L queue.  The L queue then offers two
   different treatments:

   *  the L4S treatment, which is a combination of the L4S AQM treatment
      and a priority scheduling treatment, and

   *  the low-latency treatment, which is solely the priority scheduling
      treatment, without ECN marking by the AQM.

   To identify packets for just the scheduling treatment, it would be
   inappropriate to use the L4S ECT(1) identifier, because such traffic
   is unresponsive to ECN marking.  Examples of relevant non-ECN
   identifiers are:

   *  address ranges of specific applications or hosts configured to be,
      or known to be, safe, e.g., hard-coded Internet of Things (IoT)
      devices sending low-intensity traffic;

   *  certain low data-volume applications or protocols (e.g., ARP and
      DNS); and

   *  specific Diffserv codepoints that indicate traffic with limited
      burstiness such as the EF [RFC3246], VOICE-ADMIT [RFC5865], or
      proposed Non-Queue-Building (NQB) [NQB-PHB] service classes or
      equivalent Local-use DSCPs (see [L4S-DIFFSERV]).

   To be clear, classifying into the L queue based on application-layer
   identification (e.g., DNS) is an example of a local optimization, not
   a recommendation.  Applications will not be able to rely on such
   unsolicited optimization.  A more reliable approach would be for the
   sender to set an appropriate IP-layer identifier, such as one of the
   above Diffserv codepoints.

   In summary, a network element that implements L4S in a shared queue
   MAY classify additional types of packets into the L queue based on
   identifiers other than the IP-ECN field, but the types SHOULD be
   'safe' to mix with L4S traffic, where 'safe' is explained in
   Section 5.4.1.1.1.

   A packet that carries one of these non-ECN identifiers to classify it
   into the L queue would not be subject to the L4S ECN-marking
   treatment, unless it also carried an ECT(1) or CE codepoint.  The
   specification of an L4S AQM MUST define the behaviour for packets
   with unexpected combinations of codepoints, e.g., a non-ECN-based
   classifier for the L queue but with ECT(0) in the IP-ECN field (for
   examples with appropriate behaviours, see Section 2.5.1.1 of the
   DualQ spec [RFC9332]).

   For clarity, non-ECN identifiers, such as the examples itemized
   above, might be used by some network operators who believe they
   identify non-L4S traffic that would be safe to mix with L4S traffic.
   They are not alternative ways for a host to indicate that it is
   sending L4S packets.  Only the ECT(1) ECN codepoint indicates to a
   network element that a host is sending L4S packets (and CE indicates
   that it could have originated as ECT(1)).  Specifically, ECT(1)
   indicates that the host claims its behaviour satisfies the
   prerequisite transport requirements in Section 4.

   In order to include non-L4S packets in the L queue, a network node
   MUST NOT change Not-ECT or ECT(0) in the IP-ECN field into an L4S
   identifier.  This ensures that these codepoints survive for any
   potential use later on the network path.  If a non-compliant or
   malicious network node did swap ECT(0) to ECT(1), the packet could
   subsequently be ECN-marked by a downstream L4S AQM, but the sender
   would respond to congestion indications thinking it had sent a
   Classic packet.  This could result in the flow yielding excessively
   to other L4S flows sharing the downstream bottleneck.

5.4.1.1.1.  'Safe' Unresponsive Traffic

   The above section requires unresponsive traffic to be 'safe' to mix
   with L4S traffic.  Ideally, this means that the sender never sends
   any sequence of packets at a rate that exceeds the available capacity
   of the bottleneck link.  However, typically an unresponsive transport
   does not even know the bottleneck capacity of the path, let alone its
   available capacity.  Nonetheless, an application can be considered
   safe enough if it paces packets out (not necessarily with absolute
   regularity) such that its maximum instantaneous rate from packet to
   packet stays well below a typical broadband access rate.

   This is a vague but useful definition, because many low-latency
   applications of interest, such as DNS, voice, game sync packets, RPC,
   ACKs, and keep-alives, could match this description.

   Low-rate streams, such as voice and game sync packets, might not use
   continuously adapting ECN-based congestion control, but they ought to
   at least use a 'circuit-breaker' style of congestion response
   [RFC8083].  If the volume of traffic from unresponsive applications
   is high enough to overload the link, this will at least protect the
   capacity available to responsive applications.  However, queuing
   delay in the L queue would probably then rise to the typically higher
   level targeted by a Classic (drop-based) AQM.  If a network operator
   considers that such self-restraint is not enough, it might want to
   police the L queue (see Section 8.2 of the L4S architecture
   [RFC9330]).

5.4.1.2.  Exclusion of Traffic from L4S Treatment

   To extend the above example, an operator might want to exclude some
   traffic from the L4S treatment for a policy reason, e.g., security
   (traffic from malicious sources) or commercial (e.g., the operator
   may wish to initially confine the benefits of L4S to business
   customers).

   In this exclusion case, the classifier MUST classify on the relevant
   locally used identifiers (e.g., source addresses) before classifying
   the non-matching traffic on the end-to-end L4S ECN identifier.

   A network node MUST NOT alter the end-to-end L4S ECN identifier from
   L4S to Classic, because an operator decision to exclude certain
   traffic from L4S treatment is local-only.  The end-to-end L4S
   identifier then survives for other operators to use, or indeed, they
   can apply their own policy, independently based on their own choice
   of locally used identifiers.  This approach also allows any operator
   to remove its locally applied exclusions in future, e.g., if it
   wishes to widen the benefit of the L4S treatment to all its
   customers.  If a non-compliant or malicious network node did swap
   ECT(1) to ECT(0), the packet could subsequently be ECN-marked by a
   downstream Classic ECN AQM.  L4S senders are required to detect and
   handle such treatment (see Item 3 in Section 4.3), but that does not
   make this swap OK, because such detection is not known to be perfect
   or immediate.

   A network node that supports L4S but excludes certain packets
   carrying the L4S identifier from L4S treatment MUST still apply
   marking or dropping that is compatible with an L4S congestion
   response.  For instance, it could either drop such packets with the
   same likelihood as Classic packets or ECN-mark them with a likelihood
   appropriate to L4S traffic (e.g., the coupled probability in a DualQ
   Coupled AQM) but aiming for the Classic delay target.  It MUST NOT
   ECN-mark such packets with a Classic marking probability, which could
   confuse the sender.

5.4.1.3.  Generalized Combination of L4S and Other Identifiers

   L4S concerns low latency, which it can provide for all traffic
   without differentiation and without _necessarily_ affecting bandwidth
   allocation.  Diffserv provides for differentiation of both bandwidth
   and low latency, but its control of latency depends on its control of
   bandwidth.  L4S and Diffserv can be combined if a network operator
   wants to control bandwidth allocation but also wants to provide low
   latency, i.e., for any amount of traffic within one of these
   allocations of bandwidth (rather than only providing low latency by
   limiting bandwidth) [L4S-DIFFSERV].

   The DualQ examples so far have been framed in the context of
   providing the default Best Effort Per-Hop Behaviour (PHB) using two
   queues -- a low-latency (L) queue and a Classic (C) queue.  This
   single DualQ structure is expected to be the most common and useful
   arrangement.  But, more generally, an operator might choose to
   control bandwidth allocation through a hierarchy of Diffserv PHBs at
   a node and to offer one (or more) of these PHBs using a pair of
   queues for a low latency and a Classic variant of the PHB.

   In the first case, if we assume that a network element provides no
   PHBs except the DualQ, if a packet carries ECT(1) or CE, the network
   element would classify it for the L4S treatment irrespective of its
   DSCP.  And, if a packet carried (for example) the EF DSCP, the
   network element could classify it into the L queue irrespective of
   its ECN codepoint.  However, where the DualQ is in a hierarchy of
   other PHBs, the classifier would classify some traffic into other
   PHBs based on DSCP before classifying between the low-latency and
   Classic queues (based on ECT(1), CE, and perhaps also the EF DSCP or
   other identifiers as in the above example).  [L4S-DIFFSERV] gives a
   number of examples of such arrangements to address various
   requirements.

   [L4S-DIFFSERV] describes how an operator might use L4S to offer low
   latency as well as Diffserv for bandwidth differentiation.  It
   identifies two main types of approach, which can be combined: the
   operator might split certain Diffserv PHBs between L4S and a
   corresponding Classic service.  Or it might split the L4S and/or the
   Classic service into multiple Diffserv PHBs.  In either of these
   cases, a packet would have to be classified on its Diffserv and ECN
   codepoints.

   In summary, there are numerous ways in which the L4S ECN identifier
   (ECT(1) and CE) could be combined with other identifiers to achieve
   particular objectives.  The following categorization articulates
   those that are valid, but it is not necessarily exhaustive.  Those
   tagged as 'Recommended-standard-use' could be set by the sending host
   or a network.  Those tagged as 'Local-use' would only be set by a
   network:

   1.  Identifiers Complementing the L4S Identifier

       a.  Including More Traffic in the L Queue

           (could use Recommended-standard-use or Local-use identifiers)

       b.  Excluding Certain Traffic from the L Queue

           (Local-use only)

   2.  Identifiers to Place L4S Classification in a PHB Hierarchy

       (could use Recommended-standard-use or Local-use identifiers)

       a.  PHBs before L4S ECN Classification

       b.  PHBs after L4S ECN Classification

5.4.2.  Per-flow Queuing Examples of Other Identifiers Complementing L4S
        Identifiers

   At a node with per-flow queuing (e.g., FQ-CoDel [RFC8290]), the L4S
   identifier could complement the transport-layer flow ID as a further
   level of flow granularity (i.e., Not-ECT and ECT(0) queued separately
   from ECT(1) and CE packets).  In Appendix B, the "Risk of reordering
   Classic CE packets within a flow" discusses the resulting ambiguity
   if packets originally set to ECT(0) are marked CE by an upstream AQM
   before they arrive at a node that classifies CE as L4S.  It argues
   that the risk of reordering is vanishingly small, and the consequence
   of such a low level of reordering is minimal.

   Alternatively, it could be assumed that it is not in a flow's own
   interest to mix Classic and L4S identifiers.  Then, the AQM could use
   the IP-ECN field to switch itself between a Classic and an L4S AQM
   behaviour within one per-flow queue.  For instance, for ECN-capable
   packets, the AQM might consist of a simple marking threshold, and an
   L4S ECN identifier might simply select a shallower threshold than a
   Classic ECN identifier would.

5.5.  Limiting Packet Bursts from Links

   As well as senders needing to limit packet bursts (Section 4.3),
   links need to limit the degree of burstiness they introduce.  In both
   cases (senders and links), this is a trade-off, because batch-
   handling of packets is done for good reason, e.g., for processing
   efficiency or to make efficient use of medium acquisition delay.
   Some take the attitude that there is no point reducing burst delay at
   the sender below that introduced by links (or vice versa).  However,
   delay reduction proceeds by cutting down 'the longest pole in the
   tent', which turns the spotlight on the next longest, and so on.

   This document does not set any quantified requirements for links to
   limit burst delay, primarily because link technologies are outside
   the remit of L4S specifications.  Nonetheless, the following two
   subsections outline opportunities for addressing bursty links in the
   process of L4S implementation and deployment.

5.5.1.  Limiting Packet Bursts from Links Fed by an L4S AQM

   It would not make sense to implement an L4S AQM that feeds into a
   particular link technology without also reviewing opportunities to
   reduce any form of burst delay introduced by that link technology.
   This would at least limit the bursts that the link would otherwise
   introduce into the onward traffic, which would cause jumpy feedback
   to the sender as well as potential extra queuing delay downstream.
   This document does not presume to even give guidance on an
   appropriate target for such burst delay until there is more industry
   experience of L4S.  However, as suggested in Section 4.3, it would
   not seem necessary to limit bursts lower than roughly 10% of the
   minimum base RTT expected in the typical deployment scenario (e.g.,
   250 us burst duration for links within the public Internet).

5.5.2.  Limiting Packet Bursts from Links Upstream of an L4S AQM

   The initial scope of the L4S experiment is to deploy L4S AQMs at
   bottlenecks and L4S congestion controls at senders.  This is expected
   to highlight interactions with the most bursty upstream links and
   lead operators to tune down the burstiness of those links in their
   networks that are configurable or, failing that, to have to
   compromise on the delay target of some L4S AQMs.  It might also
   require specific redesign work relevant to the most problematic link
   types.  Such knock-on effects of initial L4S deployment would all be
   a part of the learning from the L4S experiment.

   The details of such link changes are beyond the scope of the present
   document.  Nonetheless, where L4S technology is being implemented on
   an outgoing interface of a device, it would make sense to consider
   opportunities for reducing bursts arriving at other incoming
   interfaces.  For instance, where an L4S AQM is implemented to feed
   into the upstream WAN interface of a home gateway, there would be
   opportunities to alter the Wi-Fi profiles sent out of any Wi-Fi
   interfaces from the same device, in order to mitigate incoming bursts
   of aggregated Wi-Fi frames from other Wi-Fi stations.

6.  Behaviour of Tunnels and Encapsulations

6.1.  No Change to ECN Tunnels and Encapsulations in General

   The L4S identifier is expected to work through and within any tunnel
   without modification, as long as the tunnel propagates the ECN field
   in any of the ways that have been defined since the first variant in
   the year 2001 [RFC3168].  L4S will also work with (but does not rely
   on) any of the more recent updates to ECN propagation in [RFC4301],
   [RFC6040], or [ECN-SHIM].  However, it is likely that some tunnels
   still do not implement ECN propagation at all.  In these cases, L4S
   will work through such tunnels, but within them the outer header of
   L4S traffic will appear as Classic.

   AQMs are typically implemented where an IP-layer buffer feeds into a
   lower layer, so they are agnostic to link-layer encapsulations.
   Where a bottleneck link is not IP-aware, the L4S identifier is still
   expected to work within any lower-layer encapsulation without
   modification, as long it propagates the IP-ECN field as defined for
   the link technology, for example, for MPLS [RFC5129] or Transparent
   Interconnection of Lots of Links (TRILL) [TRILL-ECN-SUPPORT].  In
   some of these cases, e.g., Layer 3 Ethernet switches, the AQM
   accesses the IP-layer header within the outer encapsulation, so again
   the L4S identifier is expected to work without modification.
   Nonetheless, the programme to define ECN for other lower layers is
   still in progress [ECN-ENCAP].

6.2.  VPN Behaviour to Avoid Limitations of Anti-Replay

   If a mix of L4S and Classic packets is sent into the same security
   association (SA) of a VPN, and if the VPN egress is employing the
   optional anti-replay feature, it could inappropriately discard
   Classic packets (or discard the records in Classic packets) by
   mistaking their greater queuing delay for a replay attack (see
   "Dropped Packets for Tunnels with Replay Protection Enabled" in
   [Heist21] for the potential performance impact).  This known problem
   is common to both IPsec [RFC4301] and DTLS [RFC9147] VPNs, given they
   use similar anti-replay window mechanisms.  The mechanism used can
   only check for replay within its window, so if the window is smaller
   than the degree of reordering, it can only assume there might be a
   replay attack and discard all the packets behind the trailing edge of
   the window.  The specifications of IPsec Authentication Header (AH)
   [RFC4302] and Encapsulating Security Payload (ESP) [RFC4303] suggest
   that an implementer scales the size of the anti-replay window with
   interface speed, and DTLS v1.3 [RFC9147] states that "The receiver
   SHOULD pick a window large enough to handle any plausible reordering,
   which depends on the data rate."  However, in practice, the size of a
   VPN's anti-replay window is not always scaled appropriately.

   If a VPN carrying traffic participating in the L4S experiment
   experiences inappropriate replay detection, the foremost remedy would
   be to ensure that the egress is configured to comply with the above
   window-sizing requirements.

   If an implementation of a VPN egress does not support a sufficiently
   large anti-replay window, e.g., due to hardware limitations, one of
   the temporary alternatives listed in order of preference below might
   be feasible instead:

   *  If the VPN can be configured to classify packets into different
      SAs indexed by DSCP, apply the appropriate locally defined DSCPs
      to Classic and L4S packets.  The DSCPs could be applied by the
      network (based on the least-significant bit of the IP-ECN field),
      or by the sending host.  Such DSCPs would only need to survive as
      far as the VPN ingress.

   *  If the above is not possible and it is necessary to use L4S,
      either of the following might be appropriate as a last resort:

      -  disable anti-replay protection at the VPN egress, after
         considering the security implications (it is mandatory to allow
         the anti-replay facility to be disabled in both IPsec and
         DTLS), or

      -  configure the tunnel ingress not to propagate ECN to the outer,
         which would lose the benefits of L4S and Classic ECN over the
         VPN.

   Modification to VPN implementations is outside the present scope,
   which is why this section has so far focused on reconfiguration.
   Although this document does not define any requirements for VPN
   implementations, determining whether there is a need for such
   requirements could be one aspect of L4S experimentation.

7.  L4S Experiments

   This section describes open questions that L4S experiments ought to
   focus on.  This section also documents outstanding open issues that
   will need to be investigated as part of L4S experimentation, given
   they could not be fully resolved during the working group phase.  It
   also lists metrics that will need to be monitored during experiments
   (summarizing text elsewhere in L4S documents) and finally lists some
   potential future directions that researchers might wish to
   investigate.

   In addition to this section, i) the DualQ spec [RFC9332] sets
   operational and management requirements for experiments with DualQ
   Coupled AQMs, and ii) general operational and management requirements
   for experiments with L4S congestion controls are given in Sections 4
   and 5 above, e.g., coexistence and scaling requirements and
   incremental deployment arrangements.

   The specification of each Scalable congestion control will need to
   include protocol-specific requirements for configuration and
   monitoring performance during experiments.  Appendix A of [RFC5706]
   provides a helpful checklist.

7.1.  Open Questions

   L4S experiments would be expected to answer the following questions:

   *  Have all the parts of L4S been deployed, and if so, what
      proportion of paths support it?

      -  What types of L4S AQMs were deployed, e.g., FQ, coupled DualQ,
         uncoupled DualQ, other?  And how prevalent was each?

      -  Are the signalling patterns emitted by the deployed AQMs in any
         way different from those expected when the Prague requirements
         for endpoints were written?

   *  Does use of L4S over the Internet result in a significantly
      improved user experience?

   *  Has L4S enabled novel interactive applications?

   *  Did use of L4S over the Internet result in improvements to the
      following metrics:

      -  queue delay (mean and 99th percentile) under various loads;

      -  utilization;

      -  starvation / fairness; and

      -  scaling range of flow rates and RTTs?

   *  How dependent was the performance of L4S service on the bottleneck
      bandwidth or the path RTT?

   *  How much do bursty links in the Internet affect L4S performance
      (see "Underutilization with Bursty Links" in [Heist21]) and how
      prevalent are they?  How much limitation of burstiness from
      upstream links was needed and/or was realized -- both at senders
      and at links, especially radio links -- or how much did L4S target
      delay have to be increased to accommodate the bursts (see Item 7
      in Section 4.3 and see Section 5.5.2)?

   *  Is the initial experiment with mis-identified bursty traffic at
      high RTT (see "Underutilization with Bursty Traffic" in [Heist21])
      indicative of similar problems at lower RTTs, and if so, how
      effective is the suggested remedy in Appendix A.1 of the DualQ
      spec [RFC9332] (or possible other remedies)?

   *  Was per-flow queue protection typically (un)necessary?

      -  How well did overload protection or queue protection work?

   *  How well did L4S flows coexist with Classic flows when sharing a
      bottleneck?

      -  How frequently did problems arise?

      -  What caused any coexistence problems, and were any problems due
         to single-queue Classic ECN AQMs (this assumes single-queue
         Classic ECN AQMs can be distinguished from FQ ones)?

   *  How prevalent were problems with the L4S service due to tunnels/
      encapsulations that do not support ECN decapsulation?

   *  How easy was it to implement a fully compliant L4S congestion
      control, over various different transport protocols (TCP, QUIC,
      RMCAT, etc.)?

   Monitoring for harm to other traffic, specifically bandwidth
   starvation or excess queuing delay, will need to be conducted
   alongside all early L4S experiments.  It is hard, if not impossible,
   for an individual flow to measure its impact on other traffic.  So
   such monitoring will need to be conducted using bespoke monitoring
   across flows and/or across classes of traffic.

7.2.  Open Issues

   *  What is the best way forward to deal with L4S over single-queue
      Classic ECN AQM bottlenecks, given current problems with
      misdetecting L4S AQMs as Classic ECN AQMs?  See the L4S
      operational guidance [L4SOPS].

   *  Fixing the poor interaction between current L4S congestion
      controls and CoDel with only Classic ECN support during flow
      startup.  Originally, this was due to a bug in the initialization
      of the congestion average in the Linux implementation of TCP
      Prague.  That was quickly fixed, which removed the main
      performance impact, but further improvement would be useful (by
      modifying either CoDel or Scalable congestion controls, or both).

7.3.  Future Potential

   Researchers might find that L4S opens up the following interesting
   areas for investigation:

   *  potential for faster convergence time and tracking of available
      capacity;

   *  potential for improvements to particular link technologies and
      cross-layer interactions with them;

   *  potential for using virtual queues, e.g., to further reduce
      latency jitter or to leave headroom for capacity variation in
      radio networks;

   *  development and specification of reverse path congestion control
      using L4S building blocks (e.g., AccECN or QUIC);

   *  once queuing delay is cut down, what becomes the 'second-longest
      pole in the tent' (other than the speed of light)?

   *  novel alternatives to the existing set of L4S AQMs; and

   *  novel applications enabled by L4S.

8.  IANA Considerations

   The semantics of the 01 codepoint of the ECN field of the IP header
   are specified by this Experimental RFC.  The process for an
   Experimental RFC to assign this codepoint in the IP header (v4 and
   v6) is documented in Proposed Standard [RFC8311], which updates the
   Proposed Standard [RFC3168].

   IANA has updated the 01 entry in the "ECN Field (Bits 6-7)" registry
   (see <https://www.iana.org/assignments/dscp-registry/>) as follows:

         +========+=====================+=======================+
         | Binary | Keyword             | Reference             |
         +========+=====================+=======================+
         | 01     | ECT(1) (ECN-Capable | [RFC8311] [RFC Errata |
         |        | Transport(1))[1]    | 5399] RFC 9331        |
         +--------+---------------------+-----------------------+

                  Table 1: ECN Field (Bits 6-7) Registry

   [1] ECT(1) is for experimental use only [RFC8311], Section 4.2

9.  Security Considerations

   Approaches to assure the integrity of signals using the new
   identifier are introduced in Appendix C.1.  See the security
   considerations in the L4S architecture [RFC9330] for further
   discussion of misuse of the identifier, as well as extensive
   discussion of policing rate and latency in regard to L4S.

   Defining ECT(1) as the L4S identifier introduces a difference between
   the effects of ECT(0) and ECT(1) that [RFC3168] previously defined as
   distinct but with equivalent effect.  For L4S ECN, a network node is
   still required not to swap one to the other, even if the network
   operator chooses to locally apply the treatment associated with the
   opposite codepoint (see Sections 5.4.1.1 and 5.4.1.2).  These
   sections also describe the potential effects if a non-compliant or
   malicious network node does swap one to the other.  The present
   specification does not change the effects of other unexpected
   transitions of the IP-ECN field, which are still as described in
   Section 18 of [RFC3168].

   If the anti-replay window of a VPN egress is too small, it will
   mistake deliberate delay differences as a replay attack and discard
   higher-delay packets (e.g., Classic) carried within the same security
   association (SA) as low-delay packets (e.g., L4S).  Section 6.2
   recommends that VPNs used in L4S experiments are configured with a
   sufficiently large anti-replay window, as required by the relevant
   specifications.  It also discusses other alternatives.

   If a user taking part in the L4S experiment sets up a VPN without
   being aware of the above advice, and if the user allows anyone to
   send traffic into their VPN, they would open up a DoS vulnerability
   in which an attacker could induce the VPN's anti-replay mechanism to
   discard enough of the user's Classic (C) traffic (if they are
   receiving any) to cause a significant rate reduction.  While the user
   is actively downloading C traffic, the attacker sends C traffic into
   the VPN to fill the remainder of the bottleneck link, then sends
   intermittent L4S packets to maximize the chance of exceeding the
   VPN's replay window.  The user can prevent this attack by following
   the recommendations in Section 6.2.

   The recommendation to detect loss in time units prevents the ACK-
   splitting attacks described in [Savage-TCP].

10.  References

10.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC3168]  Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
              of Explicit Congestion Notification (ECN) to IP",
              RFC 3168, DOI 10.17487/RFC3168, September 2001,
              <https://www.rfc-editor.org/info/rfc3168>.

   [RFC4774]  Floyd, S., "Specifying Alternate Semantics for the
              Explicit Congestion Notification (ECN) Field", BCP 124,
              RFC 4774, DOI 10.17487/RFC4774, November 2006,
              <https://www.rfc-editor.org/info/rfc4774>.

   [RFC6679]  Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P.,
              and K. Carlberg, "Explicit Congestion Notification (ECN)
              for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August
              2012, <https://www.rfc-editor.org/info/rfc6679>.

10.2.  Informative References

   [A2DTCP]   Zhang, T., Wang, J., Huang, J., Huang, Y., Chen, J., and
              Y. Pan, "Adaptive-Acceleration Data Center TCP", IEEE
              Transactions on Computers, Volume 64, Issue 6, pp.
              1522-1533, DOI 10.1109/TC.2014.2345393, June 2015,
              <https://ieeexplore.ieee.org/xpl/
              articleDetails.jsp?arnumber=6871352>.

   [ACCECN]   Briscoe, B., Kühlewind, M., and R. Scheffenegger, "More
              Accurate ECN Feedback in TCP", Work in Progress, Internet-
              Draft, draft-ietf-tcpm-accurate-ecn-22, 9 November 2022,
              <https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-
              accurate-ecn-22>.

   [Ahmed19]  Ahmed, A.S., "Extending TCP for Low Round Trip Delay",
              Master's Thesis, University of Oslo, August 2019,
              <https://www.duo.uio.no/handle/10852/70966>.

   [Alizadeh-stability]
              Alizadeh, M., Javanmard, A., and B. Prabhakar, "Analysis
              of DCTCP: Stability, Convergence, and Fairness",
              SIGMETRICS '11: Proceedings of the ACM SIGMETRICS Joint
              International Conference on Measurement and Modeling of
              Computer Systems, pp. 73-84, DOI 10.1145/1993744.1993753,
              June 2011,
              <https://dl.acm.org/doi/10.1145/1993744.1993753>.

   [ARED01]   Floyd, S., Gummadi, R., and S. Shenker, "Adaptive RED: An
              Algorithm for Increasing the Robustness of RED's Active
              Queue Management", ACIRI Technical Report 301, August
              2001, <https://www.icsi.berkeley.edu/icsi/node/2032>.

   [BBR-CC]   Cardwell, N., Cheng, Y., Hassas Yeganeh, S., Swett, I.,
              and V. Jacobson, "BBR Congestion Control", Work in
              Progress, Internet-Draft, draft-cardwell-iccrg-bbr-
              congestion-control-02, 7 March 2022,
              <https://datatracker.ietf.org/doc/html/draft-cardwell-
              iccrg-bbr-congestion-control-02>.

   [BBRv2]    "TCP BBR v2 Alpha/Preview Release", commit 17700ca, June
              2022, <https://github.com/google/bbr>.

   [Bufferbloat]
              The Bufferbloat community, "Bufferbloat",
              <https://bufferbloat.net/>.

   [COBALT]   Palmei, J., Gupta, S., Imputato, P., Morton, J.,
              Tahiliani, M. P., Avallone, S., and D. Täht, "Design and
              Evaluation of COBALT Queue Discipline", IEEE International
              Symposium on Local and Metropolitan Area Networks
              (LANMAN), DOI 10.1109/LANMAN.2019.8847054, July 2019,
              <https://ieeexplore.ieee.org/abstract/document/8847054>.

   [CTCP]     Sridharan, M., Tan, K., Bansal, D., and D. Thaler,
              "Compound TCP: A New TCP Congestion Control for High-Speed
              and Long Distance Networks", Work in Progress, Internet-
              Draft, draft-sridharan-tcpm-ctcp-02, 3 November 2008,
              <https://datatracker.ietf.org/doc/html/draft-sridharan-
              tcpm-ctcp-02>.

   [DCttH19]  De Schepper, K., Bondarenko, O., Tilmans, O., and B.
              Briscoe, "'Data Centre to the Home': Ultra-Low Latency for
              All", Updated RITE project Technical Report, July 2019,
              <https://bobbriscoe.net/projects/latency/
              dctth_journal_draft20190726.pdf>.

   [DOCSIS-QPROT]
              Briscoe, B., Ed. and G. White, "The DOCSIS® Queue
              Protection Algorithm to Preserve Low Latency", Work in
              Progress, Internet-Draft, draft-briscoe-docsis-q-
              protection-06, 13 May 2022,
              <https://datatracker.ietf.org/doc/html/draft-briscoe-
              docsis-q-protection-06>.

   [DualPI2Linux]
              Albisser, O., De Schepper, K., Briscoe, B., Tilmans, O.,
              and H. Steen, "DUALPI2 - Low Latency, Low Loss and
              Scalable (L4S) AQM", Proceedings of Linux Netdev 0x13,
              March 2019, <https://www.netdevconf.org/0x13/
              session.html?talk-DUALPI2-AQM>.

   [Dukkipati06]
              Dukkipati, N. and N. McKeown, "Why Flow-Completion Time is
              the Right Metric for Congestion Control", ACM SIGCOMM
              Computer Communication Review, Volume 36, Issue 1, pp.
              59-62, DOI 10.1145/1111322.1111336, January 2006,
              <https://dl.acm.org/doi/10.1145/1111322.1111336>.

   [ECN++]    Bagnulo, M. and B. Briscoe, "ECN++: Adding Explicit
              Congestion Notification (ECN) to TCP Control Packets",
              Work in Progress, Internet-Draft, draft-ietf-tcpm-
              generalized-ecn-10, 27 July 2022,
              <https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-
              generalized-ecn-10>.

   [ECN-ENCAP]
              Briscoe, B. and J. Kaippallimalil, "Guidelines for Adding
              Congestion Notification to Protocols that Encapsulate IP",
              Work in Progress, Internet-Draft, draft-ietf-tsvwg-ecn-
              encap-guidelines-17, 11 July 2022,
              <https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-
              ecn-encap-guidelines-17>.

   [ecn-fallback]
              Briscoe, B. and A. Ahmed, "TCP Prague Fall-back on
              Detection of a Classic ECN AQM", Technical Report: TR-BB-
              2019-002, DOI 10.48550/arXiv.1911.00710, February 2021,
              <https://arxiv.org/abs/1911.00710>.

   [ECN-SHIM] Briscoe, B., "Propagating Explicit Congestion Notification
              Across IP Tunnel Headers Separated by a Shim", Work in
              Progress, Internet-Draft, draft-ietf-tsvwg-rfc6040update-
              shim-15, 11 July 2022,
              <https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-
              rfc6040update-shim-15>.

   [Heist21]  "L4S Tests", commit e21cd91, August 2021,
              <https://github.com/heistp/l4s-tests>.

   [L4S-DIFFSERV]
              Briscoe, B., "Interactions between Low Latency, Low Loss,
              Scalable Throughput (L4S) and Differentiated Services",
              Work in Progress, Internet-Draft, draft-briscoe-tsvwg-l4s-
              diffserv-02, 1 November 2018,
              <https://datatracker.ietf.org/doc/html/draft-briscoe-
              tsvwg-l4s-diffserv-02>.

   [L4Seval22]
              De Schepper, K., Albisser, O., Tilmans, O., and B.
              Briscoe, "Dual Queue Coupled AQM: Deployable Very Low
              Queuing Delay for All", Preprint submitted to IEEE/ACM
              Transactions on Networking, DOI 10.48550/arXiv.2209.01078,
              September 2022, <https://arxiv.org/abs/2209.01078>.

   [L4SOPS]   White, G., Ed., "Operational Guidance for Deployment of
              L4S in the Internet", Work in Progress, Internet-Draft,
              draft-ietf-tsvwg-l4sops-03, 28 April 2022,
              <https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-
              l4sops-03>.

   [LinuxPacedChirping]
              Misund, J. and B. Briscoe, "Paced Chirping - Rethinking
              TCP start-up", Proceedings of Linux Netdev 0x13, March
              2019, <https://legacy.netdevconf.info/0x13/
              session.html?talk-chirp>.

   [NQB-PHB]  White, G. and T. Fossati, "A Non-Queue-Building Per-Hop
              Behavior (NQB PHB) for Differentiated Services", Work in
              Progress, Internet-Draft, draft-ietf-tsvwg-nqb-15, 11
              January 2023, <https://datatracker.ietf.org/doc/html/
              draft-ietf-tsvwg-nqb-15>.

   [PI2]      De Schepper, K., Bondarenko, O., Tsang, I., and B.
              Briscoe, "PI^2: A Linearized AQM for both Classic and
              Scalable TCP", Proceedings of ACM CoNEXT 2016, pp.
              105-119, DOI 10.1145/2999572.2999578, December 2016,
              <https://dl.acm.org/citation.cfm?doid=2999572.2999578>.

   [PRAGUE-CC]
              De Schepper, K., Tilmans, O., and B. Briscoe, Ed., "Prague
              Congestion Control", Work in Progress, Internet-Draft,
              draft-briscoe-iccrg-prague-congestion-control-01, 11 July
              2022, <https://datatracker.ietf.org/doc/html/draft-
              briscoe-iccrg-prague-congestion-control-01>.

   [PragueLinux]
              Briscoe, B., De Schepper, K., Albisser, O., Misund, J.,
              Tilmans, O., Kühlewind, M., and A. Ahmed, "Implementing
              the 'TCP Prague' Requirements for L4S", Proceedings of
              Linux Netdev 0x13, March 2019,
              <https://www.netdevconf.org/0x13/session.html?talk-tcp-
              prague-l4s>.

   [QV]       Briscoe, B. and P. Hurtig, "Report on Prototype
              Development and Evaluation of Network and Interaction
              Techniques", RITE Technical Report, Deliverable 2.3,
              Appendix C.2: "Up to Speed with Queue View", September
              2015, <https://riteproject.files.wordpress.com/2015/12/
              rite-deliverable-2-3.pdf>.

   [RELENTLESS]
              Mathis, M., "Relentless Congestion Control", Work in
              Progress, Internet-Draft, draft-mathis-iccrg-relentless-
              tcp-00, 4 March 2009,
              <https://datatracker.ietf.org/doc/html/draft-mathis-iccrg-
              relentless-tcp-00>.

   [RFC2474]  Nichols, K., Blake, S., Baker, F., and D. Black,
              "Definition of the Differentiated Services Field (DS
              Field) in the IPv4 and IPv6 Headers", RFC 2474,
              DOI 10.17487/RFC2474, December 1998,
              <https://www.rfc-editor.org/info/rfc2474>.

   [RFC3246]  Davie, B., Charny, A., Bennet, J.C.R., Benson, K., Le
              Boudec, J.Y., Courtney, W., Davari, S., Firoiu, V., and D.
              Stiliadis, "An Expedited Forwarding PHB (Per-Hop
              Behavior)", RFC 3246, DOI 10.17487/RFC3246, March 2002,
              <https://www.rfc-editor.org/info/rfc3246>.

   [RFC3540]  Spring, N., Wetherall, D., and D. Ely, "Robust Explicit
              Congestion Notification (ECN) Signaling with Nonces",
              RFC 3540, DOI 10.17487/RFC3540, June 2003,
              <https://www.rfc-editor.org/info/rfc3540>.

   [RFC3649]  Floyd, S., "HighSpeed TCP for Large Congestion Windows",
              RFC 3649, DOI 10.17487/RFC3649, December 2003,
              <https://www.rfc-editor.org/info/rfc3649>.

   [RFC4301]  Kent, S. and K. Seo, "Security Architecture for the
              Internet Protocol", RFC 4301, DOI 10.17487/RFC4301,
              December 2005, <https://www.rfc-editor.org/info/rfc4301>.

   [RFC4302]  Kent, S., "IP Authentication Header", RFC 4302,
              DOI 10.17487/RFC4302, December 2005,
              <https://www.rfc-editor.org/info/rfc4302>.

   [RFC4303]  Kent, S., "IP Encapsulating Security Payload (ESP)",
              RFC 4303, DOI 10.17487/RFC4303, December 2005,
              <https://www.rfc-editor.org/info/rfc4303>.

   [RFC4340]  Kohler, E., Handley, M., and S. Floyd, "Datagram
              Congestion Control Protocol (DCCP)", RFC 4340,
              DOI 10.17487/RFC4340, March 2006,
              <https://www.rfc-editor.org/info/rfc4340>.

   [RFC4341]  Floyd, S. and E. Kohler, "Profile for Datagram Congestion
              Control Protocol (DCCP) Congestion Control ID 2: TCP-like
              Congestion Control", RFC 4341, DOI 10.17487/RFC4341, March
              2006, <https://www.rfc-editor.org/info/rfc4341>.

   [RFC4342]  Floyd, S., Kohler, E., and J. Padhye, "Profile for
              Datagram Congestion Control Protocol (DCCP) Congestion
              Control ID 3: TCP-Friendly Rate Control (TFRC)", RFC 4342,
              DOI 10.17487/RFC4342, March 2006,
              <https://www.rfc-editor.org/info/rfc4342>.

   [RFC4960]  Stewart, R., Ed., "Stream Control Transmission Protocol",
              RFC 4960, DOI 10.17487/RFC4960, September 2007,
              <https://www.rfc-editor.org/info/rfc4960>.

   [RFC5033]  Floyd, S. and M. Allman, "Specifying New Congestion
              Control Algorithms", BCP 133, RFC 5033,
              DOI 10.17487/RFC5033, August 2007,
              <https://www.rfc-editor.org/info/rfc5033>.

   [RFC5129]  Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion
              Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January
              2008, <https://www.rfc-editor.org/info/rfc5129>.

   [RFC5348]  Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP
              Friendly Rate Control (TFRC): Protocol Specification",
              RFC 5348, DOI 10.17487/RFC5348, September 2008,
              <https://www.rfc-editor.org/info/rfc5348>.

   [RFC5562]  Kuzmanovic, A., Mondal, A., Floyd, S., and K.
              Ramakrishnan, "Adding Explicit Congestion Notification
              (ECN) Capability to TCP's SYN/ACK Packets", RFC 5562,
              DOI 10.17487/RFC5562, June 2009,
              <https://www.rfc-editor.org/info/rfc5562>.

   [RFC5622]  Floyd, S. and E. Kohler, "Profile for Datagram Congestion
              Control Protocol (DCCP) Congestion ID 4: TCP-Friendly Rate
              Control for Small Packets (TFRC-SP)", RFC 5622,
              DOI 10.17487/RFC5622, August 2009,
              <https://www.rfc-editor.org/info/rfc5622>.

   [RFC5681]  Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
              Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
              <https://www.rfc-editor.org/info/rfc5681>.

   [RFC5706]  Harrington, D., "Guidelines for Considering Operations and
              Management of New Protocols and Protocol Extensions",
              RFC 5706, DOI 10.17487/RFC5706, November 2009,
              <https://www.rfc-editor.org/info/rfc5706>.

   [RFC5865]  Baker, F., Polk, J., and M. Dolly, "A Differentiated
              Services Code Point (DSCP) for Capacity-Admitted Traffic",
              RFC 5865, DOI 10.17487/RFC5865, May 2010,
              <https://www.rfc-editor.org/info/rfc5865>.

   [RFC5925]  Touch, J., Mankin, A., and R. Bonica, "The TCP
              Authentication Option", RFC 5925, DOI 10.17487/RFC5925,
              June 2010, <https://www.rfc-editor.org/info/rfc5925>.

   [RFC6040]  Briscoe, B., "Tunnelling of Explicit Congestion
              Notification", RFC 6040, DOI 10.17487/RFC6040, November
              2010, <https://www.rfc-editor.org/info/rfc6040>.

   [RFC6077]  Papadimitriou, D., Ed., Welzl, M., Scharf, M., and B.
              Briscoe, "Open Research Issues in Internet Congestion
              Control", RFC 6077, DOI 10.17487/RFC6077, February 2011,
              <https://www.rfc-editor.org/info/rfc6077>.

   [RFC6660]  Briscoe, B., Moncaster, T., and M. Menth, "Encoding Three
              Pre-Congestion Notification (PCN) States in the IP Header
              Using a Single Diffserv Codepoint (DSCP)", RFC 6660,
              DOI 10.17487/RFC6660, July 2012,
              <https://www.rfc-editor.org/info/rfc6660>.

   [RFC6675]  Blanton, E., Allman, M., Wang, L., Jarvinen, I., Kojo, M.,
              and Y. Nishida, "A Conservative Loss Recovery Algorithm
              Based on Selective Acknowledgment (SACK) for TCP",
              RFC 6675, DOI 10.17487/RFC6675, August 2012,
              <https://www.rfc-editor.org/info/rfc6675>.

   [RFC7560]  Kuehlewind, M., Ed., Scheffenegger, R., and B. Briscoe,
              "Problem Statement and Requirements for Increased Accuracy
              in Explicit Congestion Notification (ECN) Feedback",
              RFC 7560, DOI 10.17487/RFC7560, August 2015,
              <https://www.rfc-editor.org/info/rfc7560>.

   [RFC7567]  Baker, F., Ed. and G. Fairhurst, Ed., "IETF
              Recommendations Regarding Active Queue Management",
              BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015,
              <https://www.rfc-editor.org/info/rfc7567>.

   [RFC7713]  Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx)
              Concepts, Abstract Mechanism, and Requirements", RFC 7713,
              DOI 10.17487/RFC7713, December 2015,
              <https://www.rfc-editor.org/info/rfc7713>.

   [RFC8033]  Pan, R., Natarajan, P., Baker, F., and G. White,
              "Proportional Integral Controller Enhanced (PIE): A
              Lightweight Control Scheme to Address the Bufferbloat
              Problem", RFC 8033, DOI 10.17487/RFC8033, February 2017,
              <https://www.rfc-editor.org/info/rfc8033>.

   [RFC8083]  Perkins, C. and V. Singh, "Multimedia Congestion Control:
              Circuit Breakers for Unicast RTP Sessions", RFC 8083,
              DOI 10.17487/RFC8083, March 2017,
              <https://www.rfc-editor.org/info/rfc8083>.

   [RFC8085]  Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage
              Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085,
              March 2017, <https://www.rfc-editor.org/info/rfc8085>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC8257]  Bensley, S., Thaler, D., Balasubramanian, P., Eggert, L.,
              and G. Judd, "Data Center TCP (DCTCP): TCP Congestion
              Control for Data Centers", RFC 8257, DOI 10.17487/RFC8257,
              October 2017, <https://www.rfc-editor.org/info/rfc8257>.

   [RFC8290]  Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys,
              J., and E. Dumazet, "The Flow Queue CoDel Packet Scheduler
              and Active Queue Management Algorithm", RFC 8290,
              DOI 10.17487/RFC8290, January 2018,
              <https://www.rfc-editor.org/info/rfc8290>.

   [RFC8298]  Johansson, I. and Z. Sarker, "Self-Clocked Rate Adaptation
              for Multimedia", RFC 8298, DOI 10.17487/RFC8298, December
              2017, <https://www.rfc-editor.org/info/rfc8298>.

   [RFC8311]  Black, D., "Relaxing Restrictions on Explicit Congestion
              Notification (ECN) Experimentation", RFC 8311,
              DOI 10.17487/RFC8311, January 2018,
              <https://www.rfc-editor.org/info/rfc8311>.

   [RFC8312]  Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and
              R. Scheffenegger, "CUBIC for Fast Long-Distance Networks",
              RFC 8312, DOI 10.17487/RFC8312, February 2018,
              <https://www.rfc-editor.org/info/rfc8312>.

   [RFC8511]  Khademi, N., Welzl, M., Armitage, G., and G. Fairhurst,
              "TCP Alternative Backoff with ECN (ABE)", RFC 8511,
              DOI 10.17487/RFC8511, December 2018,
              <https://www.rfc-editor.org/info/rfc8511>.

   [RFC8888]  Sarker, Z., Perkins, C., Singh, V., and M. Ramalho, "RTP
              Control Protocol (RTCP) Feedback for Congestion Control",
              RFC 8888, DOI 10.17487/RFC8888, January 2021,
              <https://www.rfc-editor.org/info/rfc8888>.

   [RFC8985]  Cheng, Y., Cardwell, N., Dukkipati, N., and P. Jha, "The
              RACK-TLP Loss Detection Algorithm for TCP", RFC 8985,
              DOI 10.17487/RFC8985, February 2021,
              <https://www.rfc-editor.org/info/rfc8985>.

   [RFC9000]  Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based
              Multiplexed and Secure Transport", RFC 9000,
              DOI 10.17487/RFC9000, May 2021,
              <https://www.rfc-editor.org/info/rfc9000>.

   [RFC9001]  Thomson, M., Ed. and S. Turner, Ed., "Using TLS to Secure
              QUIC", RFC 9001, DOI 10.17487/RFC9001, May 2021,
              <https://www.rfc-editor.org/info/rfc9001>.

   [RFC9147]  Rescorla, E., Tschofenig, H., and N. Modadugu, "The
              Datagram Transport Layer Security (DTLS) Protocol Version
              1.3", RFC 9147, DOI 10.17487/RFC9147, April 2022,
              <https://www.rfc-editor.org/info/rfc9147>.

   [RFC9330]  Briscoe, B., Ed., De Schepper, K., Bagnulo, M., and G.
              White, "Low Latency, Low Loss, and Scalable Throughput
              (L4S) Internet Service: Architecture", RFC 9330,
              DOI 10.17487/RFC9330, January 2023,
              <https://www.rfc-editor.org/info/rfc9330>.

   [RFC9332]  De Schepper, K., Briscoe, B., Ed., and G. White, "Dual-
              Queue Coupled Active Queue Management (AQM) for Low
              Latency, Low Loss, and Scalable Throughput (L4S)",
              RFC 9332, DOI 10.17487/RFC9332, January 2023,
              <https://www.rfc-editor.org/info/rfc9332>.

   [Savage-TCP]
              Savage, S., Cardwell, N., Wetherall, D., and T. Anderson,
              "TCP Congestion Control with a Misbehaving Receiver", ACM
              SIGCOMM Computer Communication Review, Volume 29, Issue 5,
              pp. 71–78, DOI 10.1145/505696.505704, October 1999,
              <https://dl.acm.org/doi/abs/10.1145/505696.505704>.

   [SCReAM-L4S]
              "SCReAM", commit 140e292, November 2022,
              <https://github.com/EricssonResearch/scream>.

   [SCTP-ECN] Stewart, R., Tüxen, M., and X. Dong, "ECN for Stream
              Control Transmission Protocol (SCTP)", Work in Progress,
              Internet-Draft, draft-stewart-tsvwg-sctpecn-05, 15 January
              2014, <https://datatracker.ietf.org/doc/html/draft-
              stewart-tsvwg-sctpecn-05>.

   [sub-mss-prob]
              Briscoe, B. and K. De Schepper, "Scaling TCP's Congestion
              Window for Small Round Trip Times", BT Technical Report:
              TR-TUB8-2015-002, DOI 10.48550/arXiv.1904.07598, May 2015,
              <https://arxiv.org/abs/1904.07598>.

   [TCP-CA]   Jacobson, V. and M. J. Karels, "Congestion Avoidance and
              Control", Laurence Berkeley Labs Technical Report,
              November 1988, <https://ee.lbl.gov/papers/congavoid.pdf>.

   [TCPPrague]
              Briscoe, B., "Notes: DCTCP evolution 'bar BoF': Tue 21 Jul
              2015, 17:40, Prague", message to the tcpPrague mailing
              list, July 2015, <https://www.ietf.org/mail-
              archive/web/tcpprague/current/msg00001.html>.

   [TRILL-ECN-SUPPORT]
              Eastlake 3rd, D. and B. Briscoe, "TRILL (TRansparent
              Interconnection of Lots of Links): ECN (Explicit
              Congestion Notification) Support", Work in Progress,
              Internet-Draft, draft-ietf-trill-ecn-support-07, 25
              February 2018, <https://datatracker.ietf.org/doc/html/
              draft-ietf-trill-ecn-support-07>.

   [VCP]      Xia, Y., Subramanian, L., Stoica, I., and S. Kalyanaraman,
              "One more bit is enough", SIGCOMM '05: Proceedings of the
              2005 conference on Applications, technologies,
              architectures, and protocols for computer communications,
              pp. 37-48, DOI 10.1145/1080091.1080098, August 2005,
              <https://doi.acm.org/10.1145/1080091.1080098>.

Appendix A.  Rationale for the 'Prague L4S Requirements'

   This appendix is informative, not normative.  It gives a list of
   modifications to current Scalable congestion controls so that they
   can be deployed over the public Internet and coexist safely with
   existing traffic.  The list complements the normative requirements in
   Section 4 that a sender has to comply with before it can set the L4S
   identifier in packets it sends into the Internet.  As well as
   rationale for safety improvements (the requirements in Section 4),
   this appendix also includes preferable performance improvements
   (optimizations).

   The requirements and recommendations in Section 4 have become known
   as the 'Prague L4S Requirements', because they were originally
   identified at an ad hoc meeting during IETF 94 in Prague [TCPPrague].
   They were originally called the 'TCP Prague Requirements', but they
   are not solely applicable to TCP, so the name and wording has been
   generalized for all transport protocols, and the name 'TCP Prague' is
   now used for a specific implementation of the requirements.

   At the time of writing, DCTCP [RFC8257] is the most widely used
   Scalable transport protocol.  In its current form, DCTCP is specified
   to be deployable only in controlled environments.  Deploying it in
   the public Internet would lead to a number of issues, from both the
   safety and the performance perspective.  The modifications and
   additional mechanisms listed in this section will be necessary for
   its deployment over the global Internet.  Where an example is needed,
   DCTCP is used as a base, but the requirements in Section 4 apply
   equally to other Scalable congestion controls, covering adaptive
   real-time media, etc., not just capacity-seeking behaviours.

A.1.  Rationale for the Requirements for Scalable Transport Protocols

A.1.1.  Use of L4S Packet Identifier

   Description: A Scalable congestion control needs to distinguish the
   packets it sends from those sent by Classic congestion controls (see
   the precise normative requirement wording in Section 4.1).

   Motivation: It needs to be possible for a network node to classify
   L4S packets without flow state into a queue that applies an L4S ECN-
   marking behaviour and isolates L4S packets from the queuing delay of
   Classic packets.

A.1.2.  Accurate ECN Feedback

   Description: The transport protocol for a Scalable congestion control
   needs to provide timely, accurate feedback about the extent of ECN
   marking experienced by all packets (see the precise normative
   requirement wording in Section 4.2).

   Motivation: Classic congestion controls only need feedback about the
   existence of a congestion episode within a round trip, not precisely
   how many packets were ECN-marked or dropped.  Therefore, in 2001,
   when ECN feedback was added to TCP [RFC3168], it could not inform the
   sender of more than one ECN mark per RTT.  Since then, requirements
   for more accurate ECN feedback in TCP have been defined in [RFC7560],
   and [ACCECN] specifies a change to the TCP protocol to satisfy these
   requirements.  Most other transport protocols already satisfy this
   requirement (see Section 4.2).

A.1.3.  Capable of Replacement by Classic Congestion Control

   Description: It needs to be possible to replace the implementation of
   a Scalable congestion control with a Classic control (see the precise
   normative requirement wording in Section 4.3, Paragraph 8, Item 1).

   Motivation: L4S is an experimental protocol; therefore, it seems
   prudent to be able to disable it at source in case of insurmountable
   problems, perhaps due to some unexpected interaction on a particular
   sender; over a particular path or network; or with a particular
   receiver, or even ultimately an insurmountable problem with the
   experiment as a whole.

A.1.4.  Fall Back to Classic Congestion Control on Packet Loss

   Description: As well as responding to ECN markings in a scalable way,
   a Scalable congestion control needs to react to packet loss in a way
   that will coexist safely with a Reno congestion control [RFC5681]
   (see the precise normative requirement wording in Section 4.3,
   Paragraph 8, Item 2).

   Motivation: Part of the safety conditions for deploying a Scalable
   congestion control on the public Internet is to make sure that it
   behaves properly when it builds a queue at a network bottleneck that
   has not been upgraded to support L4S.  Packet loss can have many
   causes, but it usually has to be conservatively assumed that it is a
   sign of congestion.  Therefore, on detecting packet loss, a Scalable
   congestion control will need to fall back to Classic congestion
   control behaviour.  If it does not comply, it could starve Classic
   traffic.

   A Scalable congestion control can be used for different types of
   transport, e.g., for real-time media or for reliable transport like
   TCP.  Therefore, the particular Classic congestion control behaviour
   to fall back on will need to be dependent on the specific congestion
   control implementation.  In the particular case of DCTCP, the DCTCP
   specification [RFC8257] states that "A DCTCP sender MUST react to
   loss episodes in the same way as conventional TCP,...".  To ensure
   any Scalable congestion control is safe to deploy over the public
   Internet, Item 2 of Section 4.3 in the present spec does not require
   precisely the same response as Reno TCP, but it does require a
   response that will coexist safely with Classic congestion controls
   like Reno.

   Even though a bottleneck is L4S capable, it might still become
   overloaded and have to drop packets.  In this case, the sender may
   receive a high proportion of packets marked with the CE codepoint and
   also experience loss.  Current DCTCP implementations each react
   differently to this situation.  One approach is to react only to the
   drop signal (e.g., by halving the cwnd); another approach is to react
   to both signals, which reduces cwnd by more than half.  A compromise
   between these two has been proposed where the loss response is
   adjusted to result in a halving when combined with any ECN response
   earlier in the same round.  We believe that further experimentation
   is needed to understand what is the best behaviour for the public
   Internet, which may or may not be one of these existing approaches.

A.1.5.  Coexistence with Classic Congestion Control at Classic ECN
        Bottlenecks

   Description: Monitoring has to be in place so that a non-L4S but ECN-
   capable AQM can be detected at path bottlenecks.  This is in case
   such an AQM has been implemented in a shared queue, in which case any
   long-running Scalable flow would predominate over any simultaneous
   long-running Classic flow sharing the queue.  The precise requirement
   wording in Section 4.3, Paragraph 8, Item 3 is written so that such a
   problem could be resolved either in real time or via administrative
   intervention.

   Motivation: Similarly to the discussion in Appendix A.1.4, this
   requirement in Section 4.3 is a safety condition to ensure an L4S
   congestion control coexists well with Classic flows when it builds a
   queue at a shared network bottleneck that has not been upgraded to
   support L4S.  Nonetheless, if necessary, it is considered reasonable
   to resolve such problems over management timescales (possibly
   involving human intervention) because:

   *  although a Classic flow can considerably reduce its throughput in
      the face of a competing Scalable flow, it still makes progress and
      does not starve;

   *  implementations of a Classic ECN AQM in a queue that is intended
      to be shared are believed to be rare; and

   *  detection of such AQMs is not always clear-cut; so focused out-of-
      band testing (or even contacting the relevant network operator)
      would improve certainty.

   The relevant normative requirement (Section 4.3) is therefore divided
   into three stages: monitoring, detection, and action:

   Monitoring:  Monitoring involves collection of the measurement data
      to be analysed.  Monitoring is expressed as a "MUST" for
      uncontrolled environments, although the placement of the
      monitoring function is left open.  Whether monitoring has to be
      applied in real time is expressed as a "SHOULD".  This allows for
      the possibility that the operator of an L4S sender (e.g., a
      Content Distribution Network (CDN)) might prefer to test out-of-
      band for signs of Classic ECN AQMs, perhaps to avoid continually
      consuming resources to monitor live traffic.

   Detection:  Detection involves analysis of the monitored data to
      detect the likelihood of a Classic ECN AQM.  Detection can either
      directly detect actual coexistence problems between flows or aim
      to identify AQM technologies that are likely to present
      coexistence problems, based on knowledge of AQMs deployed at the
      time.  The requirements recommend that detection occurs live in
      real time.  However, detection is allowed to be deferred (e.g., it
      might involve further testing targeted at candidate AQMs).

   Action:  This involves the act of switching the sender to a Classic
      congestion control.  This might occur in real time within the
      congestion control for the subsequent duration of a flow, or it
      might involve administrative action to switch to Classic
      congestion control for a specific interface or for a certain set
      of destination addresses.

      Instead of the sender taking action itself, the operator of the
      sender (e.g., a CDN) might prefer to ask the network operator to
      modify the Classic AQM's treatment of L4S packets; ensure L4S
      packets bypass the AQM; or upgrade the AQM to support L4S (see the
      L4S operational guidance [L4SOPS]).  If L4S flows then no longer
      shared the Classic ECN AQM, they would obviously no longer detect
      it, and the requirement to act on it would no longer apply.

   The whole set of normative requirements concerning Classic ECN AQMs
   in Section 4.3 is worded so that it does not apply in controlled
   environments, such as private networks or data-centre networks.  CDN
   servers placed within an access ISP's network can be considered as a
   single controlled environment, but any onward networks served by the
   access network, including all the attached customer networks, would
   be unlikely to fall under the same degree of coordinated control.
   Monitoring is expressed as a "MUST" for these uncontrolled segments
   of paths (e.g., beyond the access ISP in a home network), because
   there is a possibility that there might be a shared queue Classic ECN
   AQM in that segment.  Nonetheless, the intent of the wording is to
   only require occasional monitoring of these uncontrolled regions and
   not to burden CDN operators if monitoring never uncovers any
   potential problems.

   More detailed discussion of all the above options and alternatives
   can be found in the L4S operational guidance [L4SOPS].

   Having said all the above, the approach recommended in Section 4.3 is
   to monitor, detect, and act in real time on live traffic.  A passive
   monitoring algorithm to detect a Classic ECN AQM at the bottleneck
   and fall back to Classic congestion control is described in an
   extensive technical report [ecn-fallback], which also provides a link
   to Linux source code and a large online visualization of its
   evaluation results.  Very briefly, the algorithm primarily monitors
   RTT variation using the same algorithm that maintains the mean
   deviation of TCP's smoothed RTT, but it smooths over a duration of
   the order of a Classic sawtooth.  The outcome is also conditioned on
   other metrics such as the presence of CE marking and congestion
   avoidance phase having stabilized.  The report also identifies
   further work to improve the approach, for instance, improvements with
   low-capacity links and combining the measurements with a cache of
   what had been learned about a path in previous connections.  The
   report also suggests alternative approaches.

   Although using passive measurements within live traffic (as above)
   can detect a Classic ECN AQM, it is much harder (perhaps impossible)
   to determine whether or not the AQM is in a shared queue.
   Nonetheless, this is much easier using active test traffic out-of-
   band because two flows can be used.  Section 4 of the same report
   [ecn-fallback] describes a simple technique to detect a Classic ECN
   AQM and determine whether it is in a shared queue, which is
   summarized here.

   An L4S-enabled test server could be set up so that, when a test
   client accesses it, it serves a script that gets the client to open
   two parallel long-running flows.  It could serve one with a Classic
   congestion control (C, that sets ECT(0)) and one with a Scalable CC
   (L, that sets ECT(1)).  If neither flow induces any ECN marks, it can
   be presumed that the path does not contain a Classic ECN AQM.  If
   either flow induces some ECN marks, the server could measure the
   relative flow rates and round-trip times of the two flows.  Table 2
   shows the AQM that can be inferred for various cases (presuming no
   more types of AQM behaviour than those known at the time of writing).

                +========+=======+========================+
                | Rate   | RTT   | Inferred AQM           |
                +========+=======+========================+
                | L > C  | L = C | Classic ECN AQM (FIFO) |
                +--------+-------+------------------------+
                | L = C  | L = C | Classic ECN AQM (FQ)   |
                +--------+-------+------------------------+
                | L = C  | L < C | FQ-L4S AQM             |
                +--------+-------+------------------------+
                | L ~= C | L < C | DualQ Coupled AQM      |
                +========+=======+========================+
                |           L = L4S; C = Classic          |
                +=========================================+

                   Table 2: Out-of-Band Testing with Two
                               Parallel Flows

   Finally, we motivate the recommendation in Section 4.3 that a
   Scalable congestion control is not expected to change to setting
   ECT(0) while it adapts its behaviour to coexist with Classic flows.
   This is because the sender needs to continue to check whether it made
   the right decision and switch back if it was wrong, or if a different
   link becomes the bottleneck:

   *  If, as recommended, the sender changes only its behaviour but not
      its codepoint to Classic, its codepoint will still be compatible
      with either an L4S or a Classic AQM.  If the bottleneck does
      actually support both, it will still classify ECT(1) into the same
      L4S queue, where the sender can measure that switching to Classic
      behaviour was wrong so that it can switch back.

   *  In contrast, if the sender changes both its behaviour and its
      codepoint to Classic, even if the bottleneck supports both, it
      will classify ECT(0) into the Classic queue, reinforcing the
      sender's incorrect decision so that it never switches back.

   *  Also, not changing its codepoint avoids the risk of being flipped
      to a different path by a load balancer or multipath routing that
      hashes on the whole of the former Type-of-Service (ToS) byte
      (which is unfortunately still a common pathology).

      |  Note that if a flow is configured to _only_ use a Classic
      |  congestion control, it is then entirely appropriate not to use
      |  ECT(1).

A.1.6.  Reduce RTT Dependence

   Description: A Scalable congestion control needs to reduce RTT bias
   as much as possible at least over the low-to-typical range of RTTs
   that will interact in the intended deployment scenario (see the
   precise normative requirement wording in Section 4.3, Paragraph 8,
   Item 4).

   Motivation: The throughput of Classic congestion controls is known to
   be inversely proportional to RTT, so one would expect flows over very
   low RTT paths to nearly starve flows over larger RTTs.  However,
   Classic congestion controls have never allowed a very low RTT path to
   exist because they induce a large queue.  For instance, consider two
   paths with base RTT 1 ms and 100 ms.  If a Classic congestion control
   induces a 100 ms queue, it turns these RTTs into 101 ms and 200 ms,
   leading to a throughput ratio of about 2:1.  Whereas if a Scalable
   congestion control induces only a 1 ms queue, the ratio is 2:101,
   leading to a throughput ratio of about 50:1.

   Therefore, with very small queues, long RTT flows will essentially
   starve, unless Scalable congestion controls comply with the
   requirement in Section 4.3.

   Over higher than typical RTTs, L4S flows can use the same RTT bias as
   in current Classic congestion controls and still work satisfactorily.
   So there is no additional requirement in Section 4.3 for high RTT L4S
   flows to remove RTT bias -- they can, but they don't have to.

   One way for a Scalable congestion control to satisfy these
   requirements is to make its additive increase behave as if it were a
   standard Reno flow but over a larger RTT by using a virtual RTT
   (rtt_virt) that is a function of the actual RTT (rtt).  Example
   functions might be:

      rtt_virt = max(rtt, 25 ms)

      rtt_virt = rtt + 10 ms

   These example functions are chosen so that, as the actual RTT reduces
   from high to low, the virtual RTT reduces less (see [PRAGUE-CC] for
   details).

   However, short RTT flows can more rapidly respond to changes in
   available capacity, whether due to other flows arriving and departing
   or radio capacity varying.  So it would be wrong to require short RTT
   flows to be as sluggish as long RTT flows, which would unnecessarily
   underutilize capacity and result in unnecessary overshoots and
   undershoots (instability).  Therefore, rather than requiring strict
   RTT independence, the wording in Item 4 of Section 4.3 is "as
   independent of RTT as possible without compromising stability or
   utilization".  This allows shorter RTT flows to exploit their agility
   advantage.

A.1.7.  Scaling Down to Fractional Congestion Windows

   Description: A Scalable congestion control needs to remain responsive
   to congestion when typical RTTs over the public Internet are
   significantly smaller because they are no longer inflated by queuing
   delay (see the precise normative requirement wording in Section 4.3,
   Paragraph 8, Item 5).

   Motivation: As currently specified, the minimum congestion window of
   ECN-capable TCP (and its derivatives) is expected to be 2 sender
   maximum segment sizes (SMSS), or 1 SMSS after a retransmission
   timeout.  Once the congestion window reaches this minimum, if there
   is further ECN marking, TCP is meant to wait for a retransmission
   timeout before sending another segment (see Section 6.1.2 of the ECN
   spec [RFC3168]).  In practice, most known window-based congestion
   control algorithms become unresponsive to ECN congestion signals at
   this point.  No matter how much ECN marking, the congestion window no
   longer reduces.  Instead, the sender's lack of any further congestion
   response forces the queue to grow, overriding any AQM and increasing
   queuing delay (making the window large enough to become responsive
   again).  This can result in a stable but deeper queue, or it might
   drive the queue to loss, in which case the retransmission timeout
   mechanism acts as a backstop.

   Most window-based congestion controls for other transport protocols
   have a similar minimum window, albeit when measured in bytes for
   those that use smaller packets.

   L4S mechanisms significantly reduce queuing delay so, over the same
   path, the RTT becomes lower.  Then, this problem becomes surprisingly
   common [sub-mss-prob].  This is because, for the same link capacity,
   smaller RTT implies a smaller window.  For instance, consider a
   residential setting with an upstream broadband Internet access of 8
   Mb/s, assuming a max segment size of 1500 B.  Two upstream flows will
   each have the minimum window of 2 SMSS if the RTT is 6 ms or less,
   which is quite common when accessing a nearby data centre.  So any
   more than two such parallel TCP flows will become unresponsive to ECN
   and increase queuing delay.

   Unless Scalable congestion controls address the requirement in
   Section 4.3 from the start, they will frequently become unresponsive
   to ECN, negating the low-latency benefit of L4S, for themselves and
   for others.

   That would seem to imply that Scalable congestion controllers ought
   to be required to be able work with a congestion window less than 1
   SMSS.  For instance, if an ECN-capable TCP gets an ECN mark when it
   is already sitting at a window of 1 SMSS, [RFC3168] requires it to
   defer sending for a retransmission timeout.  A less drastic but more
   complex mechanism can maintain a congestion window less than 1 SMSS
   (significantly less if necessary), as described in [Ahmed19].  Other
   approaches are likely to be feasible.

   However, the requirement in Section 4.3 is worded as a "SHOULD"
   because it is believed that the existence of a minimum window is not
   all bad.  When competing with an unresponsive flow, a minimum window
   naturally protects the flow from starvation by at least keeping some
   data flowing.

   By stating the requirement to go lower than 1 SMSS as a "SHOULD",
   while the requirement in [RFC3168] still stands as well, we shall be
   able to watch the choices of minimum window evolve in different
   Scalable congestion controllers.

A.1.8.  Measuring Reordering Tolerance in Time Units

   Description: When detecting loss, a Scalable congestion control needs
   to be tolerant to reordering over an adaptive time interval, which
   scales with throughput, rather than counting only in fixed units of
   packets, which does not scale (see the precise normative requirement
   wording in Section 4.3, Paragraph 8, Item 6).

   Motivation: A primary purpose of L4S is scalable throughput (it's in
   the name).  Scalability in all dimensions is, of course, also a goal
   of all IETF technology.  The inverse linear congestion response in
   Section 4.3 is necessary, but not sufficient, to solve the congestion
   control scalability problem identified in [RFC3649].  As well as
   maintaining frequent ECN signals as rate scales, it is also important
   to ensure that a potentially false perception of loss does not limit
   throughput scaling.

   End systems cannot know whether a missing packet is due to loss or
   reordering, except in hindsight -- if it appears later.  So they can
   only deem that there has been a loss if a gap in the sequence space
   has not been filled, either after a certain number of subsequent
   packets has arrived (e.g., the 3 DupACK rule of standard TCP
   congestion control [RFC5681]) or after a certain amount of time
   (e.g., the RACK approach [RFC8985]).

   As we attempt to scale packet rate over the years:

   *  Even if only _some_ sending hosts still deem that loss has
      occurred by counting reordered packets, _all_ networks will have
      to keep reducing the time over which they keep packets in order.
      If some link technologies keep the time within which reordering
      occurs roughly unchanged, then loss over these links, as perceived
      by these hosts, will appear to continually rise over the years.

   *  In contrast, if all senders detect loss in units of time, the time
      over which the network has to keep packets in order stays roughly
      invariant.

   Therefore, hosts have an incentive to detect loss in time units (so
   as not to fool themselves too often into detecting losses when there
   are none).  And for hosts that are changing their congestion control
   implementation to L4S, there is no downside to including time-based
   loss detection code in the change (loss recovery implemented in
   hardware is an exception, which is covered later).  Therefore,
   requiring L4S hosts to detect loss in time-based units would not be a
   burden.

   If the requirement in Section 4.3 were not placed on L4S hosts, even
   though it would be no burden on hosts to comply, all networks would
   face unnecessary uncertainty over whether some L4S hosts might be
   detecting loss by counting packets.  Then, _all_ link technologies
   would have to unnecessarily keep reducing the time within which
   reordering occurs.  That is not a problem for some link technologies,
   but it becomes increasingly challenging for other link technologies
   to continue to scale, particularly those relying on channel bonding
   for scaling, such as LTE, 5G, and Data Over Cable Service Interface
   Specification (DOCSIS).

   Given Internet paths traverse many link technologies, any scaling
   limit for these more challenging access link technologies would
   become a scaling limit for the Internet as a whole.

   It might be asked how it helps to place this loss detection
   requirement only on L4S hosts, because networks will still face
   uncertainty over whether non-L4S flows are detecting loss by counting
   DupACKs.  The answer is that those link technologies for which it is
   challenging to keep squeezing the reordering time will only need to
   do so for non-L4S traffic (which they can do because the L4S
   identifier is visible at the IP layer).  Therefore, they can focus
   their processing and memory resources into scaling non-L4S (Classic)
   traffic.  Then, the higher the proportion of L4S traffic, the less of
   a scaling challenge they will have.

   To summarize, there is no reason for L4S hosts not to be part of the
   solution instead of part of the problem.

   Requirement ("MUST") or recommendation ("SHOULD")?  As explained
   above, this is a subtle interoperability issue between hosts and
   networks, which seems to need a "MUST".  Unless networks can be
   certain that all L4S hosts follow the time-based approach, they still
   have to cater for the worst case -- continually squeeze reordering
   into a smaller and smaller duration -- just for hosts that might be
   using the counting approach.  However, it was decided to express this
   as a recommendation, using "SHOULD".  The main justification was that
   networks can still be fairly certain that L4S hosts will follow this
   recommendation, because following it offers only gain and no pain.

   Details:

   The time spent recovering a loss is much more significant for short
   flows than long; therefore, a good compromise is to adapt the
   reordering window from a small fraction of the RTT at the start of a
   flow to a larger fraction of the RTT for flows that continue for many
   round trips.

   This is broadly the approach adopted by RACK [RFC8985].  However,
   RACK starts with the 3 DupACK approach, because the RTT estimate is
   not necessarily stable.  As long as the initial window is paced, such
   initial use of 3 DupACK counting would amount to time-based loss
   detection and therefore would satisfy the time-based loss detection
   recommendation of Section 4.3.  This is because pacing of the initial
   window would ensure that 3 DupACKs early in the connection would be
   spread over a small fraction of the round trip.

   As mentioned above, hardware implementations of loss recovery using
   DupACK counting exist (e.g., some implementations of Remote Direct
   Memory Access over Converged Ethernet version 2 (RoCEv2)).  For low
   latency, these implementations can change their congestion control to
   implement L4S, because the congestion control (as distinct from loss
   recovery) is implemented in software.  But they cannot easily satisfy
   this loss recovery requirement.  However, it is believed they do not
   need to, because such implementations are believed to solely exist in
   controlled environments, where the network technology keeps
   reordering extremely low anyway.  This is why controlled environments
   with hardly any reordering are excluded from the scope of the
   normative recommendation in Section 4.3.

   Detecting loss in time units also prevents the ACK-splitting attacks
   described in [Savage-TCP].

A.2.  Scalable Transport Protocol Optimizations

A.2.1.  Setting ECT in Control Packets and Retransmissions

   Description: This item concerns TCP and its derivatives (e.g., SCTP)
   as well as RTP/RTCP [RFC6679].  The original specification of ECN for
   TCP precluded the use of ECN on control packets and retransmissions.
   Similarly, [RFC6679] precludes the use of ECT on RTCP datagrams, in
   case the path changes after it has been checked for ECN traversal.
   To improve performance, Scalable transport protocols ought to enable
   ECN at the IP layer in TCP control packets (SYN, SYN-ACK, pure ACKs,
   etc.) and in retransmitted packets.  The same is true for other
   transports, e.g., SCTP and RTCP.

   Motivation (TCP): [RFC3168] prohibits the use of ECN on these types
   of TCP packets, based on a number of arguments.  This means these
   packets are not protected from congestion loss by ECN, which
   considerably harms performance, particularly for short flows.  ECN++
   [ECN++] proposes experimental use of ECN on all types of TCP packets
   as long as AccECN feedback [ACCECN] is available (which itself
   satisfies the accurate feedback requirement in Section 4.2 for using
   a Scalable congestion control).

   Motivation (RTCP): L4S experiments in general will need to observe
   the rule in the RTP ECN spec [RFC6679] that precludes ECT on RTCP
   datagrams.  Nonetheless, as ECN usage becomes more widespread, it
   would be useful to conduct specific experiments with ECN-capable RTCP
   to gather data on whether such caution is necessary.

A.2.2.  Faster than Additive Increase

   Description: It would improve performance if Scalable congestion
   controls did not limit their congestion window increase to the
   standard additive increase of 1 SMSS per round trip [RFC5681] during
   congestion avoidance.  The same is true for derivatives of TCP
   congestion control, including similar approaches used for real-time
   media.

   Motivation: As currently defined [RFC8257], DCTCP uses the
   conventional Reno additive increase in the congestion avoidance
   phase.  When the available capacity suddenly increases (e.g., when
   another flow finishes or if radio capacity increases) it can take
   very many round trips to take advantage of the new capacity.  TCP
   CUBIC [RFC8312] was designed to solve this problem, but as flow rates
   have continued to increase, the delay accelerating into available
   capacity has become prohibitive.  See, for instance, the examples in
   Section 5.1 of the L4S architecture [RFC9330].  Even when out of its
   Reno-friendly mode, every 8 times scaling of CUBIC's flow rate leads
   to 2 times more acceleration delay.

   In the steady state, DCTCP induces about 2 ECN marks per round trip,
   so it is possible to quickly detect when these signals have
   disappeared and seek available capacity more rapidly, while
   minimizing the impact on other flows (Classic and Scalable)
   [LinuxPacedChirping].  Alternatively, approaches such as Adaptive-
   Acceleration Data Center TCP (A2DTCP) [A2DTCP]) have been proposed to
   address this problem in data centres, which might be deployable over
   the public Internet.

A.2.3.  Faster Convergence at Flow Start

   Description: It would improve performance if Scalable congestion
   controls converged (reached their steady-state share of the capacity)
   faster than Classic congestion controls or at least no slower.  This
   affects the flow start behaviour of any L4S congestion control
   derived from a Classic transport that uses TCP slow start, including
   those for real-time media.

   Motivation: As an example, a new DCTCP flow takes longer than a
   Classic congestion control to obtain its share of the capacity of the
   bottleneck when there are already ongoing flows using the bottleneck
   capacity.  In a data-centre environment, DCTCP takes about 1.5 to 2
   times longer to converge due to the much higher typical level of ECN
   marking that DCTCP background traffic induces, which causes new flows
   to exit slow start early [Alizadeh-stability].  In testing for use
   over the public Internet, the convergence time of DCTCP relative to a
   regular loss-based TCP slow start is even less favourable
   [LinuxPacedChirping] due to the shallow ECN-marking threshold needed
   for L4S.  It is exacerbated by the typically greater mismatch between
   the link rate of the sending host and typical Internet access
   bottlenecks.  This problem is detrimental in general but would
   particularly harm the performance of short flows relative to Classic
   congestion controls.

Appendix B.  Compromises in the Choice of L4S Identifier

   This appendix is informative, not normative.  As explained in
   Section 3, there is insufficient space in the IP header (v4 or v6) to
   fully accommodate every requirement.  So the choice of L4S identifier
   involves trade-offs.  This appendix records the pros and cons of the
   choice that was made.

   Non-normative recap of the chosen codepoint scheme:

      Packets with ECT(1) and conditionally packets with CE signify L4S
      semantics as an alternative to the semantics of Classic ECN
      [RFC3168], specifically:

      -  The ECT(1) codepoint signifies that the packet was sent by an
         L4S-capable sender.

      -  Given the shortage of codepoints, both the L4S and Classic ECN
         sides of an AQM have to use the same CE codepoint to indicate
         that a packet has experienced congestion.  If a packet that had
         already been marked CE in an upstream buffer arrived at a
         subsequent AQM, this AQM would then have to guess whether to
         classify CE packets as L4S or Classic ECN.  Choosing the L4S
         treatment is a safer choice, because then a few Classic packets
         might arrive early rather than a few L4S packets arriving late.

      -  Additional information might be available if the classifier
         were transport-aware.  Then, it could classify a CE packet for
         Classic ECN treatment if the most recent ECT packet in the same
         flow had been set to ECT(0).  However, the L4S service ought
         not need transport-layer awareness.

   Cons:

   Consumes the last ECN codepoint:  The L4S service could potentially
      supersede the service provided by Classic ECN; therefore, using
      ECT(1) to identify L4S packets could ultimately mean that the
      ECT(0) codepoint was 'wasted' purely to distinguish one form of
      ECN from its successor.

   ECN hard in some lower layers:  It is not always possible to support
      the equivalent of an IP-ECN field in an AQM acting in a buffer
      below the IP layer [ECN-ENCAP].  Then, depending on the lower-
      layer scheme, the L4S service might have to drop rather than mark
      frames even though they might encapsulate an ECN-capable packet.

   Risk of reordering Classic CE packets within a flow:  Classifying all
      CE packets into the L4S queue risks any CE packets that were
      originally ECT(0) being incorrectly classified as L4S.  If there
      were delay in the Classic queue, these incorrectly classified CE
      packets would arrive early, which is a form of reordering.
      Reordering within a microflow can cause TCP senders (and senders
      of similar transports) to retransmit spuriously.  However, the
      risk of spurious retransmissions would be extremely low for the
      following reasons:

      1.  It is quite unusual to experience queuing at more than one
          bottleneck on the same path (the available capacities have to
          be identical).

      2.  In only a subset of these unusual cases would the first
          bottleneck support Classic ECN marking and the second L4S ECN
          marking.  This would be the only scenario where some ECT(0)
          packets could be CE marked by an AQM supporting Classic ECN
          while subsequently the remaining ECT(0) packets would
          experience further delay through the Classic side of a
          subsequent L4S DualQ AQM.

      3.  Even then, when a few packets are delivered early, it takes
          very unusual conditions to cause a spurious retransmission, in
          contrast to when some packets are delivered late.  The first
          bottleneck has to apply CE marks to at least N contiguous
          packets, and the second bottleneck has to inject an
          uninterrupted sequence of at least N of these packets between
          two packets earlier in the stream (where N is the reordering
          window that the transport protocol allows before it considers
          a packet is lost).

             For example, consider N=3, and consider the sequence of
             packets 100, 101, 102, 103,... Imagine that packets 150,
             151, 152 from later in the flow are injected as follows:
             100, 150, 151, 101, 152, 102, 103,... If this were late
             reordering, even one packet arriving out of sequence would
             trigger a spurious retransmission, but there is no spurious
             retransmission here with early reordering, because packet
             101 moves the cumulative ACK counter forward before 3
             packets have arrived out of order.  Later, when packets
             148, 149, 153,... arrive, even though there is a 3-packet
             hole, there will be no problem, because the packets to fill
             the hole are already in the receive buffer.

      4.  Even with the current TCP recommendation of N=3 [RFC5681],
          spurious retransmissions will be unlikely for all the above
          reasons.  As RACK [RFC8985] is becoming widely deployed, it
          tends to adapt its reordering window to a larger value of N,
          which will make the chance of a contiguous sequence of N early
          arrivals vanishingly small.

      5.  Even a run of 2 CE marks within a Classic ECN flow is
          unlikely, given FQ-CoDel is the only known widely deployed AQM
          that supports Classic ECN marking, and it takes great care to
          separate out flows and to space any markings evenly along each
          flow.

      It is extremely unlikely that the above set of 5 eventualities
      that are each unusual in themselves would all happen
      simultaneously.  But, even if they did, the consequences would
      hardly be dire: the odd spurious fast retransmission.  Whenever
      the traffic source (a Classic congestion control) mistakes the
      reordering of a string of CE marks for a loss, one might think
      that it will reduce its congestion window as well as emitting a
      spurious retransmission.  However, it would have already reduced
      its congestion window when the CE markings arrived early.  If it
      is using ABE [RFC8511], it might reduce cwnd a little more for a
      loss than for a CE mark.  But it will revert that reduction once
      it detects that the retransmission was spurious.

      In conclusion, the impact of early reordering on spurious
      retransmissions due to CE being ambiguous will generally be
      vanishingly small.

   Insufficient anti-replay window in some pre-existing VPNs:  If delay
      is reduced for a subset of the flows within a VPN, the anti-replay
      feature of some VPNs is known to potentially mistake the
      difference in delay for a replay attack.  Section 6.2 recommends
      that the anti-replay window at the VPN egress is sufficiently
      sized, as required by the relevant specifications.  However, in
      some VPN implementations, the maximum anti-replay window is
      insufficient to cater for a large delay difference at prevailing
      packet rates.  Section 6.2 suggests alternative work-rounds for
      such cases, but end users using L4S over a VPN will need to be
      able to recognize the symptoms of this problem, in order to seek
      out these work-rounds.

   Hard to distinguish Classic ECN AQM:  With this scheme, when a source
      receives ECN feedback, it is not explicitly clear which type of
      AQM generated the CE markings.  This is not a problem for Classic
      ECN sources that send ECT(0) packets, because an L4S AQM will
      recognize the ECT(0) packets as Classic and apply the appropriate
      Classic ECN-marking behaviour.

      However, in the absence of explicit disambiguation of the CE
      markings, an L4S source needs to use heuristic techniques to work
      out which type of congestion response to apply (see
      Appendix A.1.5).  Otherwise, if long-running Classic flows are
      sharing a Classic ECN AQM bottleneck with long-running L4S flows,
      and the L4S flows apply an L4S response to the Classic CE signals,
      they would then outcompete the Classic flows.  Experiments have
      shown that L4S flows can take about 20 times more capacity share
      than equivalent Classic flows.  Nonetheless, as link capacity
      reduces (e.g., to 4 Mb/s), the inequality reduces.  So Classic
      flows always make progress and are not starved.

      When L4S was first proposed (in 2015, 14 years after the Classic
      ECN spec [RFC3168] was published), it was believed that Classic
      ECN AQMs had failed to be deployed because research measurements
      had found little or no evidence of CE marking.  In subsequent
      years, Classic ECN was included in FQ deployments; however, an FQ
      scheduler stops an L4S flow outcompeting Classic, because it
      enforces equality between flow rates.  It is not known whether
      there have been any non-FQ deployments of Classic ECN AQMs in the
      subsequent years or whether there will be any in future.

      An algorithm for detecting a Classic ECN AQM as soon as a flow
      stabilizes after start-up has been proposed [ecn-fallback] (see
      Appendix A.1.5 for a brief summary).  Testbed evaluations of v2 of
      the algorithm have shown detection is reasonably good for Classic
      ECN AQMs, in a wide range of circumstances.  However, although it
      can correctly detect an L4S ECN AQM in many circumstances, it is
      often incorrect at low link capacities and/or high RTTs.  Although
      this is the safe way round, there is a danger that it will
      discourage use of the algorithm.

   Non-L4S service for control packets:  Solely for the case of TCP, the
      Classic ECN RFCs [RFC3168] and [RFC5562] require a sender to clear
      the IP-ECN field to Not-ECT on retransmissions and on certain
      control packets, specifically pure ACKs, window probes, and SYNs.
      When L4S packets are classified by the IP-ECN field, these TCP
      control packets would not be classified into an L4S queue and
      could therefore be delayed relative to the other packets in the
      flow.  This would not cause reordering (because retransmissions
      are already out of order, and these control packets typically
      carry no data).  However, it would make critical TCP control
      packets more vulnerable to loss and delay.  To address this
      problem, ECN++ [ECN++] proposes an experiment in which all TCP
      control packets and retransmissions are ECN-capable as long as
      appropriate ECN feedback is available in each case.

   Pros:

   Should work end to end:  The IP-ECN field generally propagates end to
      end across the Internet without being wiped or mangled, at least
      over fixed networks.  Unlike the DSCP, the setting of the ECN
      field is at least meant to be forwarded unchanged by networks that
      do not support ECN.

   Should work in tunnels:  The L4S identifiers work across and within
      any tunnel that propagates the IP-ECN field in any of the variant
      ways it has been defined since ECN-tunneling was first specified
      in the year 2001 [RFC3168].  However, it is likely that some
      tunnels still do not implement ECN propagation at all.

   Should work for many link technologies:  At most, but not all, path
      bottlenecks there is IP awareness, so that L4S AQMs can be located
      where the IP-ECN field can be manipulated.  Bottlenecks at lower-
      layer nodes without IP awareness have to either use drop to signal
      congestion or have a specific congestion notification facility
      defined for that link technology, including propagation to and
      from IP-ECN.  The programme to define these is progressing, and in
      each case so far, the scheme already defined for ECN inherently
      supports L4S as well (see Section 6.1).

   Could migrate to one codepoint:  If all Classic ECN senders
      eventually evolve to use the L4S service, the ECT(0) codepoint
      could be reused for some future purpose but only once use of
      ECT(0) packets has reduced to zero, or near zero, which might
      never happen.

   L4 not required:  Being based on the IP-ECN field, this scheme does
      not need the network to access transport-layer flow IDs.
      Nonetheless, it does not preclude solutions that do.

Appendix C.  Potential Competing Uses for the ECT(1) Codepoint

   The ECT(1) codepoint of the IP-ECN field has already been assigned
   once for the ECN nonce spec [RFC3540], which has now been categorized
   as Historic [RFC8311].  ECN is probably the only remaining field in
   the Internet Protocol that is common to IPv4 and IPv6 and still has
   potential to work end to end, with tunnels and with lower layers.
   Therefore, ECT(1) should not be reassigned to a different
   experimental use (L4S) without carefully assessing competing
   potential uses.  These fall into the categories described below.

C.1.  Integrity of Congestion Feedback

   Receiving hosts can fool a sender into downloading faster by
   suppressing feedback of ECN marks (or of losses if retransmissions
   are not necessary or available otherwise).

   The Historic ECN nonce spec [RFC3540] proposed that a TCP sender
   could set either ECT(0) or ECT(1) in each packet of a flow and
   remember the sequence it had set.  If any packet was lost or
   congestion marked, the receiver would miss that bit of the sequence.
   An ECN nonce receiver had to feed back the least-significant bit of
   the sum, so it could not suppress feedback of a loss or mark without
   a 50-50 chance of guessing the sum incorrectly.

   It is highly unlikely that ECT(1) will be needed as a nonce for
   integrity protection of congestion notifications in future.  The ECN
   nonce spec [RFC3540] has been reclassified as Historic, partly
   because other ways (that do not consume a codepoint in the IP header)
   have been developed to protect feedback integrity of TCP and other
   transports [RFC8311].  For instance:

   *  The sender can test the integrity of a small random sample of the
      receiver's feedback by occasionally setting the IP-ECN field to a
      value normally only set by the network.  Then, it can test whether
      the receiver's feedback faithfully reports what it expects (see
      Paragraph 2 of Section 20.2 of the ECN spec [RFC3168].  This works
      for loss, and it will work for the accurate ECN feedback [RFC7560]
      intended for L4S.  Like the (Historic) ECN nonce spec, this
      technique does not protect against a misbehaving sender.  But it
      allows a well-behaved sender to check that each receiver is
      correctly feeding back congestion notifications.

   *  A network can check that its ECN markings (or packet losses) have
      been passed correctly around the full feedback loop by auditing
      Congestion Exposure (ConEx) [RFC7713].  This assures that the
      integrity of congestion notifications and feedback messages must
      have both been preserved.  ConEx information is also available
      anywhere along the network path, so it can be used to enforce a
      congestion response.  Whether the receiver or a downstream network
      is suppressing congestion feedback or the sender is unresponsive
      to the feedback, or both, ConEx is intended to neutralize any
      advantage that any of these three parties would otherwise gain.

   *  Congestion feedback fields in transport-layer headers are
      immutable end to end and therefore amenable to end-to-end
      integrity protection.  This preserves the integrity of a
      receiver's feedback messages to the sender, but it does not
      protect against misbehaving receivers or misbehaving senders.  The
      TCP Authentication Option (TCP-AO) [RFC5925], QUIC's end-to-end
      protection [RFC9001], or end-to-end IPsec integrity protection
      [RFC4303] can be used to detect any tampering with congestion
      feedback (whether malicious or accidental), respectively, in TCP,
      QUIC, or any transport.  TCP-AO covers the main TCP header and TCP
      options by default, but it is often too brittle to use on many
      end-to-end paths, where middleboxes can make verification fail in
      their attempts to improve performance or security, e.g., by
      resegmentation or shifting the sequence space.

   At the time of writing, it is becoming common to protect the
   integrity of transport feedback using QUIC.  However, it is still not
   common to protect the integrity of the wider congestion feedback
   loop, whether based on loss or Classic ECN.  If this position changes
   during the L4S experiment, one or more of the above techniques might
   need to be developed and deployed.

C.2.  Notification of Less Severe Congestion than CE

   Various researchers have proposed to use ECT(1) as a less severe
   congestion notification than CE, particularly to enable flows to fill
   available capacity more quickly after an idle period, when another
   flow departs or when a flow starts, e.g., the Variable-structure
   congestion Control Protocol (VCP) [VCP] and Queue View (QV) [QV].

   Before assigning ECT(1) as an identifier for L4S, we must carefully
   consider whether it might be better to hold ECT(1) in reserve for
   future standardization of rapid flow acceleration, which is an
   important and enduring problem [RFC6077].

   Pre-Congestion Notification (PCN) is another scheme that assigns
   alternative semantics to the IP-ECN field.  It uses ECT(1) to signify
   a less severe level of pre-congestion notification than CE [RFC6660].
   However, the IP-ECN field only takes on the PCN semantics if packets
   carry a Diffserv codepoint defined to indicate PCN marking within a
   controlled environment.  PCN is required to be applied solely to the
   outer header of a tunnel across the controlled region in order not to
   interfere with any end-to-end use of the ECN field.  Therefore, a PCN
   region on the path would not interfere with the L4S service
   identifier defined in Section 2.

Acknowledgements

   Thanks to Richard Scheffenegger, John Leslie, David Täht, Jonathan
   Morton, Gorry Fairhurst, Michael Welzl, Mikael Abrahamsson, and
   Andrew McGregor for the discussions that led to this specification.
   Ing-jyh (Inton) Tsang was a contributor to the early draft versions
   of this document.  Thanks to Mikael Abrahamsson, Lloyd Wood, Nicolas
   Kuhn, Greg White, Tom Henderson, David Black, Gorry Fairhurst, Brian
   Carpenter, Jake Holland, Rod Grimes, Richard Scheffenegger, Sebastian
   Moeller, Neal Cardwell, Praveen Balasubramanian, Reza Marandian Hagh,
   Pete Heist, Stuart Cheshire, Vidhi Goel, Mirja Kühlewind, Ermin
   Sakic, and Martin Duke for providing help and reviewing this
   document.  And thanks to Ingemar Johansson for reviewing and
   providing substantial text.  Thanks also to the area reviewers:
   Valery Smyslov, Maria Ines Robles, Bernard Aboba, Lars Eggert, Roman
   Danyliw, and Éric Vyncke.  Thanks to Sebastian Moeller for
   identifying the interaction with VPN anti-replay and to Jonathan
   Morton for identifying the attack based on this.  Particular thanks
   to tsvwg chairs Gorry Fairhurst, David Black, and Wes Eddy for
   patiently helping this and the other L4S documents through the IETF
   process.  Appendix A, which lists the Prague L4S Requirements, is
   based on text authored by Marcelo Bagnulo Braun that was originally
   an appendix to [RFC9330].  That text was in turn based on the
   collective output of the attendees listed in the minutes of a 'bar
   BoF' on DCTCP Evolution during IETF 94 [TCPPrague].

   The authors' contributions were partly funded by the European
   Community under its Seventh Framework Programme through the Reducing
   Internet Transport Latency (RITE) project (ICT-317700).  The
   contribution of Koen De Schepper was also partly funded by the
   5Growth and DAEMON EU H2020 projects.  Bob Briscoe was partly funded
   by the Research Council of Norway through the TimeIn project,
   CableLabs, and the Comcast Innovation Fund.  The views expressed here
   are solely those of the authors.

Authors' Addresses

   Koen De Schepper
   Nokia Bell Labs
   Antwerp
   Belgium
   Email: koen.de_schepper@nokia.com
   URI:   https://www.bell-labs.com/about/researcher-profiles/
   koende_schepper/


   Bob Briscoe (editor)
   Independent
   United Kingdom
   Email: ietf@bobbriscoe.net
   URI:   https://bobbriscoe.net/