doc: Add RFC documents

1 files changed, 2084 insertions, 0 deletions

diff --git a/doc/rfc/rfc9049.txt b/doc/rfc/rfc9049.txt
new file mode 100644
index 0000000..588d8c2
--- /dev/null
+++ b/doc/rfc/rfc9049.txt
@@ -0,0 +1,2084 @@
+
+
+
+
+Internet Research Task Force (IRTF)                      S. Dawkins, Ed.
+Request for Comments: 9049                               Tencent America
+Category: Informational                                        June 2021
+ISSN: 2070-1721
+
+
+             Path Aware Networking: Obstacles to Deployment
+                    (A Bestiary of Roads Not Taken)
+
+Abstract
+
+   This document is a product of the Path Aware Networking Research
+   Group (PANRG).  At the first meeting of the PANRG, the Research Group
+   agreed to catalog and analyze past efforts to develop and deploy Path
+   Aware techniques, most of which were unsuccessful or at most
+   partially successful, in order to extract insights and lessons for
+   Path Aware networking researchers.
+
+   This document contains that catalog and analysis.
+
+Status of This Memo
+
+   This document is not an Internet Standards Track specification; it is
+   published for informational purposes.
+
+   This document is a product of the Internet Research Task Force
+   (IRTF).  The IRTF publishes the results of Internet-related research
+   and development activities.  These results might not be suitable for
+   deployment.  This RFC represents the consensus of the Path Aware
+   Networking Research Group of the Internet Research Task Force (IRTF).
+   Documents approved for publication by the IRSG are not candidates for
+   any level of Internet Standard; see Section 2 of RFC 7841.
+
+   Information about the current status of this document, any errata,
+   and how to provide feedback on it may be obtained at
+   https://www.rfc-editor.org/info/rfc9049.
+
+Copyright Notice
+
+   Copyright (c) 2021 IETF Trust and the persons identified as the
+   document authors.  All rights reserved.
+
+   This document is subject to BCP 78 and the IETF Trust's Legal
+   Provisions Relating to IETF Documents
+   (https://trustee.ietf.org/license-info) in effect on the date of
+   publication of this document.  Please review these documents
+   carefully, as they describe your rights and restrictions with respect
+   to this document.
+
+Table of Contents
+
+   1.  Introduction
+     1.1.  What Do "Path" and "Path Awareness" Mean in This Document?
+   2.  A Perspective on This Document
+     2.1.  Notes for the Reader
+     2.2.  A Note about Path Aware Techniques Included in This
+           Document
+     2.3.  Architectural Guidance
+     2.4.  Terminology Used in This Document
+     2.5.  Methodology for Contributions
+   3.  Applying the Lessons We've Learned
+   4.  Summary of Lessons Learned
+     4.1.  Justifying Deployment
+     4.2.  Providing Benefits for Early Adopters
+     4.3.  Providing Benefits during Partial Deployment
+     4.4.  Outperforming End-to-End Protocol Mechanisms
+     4.5.  Paying for Path Aware Techniques
+     4.6.  Impact on Operational Practices
+     4.7.  Per-Connection State
+     4.8.  Keeping Traffic on Fast Paths
+     4.9.  Endpoints Trusting Intermediate Nodes
+     4.10. Intermediate Nodes Trusting Endpoints
+     4.11. Reacting to Distant Signals
+     4.12. Support in Endpoint Protocol Stacks
+     4.13. Planning for Failure
+   5.  Future Work
+   6.  Contributions
+     6.1.  Stream Transport (ST, ST2, ST2+)
+       6.1.1.  Reasons for Non-deployment
+       6.1.2.  Lessons Learned
+     6.2.  Integrated Services (IntServ)
+       6.2.1.  Reasons for Non-deployment
+       6.2.2.  Lessons Learned
+     6.3.  Quick-Start TCP
+       6.3.1.  Reasons for Non-deployment
+       6.3.2.  Lessons Learned
+     6.4.  ICMP Source Quench
+       6.4.1.  Reasons for Non-deployment
+       6.4.2.  Lessons Learned
+     6.5.  Triggers for Transport (TRIGTRAN)
+       6.5.1.  Reasons for Non-deployment
+       6.5.2.  Lessons Learned
+     6.6.  Shim6
+       6.6.1.  Reasons for Non-deployment
+       6.6.2.  Lessons Learned
+       6.6.3.  Addendum on Multipath TCP
+     6.7.  Next Steps in Signaling (NSIS)
+       6.7.1.  Reasons for Non-deployment
+       6.7.2.  Lessons Learned
+     6.8.  IPv6 Flow Labels
+       6.8.1.  Reasons for Non-deployment
+       6.8.2.  Lessons Learned
+     6.9.  Explicit Congestion Notification (ECN)
+       6.9.1.  Reasons for Non-deployment
+       6.9.2.  Lessons Learned
+   7.  Security Considerations
+   8.  IANA Considerations
+   9.  Informative References
+   Acknowledgments
+   Author's Address
+
+1.  Introduction
+
+   This document describes the lessons that IETF participants have
+   learned (and learned the hard way) about Path Aware networking over a
+   period of several decades.  It also provides an analysis of reasons
+   why various Path Aware techniques have seen limited or no deployment.
+
+   This document represents the consensus of the Path Aware Networking
+   Research Group (PANRG).
+
+1.1.  What Do "Path" and "Path Awareness" Mean in This Document?
+
+   One of the first questions reviewers of this document have asked is
+   "What's the definition of a Path, and what's the definition of Path
+   Awareness?"  That is not an easy question to answer for this
+   document.
+
+   These terms have definitions in other PANRG documents [PANRG] and are
+   still the subject of some discussion in the Research Group, as of the
+   date of this document.  But because this document reflects work
+   performed over several decades, the technologies described in
+   Section 6 significantly predate the current definitions of "Path" and
+   "Path Aware" in use in the Path Aware Networking Research Group, and
+   it is unlikely that all the contributors to Section 6 would have had
+   the same understanding of these terms.  Those technologies were
+   considered "Path Aware" in early PANRG discussions and so are
+   included in this retrospective document.
+
+   It is worth noting that the definitions of "Path" and "Path Aware" in
+   [PANRG-PATH-PROPERTIES] would apply to Path Aware techniques at a
+   number of levels of the Internet protocol architecture ([RFC1122],
+   plus several decades of refinements), but the contributions received
+   for this document tended to target the transport layer and to treat a
+   "Path" constructed by routers as opaque.  It would be useful to
+   consider how applicable the Lessons Learned cataloged in this
+   document are, at other layers, and that would be a fine topic for
+   follow-on research.
+
+   The current definition of "Path" in the Path Aware Networking
+   Research Group appears in Section 2 ("Terminology") in
+   [PANRG-PATH-PROPERTIES].  That definition is included here as a
+   convenience to the reader.
+
+   |  Path: A sequence of adjacent path elements over which a packet can
+   |  be transmitted, starting and ending with a node.  A path is
+   |  unidirectional.  Paths are time-dependent, i.e., the sequence of
+   |  path elements over which packets are sent from one node to another
+   |  may change.  A path is defined between two nodes.  For multicast
+   |  or broadcast, a packet may be sent by one node and received by
+   |  multiple nodes.  In this case, the packet is sent over multiple
+   |  paths at once, one path for each combination of sending and
+   |  receiving node; these paths do not have to be disjoint.  Note that
+   |  an entity may have only partial visibility of the path elements
+   |  that comprise a path and visibility may change over time.
+   |  Different entities may have different visibility of a path and/or
+   |  treat path elements at different levels of abstraction.
+
+   The current definition of Path Awareness, used by the Path Aware
+   Networking Research Group, appears in Section 1.1 ("Definition") in
+   [PANRG-QUESTIONS].  That definition is included here as a convenience
+   to the reader.
+
+   |  For purposes of this document, "path aware networking" describes
+   |  endpoint discovery of the properties of paths they use for
+   |  communication across an internetwork, and endpoint reaction to
+   |  these properties that affects routing and/or data transfer.  Note
+   |  that this can and already does happen to some extent in the
+   |  current Internet architecture; this definition expands current
+   |  techniques of path discovery and manipulation to cross
+   |  administrative domain boundaries and up to the transport and
+   |  application layers at the endpoints.
+   |  
+   |  Expanding on this definition, a "path aware internetwork" is one
+   |  in which endpoint discovery of path properties and endpoint
+   |  selection of paths used by traffic exchanged by the endpoint are
+   |  explicitly supported, regardless of the specific design of the
+   |  protocol features which enable this discovery and selection.
+
+2.  A Perspective on This Document
+
+   At the first meeting of the Path Aware Networking Research Group
+   [PANRG], at IETF 99 [PANRG-99], Olivier Bonaventure led a discussion
+   of "A Decade of Path Awareness" [PATH-Decade], on attempts, which
+   were mostly unsuccessful for a variety of reasons, to exploit Path
+   Aware techniques and achieve a variety of goals over the past decade.
+   At the end of that discussion, two things were abundantly clear.
+
+   *  The Internet community has accumulated considerable experience
+      with many Path Aware techniques over a long period of time, and
+
+   *  Although some Path Aware techniques have been deployed (for
+      example, Differentiated Services, or Diffserv [RFC2475]), most of
+      these techniques haven't seen widespread adoption and deployment.
+      Even "successful" techniques like Diffserv can face obstacles that
+      prevent wider usage.  The reasons for non-adoption and limited
+      adoption and deployment are many and are worthy of study.
+
+   The meta-lessons from that experience were as follows:
+
+   *  Path Aware networking has been more Research than Engineering, so
+      establishing an IRTF Research Group for Path Aware networking was
+      the right thing to do [RFC7418].
+
+   *  Analyzing a catalog of past experience to learn the reasons for
+      non-adoption would be a great first step for the Research Group.
+
+   Allison Mankin, as IRTF Chair, officially chartered the Path Aware
+   Networking Research Group in July 2018.
+
+   This document contains the analysis performed by that Research Group
+   (Section 4), based on that catalog (Section 6).
+
+2.1.  Notes for the Reader
+
+   This Informational document discusses Path Aware protocol mechanisms
+   considered, and in some cases standardized, by the Internet
+   Engineering Task Force (IETF), and it considers Lessons Learned from
+   those mechanisms.  The intention is to inform the work of protocol
+   designers, whether in the IRTF, the IETF, or elsewhere in the
+   Internet ecosystem.
+
+   As an Informational document published in the IRTF Stream, this
+   document has no authority beyond the quality of the analysis it
+   contains.
+
+2.2.  A Note about Path Aware Techniques Included in This Document
+
+   This document does not catalog every proposed Path Aware technique
+   that was not adopted and deployed.  Instead, we limited our focus to
+   technologies that passed through the IETF community and still
+   identified enough techniques to provide background for the lessons
+   included in Section 4 to inform researchers and protocol engineers in
+   their work.
+
+   No shame is intended for the techniques included in this document.
+   As shown in Section 4, the quality of specific techniques had little
+   to do with whether they were deployed or not.  Based on the
+   techniques cataloged in this document, it is likely that when these
+   techniques were put forward, the proponents were trying to engineer
+   something that could not be engineered without first carrying out
+   research.  Actual shame would be failing to learn from experience and
+   failing to share that experience with other networking researchers
+   and engineers.
+
+2.3.  Architectural Guidance
+
+   As background for understanding the Lessons Learned contained in this
+   document, the reader is encouraged to become familiar with the
+   Internet Architecture Board's documents on "What Makes for a
+   Successful Protocol?"  [RFC5218] and "Planning for Protocol Adoption
+   and Subsequent Transitions" [RFC8170].
+
+   Although these two documents do not specifically target Path Aware
+   networking protocols, they are helpful resources for readers seeking
+   to improve their understanding of considerations for successful
+   adoption and deployment of any protocol.  For example, the basic
+   success factors described in Section 2.1 of [RFC5218] are helpful for
+   readers of this document.
+
+   Because there is an economic aspect to decisions about deployment,
+   the IAB Workshop on Internet Technology Adoption and Transition
+   [ITAT] report [RFC7305] also provides food for thought.
+
+   Several of the Lessons Learned in Section 4 reflect considerations
+   described in [RFC5218], [RFC7305], and [RFC8170].
+
+2.4.  Terminology Used in This Document
+
+   The terms "node" and "element" in this document have the meaning
+   defined in [PANRG-PATH-PROPERTIES].
+
+2.5.  Methodology for Contributions
+
+   This document grew out of contributions by various IETF participants
+   with experience with one or more Path Aware techniques.
+
+   There are many things that could be said about the Path Aware
+   techniques that have been developed.  For the purposes of this
+   document, contributors were requested to provide
+
+   *  the name of a technique, including an abbreviation if one was
+      used.
+
+   *  if available, a long-term pointer to the best reference describing
+      the technique.
+
+   *  a short description of the problem the technique was intended to
+      solve.
+
+   *  a short description of the reasons why the technique wasn't
+      adopted.
+
+   *  a short statement of the lessons that researchers can learn from
+      our experience with this technique.
+
+3.  Applying the Lessons We've Learned
+
+   The initial scope for this document was roughly "What mistakes have
+   we made in the decade prior to [PANRG-99], that we shouldn't make
+   again?"  Some of the contributions in Section 6 predate the initial
+   scope.  The earliest Path Aware technique referred to in Section 6 is
+   [IEN-119], which was published in the late 1970s; see Section 6.1.
+   Given that the networking ecosystem has evolved continuously, it
+   seems reasonable to consider how to apply these lessons.
+
+   The PANRG reviewed the Lessons Learned (Section 4) contained in the
+   May 23, 2019 draft version of this document at IETF 105
+   [PANRG-105-Min] and carried out additional discussion at IETF 106
+   [PANRG-106-Min].  Table 1 provides the "sense of the room" about each
+   lesson after those discussions.  The intention was to capture whether
+   a specific lesson seems to be
+
+   *  "Invariant" - well-understood and is likely to be applicable for
+      any proposed Path Aware networking solution.
+
+   *  "Variable" - has impeded deployment in the past but might not be
+      applicable in a specific technique.  Engineering analysis to
+      understand whether the lesson is applicable is prudent.
+
+   *  "Not Now" - a characteristic that tends to turn up a minefield
+      full of dragons.  Prudent network engineers will wish to avoid
+      gambling on a technique that relies on this, until something
+      significant changes.
+
+   Section 6.9 on Explicit Congestion Notification (ECN) was added
+   during the review and approval process, based on a question from
+   Martin Duke.  Section 6.9, as contained in the March 8, 2021 draft
+   version of this document, was discussed at [PANRG-110] and is
+   summarized in Section 4.13, describing a new Lesson Learned.
+
+    +=====================================================+===========+
+    | Lesson                                              | Category  |
+    +=====================================================+===========+
+    | Justifying Deployment (Section 4.1)                 | Invariant |
+    +-----------------------------------------------------+-----------+
+    | Providing Benefits for Early Adopters (Section 4.2) | Invariant |
+    +-----------------------------------------------------+-----------+
+    | Providing Benefits during Partial Deployment        | Invariant |
+    | (Section 4.3)                                       |           |
+    +-----------------------------------------------------+-----------+
+    | Outperforming End-to-End Protocol Mechanisms        | Variable  |
+    | (Section 4.4)                                       |           |
+    +-----------------------------------------------------+-----------+
+    | Paying for Path Aware Techniques (Section 4.5)      | Invariant |
+    +-----------------------------------------------------+-----------+
+    | Impact on Operational Practices (Section 4.6)       | Invariant |
+    +-----------------------------------------------------+-----------+
+    | Per-Connection State (Section 4.7)                  | Variable  |
+    +-----------------------------------------------------+-----------+
+    | Keeping Traffic on Fast Paths (Section 4.8)         | Variable  |
+    +-----------------------------------------------------+-----------+
+    | Endpoints Trusting Intermediate Nodes (Section 4.9) | Not Now   |
+    +-----------------------------------------------------+-----------+
+    | Intermediate Nodes Trusting Endpoints               | Not Now   |
+    | (Section 4.10)                                      |           |
+    +-----------------------------------------------------+-----------+
+    | Reacting to Distant Signals (Section 4.11)          | Variable  |
+    +-----------------------------------------------------+-----------+
+    | Support in Endpoint Protocol Stacks (Section 4.12)  | Variable  |
+    +-----------------------------------------------------+-----------+
+    | Planning for Failure (Section 4.13)                 | Invariant |
+    +-----------------------------------------------------+-----------+
+
+                                  Table 1
+
+   "Justifying Deployment", "Providing Benefits for Early Adopters",
+   "Paying for Path Aware Techniques", "Impact on Operational
+   Practices", and "Planning for Failure" were considered to be
+   Invariant -- the sense of the room was that these would always be
+   considerations for any proposed Path Aware technique.
+
+   "Providing Benefits during Partial Deployment" was added after IETF
+   105, during Research Group Last Call, and is also considered to be
+   Invariant.
+
+   For "Outperforming End-to-End Protocol Mechanisms", there is a trade-
+   off between improved performance from Path Aware techniques and
+   additional complexity required by some Path Aware techniques.
+
+   *  For example, if you can obtain the same understanding of path
+      characteristics from measurements obtained over a few more round
+      trips, endpoint implementers are unlikely to be eager to add
+      complexity, and many attributes can be measured from an endpoint,
+      without assistance from intermediate nodes.
+
+   For "Per-Connection State", the key questions discussed in the
+   Research Group were "how much state" and "where state is maintained".
+
+   *  Integrated Services (IntServ) (Section 6.2) required state at
+      every participating intermediate node for every connection between
+      two endpoints.  As the Internet ecosystem has evolved, carrying
+      many connections in a tunnel that appears to intermediate nodes as
+      a single connection has become more common, so that additional
+      end-to-end connections don't add additional state to intermediate
+      nodes between tunnel endpoints.  If these tunnels are encrypted,
+      intermediate nodes between tunnel endpoints can't distinguish
+      between connections, even if that were desirable.
+
+   For "Keeping Traffic on Fast Paths", we noted that this was true for
+   many platforms, but not for all.
+
+   *  For backbone routers, this is likely an Invariant, but for
+      platforms that rely more on general-purpose computers to make
+      forwarding decisions, this may not be a fatal flaw for Path Aware
+      techniques.
+
+   For "Endpoints Trusting Intermediate Nodes" and "Intermediate Nodes
+   Trusting Endpoints", these lessons point to the broader need to
+   revisit the Internet Threat Model.
+
+   *  We noted with relief that discussions about this were already
+      underway in the IETF community at IETF 105 (see the Security Area
+      Open Meeting minutes [SAAG-105-Min] for discussion of
+      [INTERNET-THREAT-MODEL] and [FARRELL-ETM]), and the Internet
+      Architecture Board has created a mailing list for continued
+      discussions [model-t], but we recognize that there are Path Aware
+      networking aspects of this effort, requiring research.
+
+   For "Reacting to Distant Signals", we noted that not all attributes
+   are equal.
+
+   *  If an attribute is stable over an extended period of time, is
+      difficult to observe via end-to-end mechanisms, and is valuable,
+      Path Aware techniques that rely on that attribute to provide a
+      significant benefit become more attractive.
+
+   *  Analysis to help identify attributes that are useful enough to
+      justify deployment of Path Aware techniques that make use of those
+      attributes would be helpful.
+
+   For "Support in Endpoint Protocol Stacks", we noted that Path Aware
+   applications must be able to identify and communicate requirements
+   about path characteristics.
+
+   *  The de facto sockets API has no way of signaling application
+      expectations for the network path to the protocol stack.
+
+4.  Summary of Lessons Learned
+
+   This section summarizes the Lessons Learned from the contributed
+   subsections in Section 6.
+
+   Each Lesson Learned is tagged with one or more contributions that
+   encountered this obstacle as a significant impediment to deployment.
+   Other contributed techniques may have also encountered this obstacle,
+   but this obstacle may not have been the biggest impediment to
+   deployment for those techniques.
+
+   It is useful to notice that sometimes an obstacle might impede
+   deployment, while at other times, the same obstacle might prevent
+   adoption and deployment entirely.  The Research Group discussed
+   distinguishing between obstacles that impede and obstacles that
+   prevent, but it appears that the boundary between "impede" and
+   "prevent" can shift over time -- some of the Lessons Learned are
+   based on both a) Path Aware techniques that were not deployed and b)
+   Path Aware techniques that were deployed but were not deployed widely
+   or quickly.  See Sections 6.6 and 6.6.3 for examples of this shifting
+   boundary.
+
+4.1.  Justifying Deployment
+
+   The benefit of Path Awareness must be great enough to justify making
+   changes in an operational network.  The colloquial U.S. American
+   English expression, "If it ain't broke, don't fix it" is a "best
+   current practice" on today's Internet.  (See Sections 6.3, 6.4, 6.5,
+   and 6.9, in addition to [RFC5218].)
+
+4.2.  Providing Benefits for Early Adopters
+
+   Providing benefits for early adopters can be key -- if everyone must
+   deploy a technique in order for the technique to provide benefits, or
+   even to work at all, the technique is unlikely to be adopted widely
+   or quickly.  (See Sections 6.2 and 6.3, in addition to [RFC5218].)
+
+4.3.  Providing Benefits during Partial Deployment
+
+   Some proposals require that all path elements along the full length
+   of the path must be upgraded to support a new technique, before any
+   benefits can be seen.  This is likely to require coordination between
+   operators who control a subset of path elements, and between
+   operators and end users if endpoint upgrades are required.  If a
+   technique provides benefits when only a part of the path has been
+   upgraded, this is likely to encourage adoption and deployment.  (See
+   Sections 6.2, 6.3, and 6.9, in addition to [RFC5218].)
+
+4.4.  Outperforming End-to-End Protocol Mechanisms
+
+   Adaptive end-to-end protocol mechanisms may respond to feedback
+   quickly enough that the additional realizable benefit from a new Path
+   Aware mechanism that tries to manipulate nodes along a path, or
+   observe the attributes of nodes along a path, may be much smaller
+   than anticipated.  (See Sections 6.3 and 6.5.)
+
+4.5.  Paying for Path Aware Techniques
+
+   "Follow the money."  If operators can't charge for a Path Aware
+   technique to recover the costs of deploying it, the benefits to the
+   operator must be really significant.  Corollary: if operators charge
+   for a Path Aware technique, the benefits to users of that Path Aware
+   technique must be significant enough to justify the cost.  (See
+   Sections 6.1, 6.2, 6.5, and 6.9.)
+
+4.6.  Impact on Operational Practices
+
+   The impact of a Path Aware technique requiring changes to operational
+   practices can affect how quickly or widely a promising technique is
+   deployed.  The impacts of these changes may make deployment more
+   likely, but they often discourage deployment.  (See Section 6.6,
+   including Section 6.6.3.)
+
+4.7.  Per-Connection State
+
+   Per-connection state in intermediate nodes has been an impediment to
+   adoption and deployment in the past, because of added cost and
+   complexity.  Often, similar benefits can be achieved with much less
+   finely grained state.  This is especially true as we move from the
+   edge of the network, further into the routing core.  (See
+   Sections 6.1 and 6.2.)
+
+4.8.  Keeping Traffic on Fast Paths
+
+   Many modern platforms, especially high-end routers, have been
+   designed with hardware that can make simple per-packet forwarding
+   decisions ("fast paths") but have not been designed to make heavy use
+   of in-band mechanisms such as IPv4 and IPv6 Router Alert Options
+   (RAOs) that require more processing to make forwarding decisions.
+   Packets carrying in-band mechanisms are diverted to other processors
+   in the router with much lower packet-processing rates.  Operators can
+   be reluctant to deploy techniques that rely heavily on in-band
+   mechanisms because they may significantly reduce packet throughput.
+   (See Section 6.7.)
+
+4.9.  Endpoints Trusting Intermediate Nodes
+
+   If intermediate nodes along the path can't be trusted, it's unlikely
+   that endpoints will rely on signals from intermediate nodes to drive
+   changes to endpoint behaviors.  We note that "trust" is not binary --
+   one low level of trust applies when a node receiving a message can
+   confirm that the sender of the message has visibility of the packets
+   on the path it is seeking to control [RFC8085] (e.g., an ICMP
+   Destination Unreachable message [RFC0792] that includes the Internet
+   Header + 64 bits of Original Data Datagram payload from the source).
+   A higher level of trust can arise when an endpoint has established a
+   short-term, or even long-term, trust relationship with network nodes.
+   (See Sections 6.4 and 6.5.)
+
+4.10.  Intermediate Nodes Trusting Endpoints
+
+   If the endpoints do not have any trust relationship with the
+   intermediate nodes along a path, operators have been reluctant to
+   deploy techniques that rely on endpoints sending unauthenticated
+   control signals to routers.  (See Sections 6.2 and 6.7.)  (We also
+   note that this still remains a factor hindering deployment of
+   Diffserv.)
+
+4.11.  Reacting to Distant Signals
+
+   Because the Internet is a distributed system, if the distance that
+   information from distant path elements travels to a Path Aware host
+   is sufficiently large, the information may no longer accurately
+   represent the state and situation at the distant host or elements
+   along the path when it is received locally.  In this case, the
+   benefit that a Path Aware technique provides will be inconsistent and
+   may not always be beneficial.  (See Section 6.3.)
+
+4.12.  Support in Endpoint Protocol Stacks
+
+   Just because a protocol stack provides a new feature/signal does not
+   mean that applications will use the feature/signal.  Protocol stacks
+   may not know how to effectively utilize Path Aware techniques,
+   because the protocol stack may require information from applications
+   to permit the technique to work effectively, but applications may not
+   a priori know that information.  Even if the application does know
+   that information, the de facto sockets API has no way of signaling
+   application expectations for the network path to the protocol stack.
+   In order for applications to provide these expectations to protocol
+   stacks, we need an API that signals more than the packets to be sent.
+   (See Sections 6.1 and 6.2.)
+
+4.13.  Planning for Failure
+
+   If early implementers discover severe problems with a new feature,
+   that feature is likely to be disabled, and convincing implementers to
+   re-enable that feature can be very difficult and can require years or
+   decades.  In addition to testing, partial deployment for a subset of
+   users, implementing instrumentation that will detect degraded user
+   experience, and even "failback" to a previous version or "failover"
+   to an entirely different implementation are likely to be helpful.
+   (See Section 6.9.)
+
+5.  Future Work
+
+   By its nature, this document has been retrospective.  In addition to
+   considering how the Lessons Learned to date apply to current and
+   future Path Aware networking proposals, it's also worth considering
+   whether there is deeper investigation left to do.
+
+   *  We note that this work was based on contributions from experts on
+      various Path Aware techniques, and all of the contributed
+      techniques involved unicast protocols.  We didn't consider how
+      these lessons might apply to multicast, and, given anecdotal
+      reports at the IETF 109 Media Operations (MOPS) Working Group
+      meeting of IP multicast offerings within data centers at one or
+      more cloud providers [MOPS-109-Min], it might be useful to think
+      about Path Awareness in multicast, before we have a history of
+      unsuccessful deployments to document.
+
+   *  The question of whether a mechanism supports admission control,
+      based on either endpoints or applications, is associated with Path
+      Awareness.  One of the motivations of IntServ and a number of
+      other architectures (e.g., Deterministic Networking [RFC8655]) is
+      the ability to "say no" to an application based on resource
+      availability on a path, before the application tries to inject
+      traffic onto that path and discovers the path does not have the
+      capacity to sustain enough utility to meet the application's
+      minimum needs.  The question of whether admission control is
+      needed comes up repeatedly, but we have learned a few useful
+      lessons that, while covered implicitly in some of the Lessons
+      Learned provided in this document, might be explained explicitly:
+
+      -  We have gained a lot of experience with application-based
+         adaptation since the days where applications just injected
+         traffic inelastically into the network.  Such adaptations seem
+         to work well enough that admission control is of less value to
+         these applications.
+
+      -  There are end-to-end measurement techniques that can steer
+         traffic at the application layer (Content Delivery Networks
+         (CDNs), multi-CDNs like Conviva [Conviva], etc.).
+
+      -  We noted in Section 4.12 that applications often don't know how
+         to utilize Path Aware techniques.  This includes not knowing
+         enough about their admission control threshold to be able to
+         ask accurately for the resources they need, whether this is
+         because the application itself doesn't know or because the
+         application has no way to signal its expectations to the
+         underlying protocol stack.  To date, attempts to help them
+         haven't gotten anywhere (e.g., the multiple-TSPEC (Traffic
+         Specification) additions to RSVP to attempt to mirror codec
+         selection by applications [INTSERV-MULTIPLE-TSPEC] expired in
+         2013).
+
+   *  We note that this work took the then-current IP network
+      architecture as given, at least at the time each technique was
+      proposed.  It might be useful to consider aspects of the now-
+      current IP network architecture that ease, or impede, Path Aware
+      techniques.  For example, there is limited ability in IP to
+      constrain bidirectional paths to be symmetric, and information-
+      centric networking protocols such as Named Data Networking (NDN)
+      and Content-Centric Networking (CCNx) [RFC8793] must force
+      bidirectional path symmetry using protocol-specific mechanisms.
+
+6.  Contributions
+
+   Contributions on these Path Aware techniques were analyzed to arrive
+   at the Lessons Learned captured in Section 4.
+
+   Our expectation is that most readers will not need to read through
+   this section carefully, but we wanted to record these hard-fought
+   lessons as a service to others who may revisit this document, so
+   they'll have the details close at hand.
+
+6.1.  Stream Transport (ST, ST2, ST2+)
+
+   The suggested references for Stream Transport are:
+
+   *  "ST - A Proposed Internet Stream Protocol" [IEN-119]
+
+   *  "Experimental Internet Stream Protocol: Version 2 (ST-II)"
+      [RFC1190]
+
+   *  "Internet Stream Protocol Version 2 (ST2) Protocol Specification -
+      Version ST2+" [RFC1819]
+
+   The first version of Stream Transport, ST [IEN-119], was published in
+   the late 1970s and was implemented and deployed on the ARPANET at
+   small scale.  It was used throughout the 1980s for experimental
+   transmission of voice, video, and distributed simulation.
+
+   The second version of the ST specification (ST2) [RFC1190] [RFC1819]
+   was an experimental connection-oriented internetworking protocol that
+   operated at the same layer as connectionless IP.  ST2 packets could
+   be distinguished by their IP header version numbers (IP, at that
+   time, used version number 4, while ST2 used version number 5).
+
+   ST2 used a control plane layered over IP to select routes and reserve
+   capacity for real-time streams across a network path, based on a flow
+   specification communicated by a separate protocol.  The flow
+   specification could be associated with QoS state in routers,
+   producing an experimental resource reservation protocol.  This
+   allowed ST2 routers along a path to offer end-to-end guarantees,
+   primarily to satisfy the QoS requirements for real-time services over
+   the Internet.
+
+6.1.1.  Reasons for Non-deployment
+
+   Although implemented in a range of equipment, ST2 was not widely used
+   after completion of the experiments.  It did not offer the
+   scalability and fate-sharing properties that have come to be desired
+   by the Internet community.
+
+   The ST2 protocol is no longer in use.
+
+6.1.2.  Lessons Learned
+
+   As time passed, the trade-off between router processing and link
+   capacity changed.  Links became faster, and the cost of router
+   processing became comparatively more expensive.
+
+   The ST2 control protocol used "hard state" -- once a route was
+   established, and resources were reserved, routes and resources
+   existed until they were explicitly released via signaling.  A soft-
+   state approach was thought superior to this hard-state approach and
+   led to development of the IntServ model described in Section 6.2.
+
+6.2.  Integrated Services (IntServ)
+
+   The suggested references for IntServ are:
+
+   *  "Integrated Services in the Internet Architecture: an Overview"
+      [RFC1633]
+
+   *  "Specification of the Controlled-Load Network Element Service"
+      [RFC2211]
+
+   *  "Specification of Guaranteed Quality of Service" [RFC2212]
+
+   *  "General Characterization Parameters for Integrated Service
+      Network Elements" [RFC2215]
+
+   *  "Resource ReSerVation Protocol (RSVP) -- Version 1 Functional
+      Specification" [RFC2205]
+
+   In 1994, when the IntServ architecture document [RFC1633] was
+   published, real-time traffic was first appearing on the Internet.  At
+   that time, bandwidth was still a scarce commodity.  Internet Service
+   Providers built networks over DS3 (45 Mbps) infrastructure, and sub-
+   rate (< 1 Mbps) access was common.  Therefore, the IETF anticipated a
+   need for a fine-grained QoS mechanism.
+
+   In the IntServ architecture, some applications can require service
+   guarantees.  Therefore, those applications use RSVP [RFC2205] to
+   signal QoS reservations across network paths.  Every router in the
+   network that participates in IntServ maintains per-flow soft state to
+   a) perform call admission control and b) deliver guaranteed service.
+
+   Applications use Flow Specifications (Flow Specs, or FLOWSPECs)
+   [RFC2210] to describe the traffic that they emit.  RSVP reserves
+   capacity for traffic on a per-Flow-Spec basis.
+
+6.2.1.  Reasons for Non-deployment
+
+   Although IntServ has been used in enterprise and government networks,
+   IntServ was never widely deployed on the Internet because of its
+   cost.  The following factors contributed to operational cost:
+
+   *  IntServ must be deployed on every router that is on a path where
+      IntServ is to be used.  Although it is possible to include a
+      router that does not participate in IntServ along the path being
+      controlled, if that router is likely to become a bottleneck,
+      IntServ cannot be used to avoid that bottleneck along the path.
+
+   *  IntServ maintained per-flow state.
+
+   As IntServ was being discussed, the following occurred:
+
+   *  For many expected uses, it became more cost effective to solve the
+      QoS problem by adding bandwidth.  Between 1994 and 2000, Internet
+      Service Providers upgraded their infrastructures from DS3 (45
+      Mbps) to OC-48 (2.4 Gbps).  This meant that even if an endpoint
+      was using IntServ in an IntServ-enabled network, its requests
+      would rarely, if ever, be denied, so endpoints and Internet
+      Service Providers had little reason to enable IntServ.
+
+   *  Diffserv [RFC2475] offered a more cost-effective, albeit less
+      fine-grained, solution to the QoS problem.
+
+6.2.2.  Lessons Learned
+
+   The following lessons were learned:
+
+   *  Any mechanism that requires every participating on-path router to
+      maintain per-flow state is not likely to succeed, unless the
+      additional cost for offering the feature can be recovered from the
+      user.
+
+   *  Any mechanism that requires an operator to upgrade all of its
+      routers is not likely to succeed, unless the additional cost for
+      offering the feature can be recovered from the user.
+
+   In environments where IntServ has been deployed, trust relationships
+   with endpoints are very different from trust relationships on the
+   Internet itself.  There are often clearly defined hierarchies in
+   Service Level Agreements (SLAs) governing well-defined transport
+   flows operating with predetermined capacity and latency requirements
+   over paths where capacity or other attributes are constrained.
+
+   IntServ was never widely deployed to manage capacity across the
+   Internet.  However, the technique that it produced was deployed for
+   reasons other than bandwidth management.  RSVP is widely deployed as
+   an MPLS signaling mechanism.  BGP reuses the RSVP concept of Filter
+   Specs to distribute firewall filters, although they are called "Flow
+   Spec Component Types" in BGP [RFC5575].
+
+6.3.  Quick-Start TCP
+
+   The suggested references for Quick-Start TCP are:
+
+   *  "Quick-Start for TCP and IP" [RFC4782]
+
+   *  "Determining an appropriate sending rate over an underutilized
+      network path" [SAF07]
+
+   *  "Fast Startup Internet Congestion Control for Broadband
+      Interactive Applications" [Sch11]
+
+   *  "Using Quick-Start to enhance TCP-friendly rate control
+      performance in bidirectional satellite networks" [QS-SAT]
+
+   Quick-Start is defined in an Experimental RFC [RFC4782] and is a TCP
+   extension that leverages support from the routers on the path to
+   determine an allowed initial sending rate for a path through the
+   Internet, either at the start of data transfers or after idle
+   periods.  Without information about the path, a sender cannot easily
+   determine an appropriate initial sending rate.  The default TCP
+   congestion control therefore uses the safe but time-consuming slow-
+   start algorithm [RFC5681].  With Quick-Start, connections are allowed
+   to use higher initial sending rates if there is significant unused
+   bandwidth along the path and if the sender and all of the routers
+   along the path approve the request.
+
+   By examining the Time To Live (TTL) field in Quick-Start packets, a
+   sender can determine if routers on the path have approved the Quick-
+   Start request.  However, this method is unable to take into account
+   the routers hidden by tunnels or other network nodes invisible at the
+   IP layer.
+
+   The protocol also includes a nonce that provides protection against
+   cheating routers and receivers.  If the Quick-Start request is
+   explicitly approved by all routers along the path, the TCP host can
+   send at up to the approved rate; otherwise, TCP would use the default
+   congestion control.  Quick-Start requires modifications in the
+   involved end systems as well as in routers.  Due to the resulting
+   deployment challenges, Quick-Start was only proposed in [RFC4782] for
+   controlled environments.
+
+   The Quick-Start mechanism is a lightweight, coarse-grained, in-band,
+   network-assisted fast startup mechanism.  The benefits are studied by
+   simulation in a research paper [SAF07] that complements the protocol
+   specification.  The study confirms that Quick-Start can significantly
+   speed up mid-sized data transfers.  That paper also presents router
+   algorithms that do not require keeping per-flow state.  Later studies
+   [Sch11] comprehensively analyze Quick-Start with a full Linux
+   implementation and with a router fast-path prototype using a network
+   processor.  In both cases, Quick-Start could be implemented with
+   limited additional complexity.
+
+6.3.1.  Reasons for Non-deployment
+
+   However, experiments with Quick-Start in [Sch11] revealed several
+   challenges:
+
+   *  Having information from the routers along the path can reduce the
+      risk of congestion but cannot avoid it entirely.  Determining
+      whether there is unused capacity is not trivial in actual router
+      and host implementations.  Data about available capacity visible
+      at the IP layer may be imprecise, and due to the propagation
+      delay, information can already be outdated when it reaches a
+      sender.  There is a trade-off between the speedup of data
+      transfers and the risk of congestion even with Quick-Start.  This
+      could be mitigated by only allowing Quick-Start to access a
+      proportion of the unused capacity along a path.
+
+   *  For scalable router fast-path implementations, it is important to
+      enable parallel processing of packets, as this is a widely used
+      method, e.g., in network processors.  One challenge is
+      synchronization of information between packets that are processed
+      in parallel, which should be avoided as much as possible.
+
+   *  Only some types of application traffic can benefit from Quick-
+      Start.  Capacity needs to be requested and discovered.  The
+      discovered capacity needs to be utilized by the flow, or it
+      implicitly becomes available for other flows.  Failing to use the
+      requested capacity may have already reduced the pool of Quick-
+      Start capacity that was made available to other competing Quick-
+      Start requests.  The benefit is greatest when senders use this
+      only for bulk flows and avoid sending unnecessary Quick-Start
+      requests, e.g., for flows that only send a small amount of data.
+      Choosing an appropriate request size requires application-internal
+      knowledge that is not commonly expressed by the transport API.
+      How a sender can determine the rate for an initial Quick-Start
+      request is still a largely unsolved problem.
+
+   There is no known deployment of Quick-Start for TCP or other IETF
+   transports.
+
+6.3.2.  Lessons Learned
+
+   Some lessons can be learned from Quick-Start.  Despite being a very
+   lightweight protocol, Quick-Start suffers from poor incremental
+   deployment properties regarding both a) the required modifications in
+   network infrastructure and b) its interactions with applications.
+   Except for corner cases, congestion control can be quite efficiently
+   performed end to end in the Internet, and in modern stacks there is
+   not much room for significant improvement by additional network
+   support.
+
+   After publication of the Quick-Start specification, there have been
+   large-scale experiments with an initial window of up to 10 segments
+   [RFC6928].  This alternative "IW10" approach can also ramp up data
+   transfers faster than the standard congestion control, but it only
+   requires sender-side modifications.  As a result, this approach can
+   be easier and incrementally deployed in the Internet.  While
+   theoretically Quick-Start can outperform "IW10", the improvement in
+   completion time for data transfer times can, in many cases, be small.
+   After publication of [RFC6928], most modern TCP stacks have increased
+   their default initial window.
+
+6.4.  ICMP Source Quench
+
+   The suggested reference for ICMP Source Quench is:
+
+   *  "Internet Control Message Protocol" [RFC0792]
+
+   The ICMP Source Quench message [RFC0792] allowed an on-path router to
+   request the source of a flow to reduce its sending rate.  This method
+   allowed a router to provide an early indication of impending
+   congestion on a path to the sources that contribute to that
+   congestion.
+
+6.4.1.  Reasons for Non-deployment
+
+   This method was deployed in Internet routers over a period of time;
+   the reaction of endpoints to receiving this signal has varied.  For
+   low-speed links, with low multiplexing of flows the method could be
+   used to regulate (momentarily reduce) the transmission rate.
+   However, the simple signal does not scale with link speed or with the
+   number of flows sharing a link.
+
+   The approach was overtaken by the evolution of congestion control
+   methods in TCP [RFC2001], and later also by other IETF transports.
+   Because these methods were based upon measurement of the end-to-end
+   path and an algorithm in the endpoint, they were able to evolve and
+   mature more rapidly than methods relying on interactions between
+   operational routers and endpoint stacks.
+
+   After ICMP Source Quench was specified, the IETF began to recommend
+   that transports provide end-to-end congestion control [RFC2001].  The
+   Source Quench method has been obsoleted by the IETF [RFC6633], and
+   both hosts and routers must now silently discard this message.
+
+6.4.2.  Lessons Learned
+
+   This method had several problems.
+
+   First, [RFC0792] did not sufficiently specify how the sender would
+   react to the ICMP Source Quench signal from the path (e.g.,
+   [RFC1016]).  There was ambiguity in how the sender should utilize
+   this additional information.  This could lead to unfairness in the
+   way that receivers (or routers) responded to this message.
+
+   Second, while the message did provide additional information, the
+   Explicit Congestion Notification (ECN) mechanism [RFC3168] provided a
+   more robust and informative signal for network nodes to provide early
+   indication that a path has become congested.
+
+   The mechanism originated at a time when the Internet trust model was
+   very different.  Most endpoint implementations did not attempt to
+   verify that the message originated from an on-path node before they
+   utilized the message.  This made it vulnerable to Denial-of-Service
+   (DoS) attacks.  In theory, routers might have chosen to use the
+   quoted packet contained in the ICMP payload to validate that the
+   message originated from an on-path node, but this would have
+   increased per-packet processing overhead for each router along the
+   path and would have required transport functionality in the router to
+   verify whether the quoted packet header corresponded to a packet the
+   router had sent.  In addition, Section 5.2 of [RFC4443] noted
+   ICMPv6-based attacks on hosts that would also have threatened routers
+   processing ICMPv6 Source Quench payloads.  As time passed, it became
+   increasingly obvious that the lack of validation of the messages
+   exposed receivers to a security vulnerability where the messages
+   could be forged to create a tangible DoS opportunity.
+
+6.5.  Triggers for Transport (TRIGTRAN)
+
+   The suggested references for TRIGTRAN are:
+
+   *  TRIGTRAN BOF at IETF 55 [TRIGTRAN-55]
+
+   *  TRIGTRAN BOF at IETF 56 [TRIGTRAN-56]
+
+   TCP [RFC0793] has a well-known weakness -- the end-to-end flow
+   control mechanism has only a single signal, the loss of a segment,
+   detected when no acknowledgment for the lost segment is received at
+   the sender.  There are multiple reasons why the sender might not have
+   received an acknowledgment for the segment.  To name several, the
+   segment could have been trapped in a routing loop, damaged in
+   transmission and failed checksum verification at the receiver, or
+   lost because some intermediate device discarded the packet, or any of
+   a variety of other things could have happened to the acknowledgment
+   on the way back from the receiver to the sender.  TCP implementations
+   since the late 1980s have made the "safe" decision and have
+   interpreted the loss of a segment as evidence that the path between
+   two endpoints may have become congested enough to exhaust buffers on
+   intermediate hops, so that the TCP sender should "back off" -- reduce
+   its sending rate until it knows that its segments are now being
+   delivered without loss [RFC5681].
+
+   The thinking behind TRIGTRAN was that if a path completely stopped
+   working because a link along the path was "down", somehow something
+   along the path could signal TCP when that link returned to service,
+   and the sending TCP could retry immediately, without waiting for a
+   full retransmission timeout (RTO) period.
+
+6.5.1.  Reasons for Non-deployment
+
+   The early dreams for TRIGTRAN were dashed because of an assumption
+   that TRIGTRAN triggers would be unauthenticated.  This meant that any
+   "safe" TRIGTRAN mechanism would have relied on a mechanism such as
+   setting the IPv4 TTL or IPv6 Hop Count to 255 at a sender and testing
+   that it was 254 upon receipt, so that a receiver could verify that a
+   signal was generated by an adjacent sender known to be on the path
+   being used and not some unknown sender that might not even be on the
+   path (e.g., "The Generalized TTL Security Mechanism (GTSM)"
+   [RFC5082]).  This situation is very similar to the case for ICMP
+   Source Quench messages as described in Section 6.4, which were also
+   unauthenticated and could be sent by an off-path attacker, resulting
+   in deprecation of ICMP Source Quench message processing [RFC6633].
+
+   TRIGTRAN's scope shrunk from "the path is down" to "the first-hop
+   link is down."
+
+   But things got worse.
+
+   Because TRIGTRAN triggers would only be provided when the first-hop
+   link was "down", TRIGTRAN triggers couldn't replace normal TCP
+   retransmission behavior if the path failed because some link further
+   along the network path was "down".  So TRIGTRAN triggers added
+   complexity to an already-complex TCP state machine and did not allow
+   any existing complexity to be removed.
+
+   There was also an issue that the TRIGTRAN signal was not sent in
+   response to a specific host that had been sending packets and was
+   instead a signal that stimulated a response by any sender on the
+   link.  This needs to scale when there are multiple flows trying to
+   use the same resource, yet the sender of a trigger has no
+   understanding of how many of the potential traffic sources will
+   respond by sending packets -- if recipients of the signal "back off"
+   their responses to a trigger to improve scaling, then that
+   immediately mitigates the benefit of the signal.
+
+   Finally, intermediate forwarding nodes required modification to
+   provide TRIGTRAN triggers, but operators couldn't charge for TRIGTRAN
+   triggers, so there was no way to recover the cost of modifying,
+   testing, and deploying updated intermediate nodes.
+
+   Two TRIGTRAN BOFs were held, at IETF 55 [TRIGTRAN-55] and IETF 56
+   [TRIGTRAN-56], but this work was not chartered, and there was no
+   interest in deploying TRIGTRAN unless it was chartered and
+   standardized in the IETF.
+
+6.5.2.  Lessons Learned
+
+   The reasons why this work was not chartered, much less deployed,
+   provide several useful lessons for researchers.
+
+   *  TRIGTRAN started with a plausible value proposition, but
+      networking realities in the early 2000s forced reductions in scope
+      that led directly to reductions in potential benefits but no
+      corresponding reductions in costs and complexity.
+
+   *  These reductions in scope were the direct result of an inability
+      for hosts to trust or authenticate TRIGTRAN signals they received
+      from the network.
+
+   *  Operators did not believe they could charge for TRIGTRAN
+      signaling, because first-hop links didn't fail frequently and
+      TRIGTRAN provided no reduction in operating expenses, so there was
+      little incentive to purchase and deploy TRIGTRAN-capable network
+      equipment.
+
+   It is also worth noting that the targeted environment for TRIGTRAN in
+   the late 1990s contained links with a relatively small number of
+   directly connected hosts -- for instance, cellular or satellite
+   links.  The transport community was well aware of the dangers of
+   sender synchronization based on multiple senders receiving the same
+   stimulus at the same time, but the working assumption for TRIGTRAN
+   was that there wouldn't be enough senders for this to be a meaningful
+   problem.  In the 2010s, it was common for a single "link" to support
+   many senders and receivers, likely requiring TRIGTRAN senders to wait
+   some random amount of time before sending after receiving a TRIGTRAN
+   signal, which would have reduced the benefits of TRIGTRAN even more.
+
+6.6.  Shim6
+
+   The suggested reference for Shim6 is:
+
+   *  "Shim6: Level 3 Multihoming Shim Protocol for IPv6" [RFC5533]
+
+   The IPv6 routing architecture [RFC1887] assumed that most sites on
+   the Internet would be identified by Provider Assigned IPv6 prefixes,
+   so that Default-Free Zone routers only contained routes to other
+   providers, resulting in a very small IPv6 global routing table.
+
+   For a single-homed site, this could work well.  A multihomed site
+   with only one upstream provider could also work well, although BGP
+   multihoming from a single upstream provider was often a premium
+   service (costing more than twice as much as two single-homed sites),
+   and if the single upstream provider went out of service, all of the
+   multihomed paths could fail simultaneously.
+
+   IPv4 sites often multihomed by obtaining Provider Independent
+   prefixes and advertising these prefixes through multiple upstream
+   providers.  With the assumption that any multihomed IPv4 site would
+   also multihome in IPv6, it seemed likely that IPv6 routing would be
+   subject to the same pressures to announce Provider Independent
+   prefixes, resulting in an IPv6 global routing table that exhibited
+   the same explosive growth as the IPv4 global routing table.  During
+   the early 2000s, work began on a protocol that would provide
+   multihoming for IPv6 sites without requiring sites to advertise
+   Provider Independent prefixes into the IPv6 global routing table.
+
+   This protocol, called "Shim6", allowed two endpoints to exchange
+   multiple addresses ("Locators") that all mapped to the same endpoint
+   ("Identity").  After an endpoint learned multiple Locators for the
+   other endpoint, it could send to any of those Locators with the
+   expectation that those packets would all be delivered to the endpoint
+   with the same Identity.  Shim6 was an example of an "Identity/Locator
+   Split" protocol.
+
+   Shim6, as defined in [RFC5533] and related RFCs, provided a workable
+   solution for IPv6 multihoming using Provider Assigned prefixes,
+   including capability discovery and negotiation, and allowing end-to-
+   end application communication to continue even in the face of path
+   failure, because applications don't see Locator failures and continue
+   to communicate with the same Identity using a different Locator.
+
+6.6.1.  Reasons for Non-deployment
+
+   Note that the problem being addressed was "site multihoming", but
+   Shim6 was providing "host multihoming".  That meant that the decision
+   about what path would be used was under host control, not under edge
+   router control.
+
+   Although more work could have been done to provide a better technical
+   solution, the biggest impediments to Shim6 deployment were
+   operational and business considerations.  These impediments were
+   discussed at multiple network operator group meetings, including
+   [Shim6-35] at [NANOG-35].
+
+   The technical issues centered around concerns that Shim6 relied on
+   the host to track all the connections, while also tracking Identity/
+   Locator mappings in the kernel and tracking failures to recognize
+   that an available path has failed.
+
+   The operational issues centered around concerns that operators were
+   performing traffic engineering on traffic aggregates.  With Shim6,
+   these operator traffic engineering policies must be pushed down to
+   individual hosts.
+
+   In addition, operators would have no visibility or control over the
+   decision of hosts choosing to switch to another path.  They expressed
+   concerns that relying on hosts to steer traffic exposed operator
+   networks to oscillation based on feedback loops, if hosts moved from
+   path to path frequently.  Given that Shim6 was intended to support
+   multihoming across operators, operators providing only one of the
+   paths would have even less visibility as traffic suddenly appeared
+   and disappeared on their networks.
+
+   In addition, firewalls that expected to find a TCP or UDP transport-
+   level protocol header in the IP payload would see a Shim6 Identity
+   header instead, and they would not perform transport-protocol-based
+   firewalling functions because the firewall's normal processing logic
+   would not look past the Identity header.  The firewall would perform
+   its default action, which would most likely be to drop packets that
+   don't match any processing rule.
+
+   The business issues centered on reducing or removing the ability to
+   sell BGP multihoming service to their own customers, which is often
+   more expensive than two single-homed connectivity services.
+
+6.6.2.  Lessons Learned
+
+   It is extremely important to take operational concerns into account
+   when a Path Aware protocol is making decisions about path selection
+   that may conflict with existing operational practices and business
+   considerations.
+
+6.6.3.  Addendum on Multipath TCP
+
+   During discussions in the PANRG session at IETF 103 [PANRG-103-Min],
+   Lars Eggert, past Transport Area Director, pointed out that during
+   charter discussions for the Multipath TCP Working Group [MP-TCP],
+   operators expressed concerns that customers could use Multipath TCP
+   to load-share TCP connections across operators simultaneously and
+   compare passive performance measurements across network paths in real
+   time, changing the balance of power in those business relationships.
+   Although the Multipath TCP Working Group was chartered, this concern
+   could have acted as an obstacle to deployment.
+
+   Operator objections to Shim6 were focused on technical concerns, but
+   this concern could have also been an obstacle to Shim6 deployment if
+   the technical concerns had been overcome.
+
+6.7.  Next Steps in Signaling (NSIS)
+
+   The suggested references for Next Steps in Signaling (NSIS) are:
+
+   *  the concluded working group charter [NSIS-CHARTER-2001]
+
+   *  "GIST: General Internet Signalling Transport" [RFC5971]
+
+   *  "NAT/Firewall NSIS Signaling Layer Protocol (NSLP)" [RFC5973]
+
+   *  "NSIS Signaling Layer Protocol (NSLP) for Quality-of-Service
+      Signaling" [RFC5974]
+
+   *  "Authorization for NSIS Signaling Layer Protocols" [RFC5981]
+
+   The NSIS Working Group worked on signaling techniques for network-
+   layer resources (e.g., QoS resource reservations, Firewall and NAT
+   traversal).
+
+   When RSVP [RFC2205] was used in deployments, a number of questions
+   came up about its perceived limitations and potential missing
+   features.  The issues noted in the NSIS Working Group charter
+   [NSIS-CHARTER-2001] include interworking between domains with
+   different QoS architectures, mobility and roaming for IP interfaces,
+   and complexity.  Later, the lack of security in RSVP was also
+   recognized [RFC4094].
+
+   The NSIS Working Group was chartered to tackle those issues and
+   initially focused on QoS signaling as its primary use case.  However,
+   over time a new approach evolved that introduced a modular
+   architecture using two application-specific signaling protocols: a)
+   the NSIS Signaling Layer Protocol (NSLP) on top of b) a generic
+   signaling transport protocol (the NSIS Transport Layer Protocol
+   (NTLP)).
+
+   NTLP is defined in [RFC5971].  Two types of NSLPs are defined: an
+   NSLP for QoS signaling [RFC5974] and an NSLP for NATs/firewalls
+   [RFC5973].
+
+6.7.1.  Reasons for Non-deployment
+
+   The obstacles for deployment can be grouped into implementation-
+   related aspects and operational aspects.
+
+   *  Implementation-related aspects:
+
+      Although NSIS provides benefits with respect to flexibility,
+      mobility, and security compared to other network signaling
+      techniques, hardware vendors were reluctant to deploy this
+      solution, because it would require additional implementation
+      effort and would result in additional complexity for router
+      implementations.
+
+      NTLP mainly operates as a path-coupled signaling protocol, i.e.,
+      its messages are processed at the control plane of each
+      intermediate node that is also forwarding the data flows.  This
+      requires a mechanism to intercept signaling packets while they are
+      forwarded in the same manner (especially along the same path) as
+      data packets.  NSIS uses the IPv4 and IPv6 Router Alert Option
+      (RAO) to allow for interception of those path-coupled signaling
+      messages, and this technique requires router implementations to
+      correctly understand and implement the handling of RAOs, e.g., to
+      only process packets with RAOs of interest and to leave packets
+      with irrelevant RAOs in the fast forwarding processing path (a
+      comprehensive discussion of these issues can be found in
+      [RFC6398]).  The latter was an issue with some router
+      implementations at the time of standardization.
+
+      Another reason is that path-coupled signaling protocols that
+      interact with routers and request manipulation of state at these
+      routers (or any other network element in general) are under
+      scrutiny: a packet (or sequence of packets) out of the mainly
+      untrusted data path is requesting creation and manipulation of
+      network state.  This is seen as potentially dangerous (e.g., opens
+      up a DoS threat to a router's control plane) and difficult for an
+      operator to control.  Path-coupled signaling approaches were
+      considered problematic (see also Section 3 of [RFC6398]).  There
+      are recommendations on how to secure NSIS nodes and deployments
+      (e.g., [RFC5981]).
+
+   *  Operational Aspects:
+
+      NSIS not only required trust between customers and their provider,
+      but also among different providers.  In particular, QoS signaling
+      techniques would require some kind of dynamic SLA support that
+      would imply (potentially quite complex) bilateral negotiations
+      between different Internet Service Providers.  This complexity was
+      not considered to be justified, and increasing the bandwidth (and
+      thus avoiding bottlenecks) was cheaper than actively managing
+      network resource bottlenecks by using path-coupled QoS signaling
+      techniques.  Furthermore, an end-to-end path typically involves
+      several provider domains, and these providers need to closely
+      cooperate in cases of failures.
+
+6.7.2.  Lessons Learned
+
+   One goal of NSIS was to decrease the complexity of the signaling
+   protocol, but a path-coupled signaling protocol comes with the
+   intrinsic complexity of IP-based networks, beyond the complexity of
+   the signaling protocol itself.  Sources of intrinsic complexity
+   include:
+
+   *  the presence of asymmetric routes between endpoints and routers.
+
+   *  the lack of security and trust at large in the Internet
+      infrastructure.
+
+   *  the presence of different trust boundaries.
+
+   *  the effects of best-effort networks (e.g., robustness to packet
+      loss).
+
+   *  divergence from the fate-sharing principle (e.g., state within the
+      network).
+
+   Any path-coupled signaling protocol has to deal with these realities.
+
+   Operators view the use of IPv4 and IPv6 Router Alert Options (RAOs)
+   to signal routers along the path from end systems with suspicion,
+   because these end systems are usually not authenticated and heavy use
+   of RAOs can easily increase the CPU load on routers that are designed
+   to process most packets using a hardware "fast path" and diverting
+   packets containing RAOs to a slower, more capable processor.
+
+6.8.  IPv6 Flow Labels
+
+   The suggested reference for IPv6 Flow Labels is:
+
+   *  "IPv6 Flow Label Specification" [RFC6437]
+
+   IPv6 specifies a 20-bit Flow Label field [RFC6437], included in the
+   fixed part of the IPv6 header and hence present in every IPv6 packet.
+   An endpoint sets the value in this field to one of a set of
+   pseudorandomly assigned values.  If a packet is not part of any flow,
+   the flow label value is set to zero [RFC3697].  A number of Standards
+   Track and Best Current Practice RFCs (e.g., [RFC8085], [RFC6437],
+   [RFC6438]) encourage IPv6 endpoints to set a non-zero value in this
+   field.  A multiplexing transport could choose to use multiple flow
+   labels to allow the network to either independently forward its
+   subflows or use one common value for the traffic aggregate.  The flow
+   label is present in all fragments.  IPsec was originally put forward
+   as one important use case for this mechanism and does encrypt the
+   field [RFC6438].
+
+   Once set, the flow label can provide information that can help inform
+   network nodes about subflows present at the transport layer, without
+   needing to interpret the setting of upper-layer protocol fields
+   [RFC6294].  This information can also be used to coordinate how
+   aggregates of transport subflows are grouped when queued in the
+   network and to select appropriate per-flow forwarding when choosing
+   between alternate paths [RFC6438] (e.g., for Equal-Cost Multipath
+   (ECMP) routing and Link Aggregation Groups (LAGs)).
+
+6.8.1.  Reasons for Non-deployment
+
+   Despite the field being present in every IPv6 packet, the mechanism
+   did not receive as much use as originally envisioned.  One reason is
+   that to be useful it requires engagement by two different
+   stakeholders:
+
+   *  Endpoint Implementation:
+
+      For network nodes along a path to utilize the flow label, there
+      needs to be a non-zero value inserted in the field [RFC6437] at
+      the sending endpoint.  There needs to be an incentive for an
+      endpoint to set an appropriate non-zero value.  The value should
+      appropriately reflect the level of aggregation the traffic expects
+      to be provided by the network.  However, this requires the stack
+      to know granularity at which flows should be identified (or,
+      conversely, which flows should receive aggregated treatment),
+      i.e., which packets carry the same flow label.  Therefore, setting
+      a non-zero value may result in additional choices that need to be
+      made by an application developer.
+
+      Although the original flow label standard [RFC3697] forbids any
+      encoding of meaning into the flow label value, the opportunity to
+      use the flow label as a covert channel or to signal other meta-
+      information may have raised concerns about setting a non-zero
+      value [RFC6437].
+
+      Before methods are widely deployed to use this method, there could
+      be no incentive for an endpoint to set the field.
+
+   *  Operational support in network nodes:
+
+      A benefit can only be realized when a network node along the path
+      also uses this information to inform its decisions.  Network
+      equipment (routers and/or middleboxes) need to include appropriate
+      support in order to utilize the field when making decisions about
+      how to classify flows or forward packets.  The use of any optional
+      feature in a network node also requires corresponding updates to
+      operational procedures and therefore is normally only introduced
+      when the cost can be justified.
+
+      A benefit from utilizing the flow label is expected to be
+      increased quality of experience for applications -- but this comes
+      at some operational cost to an operator and requires endpoints to
+      set the field.
+
+6.8.2.  Lessons Learned
+
+   The flow label is a general-purpose header field for use by the path.
+   Multiple uses have been proposed.  One candidate use was to reduce
+   the complexity of forwarding decisions.  However, modern routers can
+   use a "fast path", often taking advantage of hardware to accelerate
+   processing.  The method can assist in more complex forwarding, such
+   as ECMP routing and load balancing.
+
+   Although [RFC6437] recommended that endpoints should by default
+   choose uniformly distributed labels for their traffic, the
+   specification permitted an endpoint to choose to set a zero value.
+   This ability of endpoints to choose to set a flow label of zero has
+   had consequences on deployability:
+
+   *  Before wide-scale support by endpoints, it would be impossible to
+      rely on a non-zero flow label being set.  Network nodes therefore
+      would need to also employ other techniques to realize equivalent
+      functions.  An example of a method is one assuming semantics of
+      the source port field to provide entropy input to a network-layer
+      hash.  This use of a 5-tuple to classify a packet represents a
+      layering violation [RFC6294].  When other methods have been
+      deployed, they increase the cost of deploying standards-based
+      methods, even though they may offer less control to endpoints and
+      result in potential interaction with other uses/interpretation of
+      the field.
+
+   *  Even though the flow label is specified as an end-to-end field,
+      some network paths have been observed to not transparently forward
+      the flow label.  This could result from non-conformant equipment
+      or could indicate that some operational networks have chosen to
+      reuse the protocol field for other (e.g., internal) purposes.
+      This results in lack of transparency, and a deployment hurdle to
+      endpoints expecting that they can set a flow label that is
+      utilized by the network.  The more recent practice of "greasing"
+      [GREASE] would suggest that a different outcome could have been
+      achieved if endpoints were always required to set a non-zero
+      value.
+
+   *  [RFC1809] noted that setting the choice of the flow label value
+      can depend on the expectations of the traffic generated by an
+      application, which suggests that an API should be presented to
+      control the setting or policy that is used.  However, many
+      currently available APIs do not have this support.
+
+   A growth in the use of encrypted transports (e.g., QUIC [RFC9000])
+   seems likely to raise issues similar to those discussed above and
+   could motivate renewed interest in utilizing the flow label.
+
+6.9.  Explicit Congestion Notification (ECN)
+
+   The suggested references for Explicit Congestion Notification (ECN)
+   are:
+
+   *  "Recommendations on Queue Management and Congestion Avoidance in
+      the Internet" [RFC2309]
+
+   *  "A Proposal to add Explicit Congestion Notification (ECN) to IP"
+      [RFC2481]
+
+   *  "The Addition of Explicit Congestion Notification (ECN) to IP"
+      [RFC3168]
+
+   *  "Implementation Report on Experiences with Various TCP RFCs"
+      [vista-impl], slides 6 and 7
+
+   *  "Implementation and Deployment of ECN" (at [SallyFloyd])
+
+   In the early 1990s, the large majority of Internet traffic used TCP
+   as its transport protocol, but TCP had no way to detect path
+   congestion before the path was so congested that packets were being
+   dropped.  These congestion events could affect all senders using a
+   path, either by "lockout", where long-lived flows monopolized the
+   queues along a path, or by "full queues", where queues remain full,
+   or almost full, for a long period of time.
+
+   In response to this situation, "Active Queue Management" (AQM) was
+   deployed in the network.  A number of AQM disciplines have been
+   deployed, but one common approach was that routers dropped packets
+   when a threshold buffer length was reached, so that transport
+   protocols like TCP that were responsive to loss would detect this
+   loss and reduce their sending rates.  Random Early Detection (RED)
+   was one such proposal in the IETF.  As the name suggests, a router
+   using RED as its AQM discipline that detected time-averaged queue
+   lengths passing a threshold would choose incoming packets
+   probabilistically to be dropped [RFC2309].
+
+   Researchers suggested providing "explicit congestion notifications"
+   to senders when routers along the path detected that their queues
+   were building, giving senders an opportunity to "slow down" as if a
+   loss had occurred, giving path queues time to drain, while the path
+   still had sufficient buffer capacity to accommodate bursty arrivals
+   of packets from other senders.  This was proposed as an experiment in
+   [RFC2481] and standardized in [RFC3168].
+
+   A key aspect of ECN was the use of IP header fields rather than IP
+   options to carry explicit congestion notifications, since the
+   proponents recognized that
+
+      Many routers process the "regular" headers in IP packets more
+      efficiently than they process the header information in IP
+      options.
+
+   Unlike most of the Path Aware technologies included in this document,
+   the story of ECN continues to the present day and encountered a large
+   number of Lessons Learned during that time.  The early history of ECN
+   (non-)deployment provides Lessons Learned that were not captured by
+   other contributions in Section 6, so that is the emphasis in this
+   section of the document.
+
+6.9.1.  Reasons for Non-deployment
+
+   ECN deployment relied on three factors -- support in client
+   implementations, support in router implementations, and deployment
+   decisions in operational networks.
+
+   The proponents of ECN did so much right, anticipating many of the
+   Lessons Learned now recognized in Section 4.  They recognized the
+   need to support incremental deployment (Section 4.2).  They
+   considered the impact on router throughput (Section 4.8).  They even
+   considered trust issues between end nodes and the network, for both
+   non-compliant end nodes (Section 4.10) and non-compliant routers
+   (Section 4.9).
+
+   They were rewarded with ECN being implemented in major operating
+   systems, for both end nodes and routers.  A number of implementations
+   are listed under "Implementation and Deployment of ECN" at
+   [SallyFloyd].
+
+   What they did not anticipate was routers that would crash when they
+   saw bits 6 and 7 in the IPv4 Type of Service (TOS) octet [RFC0791] /
+   IPv6 Traffic Class field [RFC2460], which [RFC2481] redefined to be
+   "Currently Unused", being set to a non-zero value.
+
+   As described in [vista-impl] ("IGD" stands for "Intermediate Gateway
+   Device"),
+
+   |  IGD problem #1: one of the most popular versions from one of the
+   |  most popular vendors.  When a data packet arrives with either
+   |  ECT(0) or ECT(1) (indicating successful ECN capability
+   |  negotiation) indicated, router crashed.  Cannot be recovered at
+   |  TCP layer [sic]
+
+   This implementation, which would be run on a significant percentage
+   of Internet end nodes, was shipped with ECN disabled, as was true for
+   several of the other implementations listed under "Implementation and
+   Deployment of ECN" at [SallyFloyd].  Even if subsequent router
+   vendors fixed these implementations, ECN was still disabled on end
+   nodes, and given the trade-off between the benefits of enabling ECN
+   (somewhat better behavior during congestion) and the risks of
+   enabling ECN (possibly crashing a router somewhere along the path),
+   ECN tended to stay disabled on implementations that supported ECN for
+   decades afterwards.
+
+6.9.2.  Lessons Learned
+
+   Of the contributions included in Section 6, ECN may be unique in
+   providing these lessons:
+
+   *  Even if you do everything right, you may trip over implementation
+      bugs in devices you know nothing about, that will cause severe
+      problems that prevent successful deployment of your Path Aware
+      technology.
+
+   *  After implementations disable your Path Aware technology, it may
+      take years, or even decades, to convince implementers to re-enable
+      it by default.
+
+   These two lessons, taken together, could be summarized as "you get
+   one chance to get it right."
+
+   During discussion of ECN at [PANRG-110], we noted that "you get one
+   chance to get it right" isn't quite correct today, because operating
+   systems on so many host systems are frequently updated, and transport
+   protocols like QUIC [RFC9000] are being implemented in user space and
+   can be updated without touching installed operating systems.  Neither
+   of these factors were true in the early 2000s.
+
+   We think that these restatements of the ECN Lessons Learned are more
+   useful for current implementers:
+
+   *  Even if you do everything right, you may trip over implementation
+      bugs in devices you know nothing about, that will cause severe
+      problems that prevent successful deployment of your Path Aware
+      technology.  Testing before deployment isn't enough to ensure
+      successful deployment.  It is also necessary to "deploy gently",
+      which often means deploying for a small subset of users to gain
+      experience and implementing feedback mechanisms to detect that
+      user experience is being degraded.
+
+   *  After implementations disable your Path Aware technology, it may
+      take years, or even decades, to convince implementers to re-enable
+      it by default.  This might be based on the difficulty of
+      distributing implementations that enable it by default, but it is
+      just as likely to be based on the "bad taste in the mouth" that
+      implementers have after an unsuccessful deployment attempt that
+      degraded user experience.
+
+   With these expansions, the two lessons, taken together, could be more
+   helpfully summarized as "plan for failure" -- anticipate what your
+   next step will be, if initial deployment is unsuccessful.
+
+   ECN deployment was also hindered by non-deployment of AQM in many
+   devices, because of operator interest in QoS features provided in the
+   network, rather than using the network to assist end systems in
+   providing for themselves.  But that's another story, and the AQM
+   Lessons Learned are already covered in other contributions in
+   Section 6.
+
+7.  Security Considerations
+
+   This document describes Path Aware techniques that were not adopted
+   and widely deployed on the Internet, so it doesn't affect the
+   security of the Internet.
+
+   If this document meets its goals, we may develop new techniques for
+   Path Aware networking that would affect the security of the Internet,
+   but security considerations for those techniques will be described in
+   the corresponding RFCs that specify them.
+
+8.  IANA Considerations
+
+   This document has no IANA actions.
+
+9.  Informative References
+
+   [Colossal-Cave]
+              Wikipedia, "Colossal Cave Adventure", June 2021,
+              <https://en.wikipedia.org/w/
+              index.php?title=Colossal_Cave_Adventure&oldid=1027119625>.
+
+   [Conviva]  "Conviva Precision : Data Sheet", January 2021,
+              <https://www.conviva.com/datasheets/precision-delivery-
+              intelligence/>.
+
+   [FARRELL-ETM]
+              Farrell, S., "We're gonna need a bigger threat model",
+              Work in Progress, Internet-Draft, draft-farrell-etm-03, 6
+              July 2019, <https://datatracker.ietf.org/doc/html/draft-
+              farrell-etm-03>.
+
+   [GREASE]   Thomson, M., "Long-term Viability of Protocol Extension
+              Mechanisms", Work in Progress, Internet-Draft, draft-iab-
+              use-it-or-lose-it-00, 7 August 2019,
+              <https://datatracker.ietf.org/doc/html/draft-iab-use-it-
+              or-lose-it-00>.
+
+   [IEN-119]  Forgie, J., "ST - A Proposed Internet Stream Protocol",
+              September 1979,
+              <https://www.rfc-editor.org/ien/ien119.txt>.
+
+   [INTERNET-THREAT-MODEL]
+              Arkko, J., "Changes in the Internet Threat Model", Work in
+              Progress, Internet-Draft, draft-arkko-arch-internet-
+              threat-model-01, 8 July 2019,
+              <https://datatracker.ietf.org/doc/html/draft-arkko-arch-
+              internet-threat-model-01>.
+
+   [INTSERV-MULTIPLE-TSPEC]
+              Polk, J. and S. Dhesikan, "Integrated Services (IntServ)
+              Extension to Allow Signaling of Multiple Traffic
+              Specifications and Multiple Flow Specifications in
+              RSVPv1", Work in Progress, Internet-Draft, draft-ietf-
+              tsvwg-intserv-multiple-tspec-02, 25 February 2013,
+              <https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-
+              intserv-multiple-tspec-02>.
+
+   [ITAT]     "IAB Workshop on Internet Technology Adoption and
+              Transition (ITAT) 2013", December 2013,
+              <https://www.iab.org/activities/workshops/itat/>.
+
+   [model-t]  "Model-t -- Discussions of changes in Internet deployment
+              patterns and their impact on the Internet threat model",
+              model-t mailing list,
+              <https://www.iab.org/mailman/listinfo/model-t>.
+
+   [MOPS-109-Min]
+              "Media Operations Working Group - IETF 109 Minutes",
+              November 2020,
+              <https://datatracker.ietf.org/meeting/109/materials/
+              minutes-109-mops-00>.
+
+   [MP-TCP]   "Multipath TCP Working Group Home Page",
+              <https://datatracker.ietf.org/wg/mptcp/>.
+
+   [NANOG-35] "NANOG 35 Agenda", North American Network Operators' Group
+              (NANOG), October 2005,
+              <https://archive.nanog.org/meetings/nanog35/agenda>.
+
+   [NSIS-CHARTER-2001]
+              "Next Steps In Signaling Working Group Charter", March
+              2011,
+              <https://datatracker.ietf.org/doc/charter-ietf-nsis/>.
+
+   [PANRG]    "Path Aware Networking Research Group Home Page",
+              <https://irtf.org/panrg>.
+
+   [PANRG-103-Min]
+              "Path Aware Networking Research Group - IETF 103 Minutes",
+              November 2018,
+              <https://datatracker.ietf.org/doc/minutes-103-panrg/>.
+
+   [PANRG-105-Min]
+              "Path Aware Networking Research Group - IETF 105 Minutes",
+              July 2019,
+              <https://datatracker.ietf.org/doc/minutes-105-panrg/>.
+
+   [PANRG-106-Min]
+              "Path Aware Networking Research Group - IETF 106 Minutes",
+              November 2019,
+              <https://datatracker.ietf.org/doc/minutes-106-panrg/>.
+
+   [PANRG-110]
+              "Path Aware Networking Research Group - IETF 110", March
+              2021,
+              <https://datatracker.ietf.org/meeting/110/session/panrg>.
+
+   [PANRG-99] "Path Aware Networking Research Group - IETF 99", July
+              2017,
+              <https://datatracker.ietf.org/meeting/99/session/panrg>.
+
+   [PANRG-PATH-PROPERTIES]
+              Enghardt, T. and C. Krähenbühl, "A Vocabulary of Path
+              Properties", Work in Progress, Internet-Draft, draft-irtf-
+              panrg-path-properties-02, 22 February 2021,
+              <https://datatracker.ietf.org/doc/html/draft-irtf-panrg-
+              path-properties-02>.
+
+   [PANRG-QUESTIONS]
+              Trammell, B., "Current Open Questions in Path Aware
+              Networking", Work in Progress, Internet-Draft, draft-irtf-
+              panrg-questions-09, 16 April 2021,
+              <https://datatracker.ietf.org/doc/html/draft-irtf-panrg-
+              questions-09>.
+
+   [PATH-Decade]
+              Bonaventure, O., "A Decade of Path Awareness", July 2017,
+              <https://datatracker.ietf.org/doc/slides-99-panrg-a-
+              decade-of-path-awareness/>.
+
+   [QS-SAT]   Secchi, R., Sathiaseelan, A., Potortì, F., Gotta, A., and
+              G. Fairhurst, "Using Quick-Start to enhance TCP-friendly
+              rate control performance in bidirectional satellite
+              networks", DOI 10.1002/sat.929, May 2009,
+              <https://dl.acm.org/citation.cfm?id=3160304.3160305>.
+
+   [RFC0791]  Postel, J., "Internet Protocol", STD 5, RFC 791,
+              DOI 10.17487/RFC0791, September 1981,
+              <https://www.rfc-editor.org/info/rfc791>.
+
+   [RFC0792]  Postel, J., "Internet Control Message Protocol", STD 5,
+              RFC 792, DOI 10.17487/RFC0792, September 1981,
+              <https://www.rfc-editor.org/info/rfc792>.
+
+   [RFC0793]  Postel, J., "Transmission Control Protocol", STD 7,
+              RFC 793, DOI 10.17487/RFC0793, September 1981,
+              <https://www.rfc-editor.org/info/rfc793>.
+
+   [RFC1016]  Prue, W. and J. Postel, "Something a Host Could Do with
+              Source Quench: The Source Quench Introduced Delay
+              (SQuID)", RFC 1016, DOI 10.17487/RFC1016, July 1987,
+              <https://www.rfc-editor.org/info/rfc1016>.
+
+   [RFC1122]  Braden, R., Ed., "Requirements for Internet Hosts -
+              Communication Layers", STD 3, RFC 1122,
+              DOI 10.17487/RFC1122, October 1989,
+              <https://www.rfc-editor.org/info/rfc1122>.
+
+   [RFC1190]  Topolcic, C., "Experimental Internet Stream Protocol:
+              Version 2 (ST-II)", RFC 1190, DOI 10.17487/RFC1190,
+              October 1990, <https://www.rfc-editor.org/info/rfc1190>.
+
+   [RFC1633]  Braden, R., Clark, D., and S. Shenker, "Integrated
+              Services in the Internet Architecture: an Overview",
+              RFC 1633, DOI 10.17487/RFC1633, June 1994,
+              <https://www.rfc-editor.org/info/rfc1633>.
+
+   [RFC1809]  Partridge, C., "Using the Flow Label Field in IPv6",
+              RFC 1809, DOI 10.17487/RFC1809, June 1995,
+              <https://www.rfc-editor.org/info/rfc1809>.
+
+   [RFC1819]  Delgrossi, L., Ed. and L. Berger, Ed., "Internet Stream
+              Protocol Version 2 (ST2) Protocol Specification - Version
+              ST2+", RFC 1819, DOI 10.17487/RFC1819, August 1995,
+              <https://www.rfc-editor.org/info/rfc1819>.
+
+   [RFC1887]  Rekhter, Y., Ed. and T. Li, Ed., "An Architecture for IPv6
+              Unicast Address Allocation", RFC 1887,
+              DOI 10.17487/RFC1887, December 1995,
+              <https://www.rfc-editor.org/info/rfc1887>.
+
+   [RFC2001]  Stevens, W., "TCP Slow Start, Congestion Avoidance, Fast
+              Retransmit, and Fast Recovery Algorithms", RFC 2001,
+              DOI 10.17487/RFC2001, January 1997,
+              <https://www.rfc-editor.org/info/rfc2001>.
+
+   [RFC2205]  Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S.
+              Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1
+              Functional Specification", RFC 2205, DOI 10.17487/RFC2205,
+              September 1997, <https://www.rfc-editor.org/info/rfc2205>.
+
+   [RFC2210]  Wroclawski, J., "The Use of RSVP with IETF Integrated
+              Services", RFC 2210, DOI 10.17487/RFC2210, September 1997,
+              <https://www.rfc-editor.org/info/rfc2210>.
+
+   [RFC2211]  Wroclawski, J., "Specification of the Controlled-Load
+              Network Element Service", RFC 2211, DOI 10.17487/RFC2211,
+              September 1997, <https://www.rfc-editor.org/info/rfc2211>.
+
+   [RFC2212]  Shenker, S., Partridge, C., and R. Guerin, "Specification
+              of Guaranteed Quality of Service", RFC 2212,
+              DOI 10.17487/RFC2212, September 1997,
+              <https://www.rfc-editor.org/info/rfc2212>.
+
+   [RFC2215]  Shenker, S. and J. Wroclawski, "General Characterization
+              Parameters for Integrated Service Network Elements",
+              RFC 2215, DOI 10.17487/RFC2215, September 1997,
+              <https://www.rfc-editor.org/info/rfc2215>.
+
+   [RFC2309]  Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering,
+              S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G.,
+              Partridge, C., Peterson, L., Ramakrishnan, K., Shenker,
+              S., Wroclawski, J., and L. Zhang, "Recommendations on
+              Queue Management and Congestion Avoidance in the
+              Internet", RFC 2309, DOI 10.17487/RFC2309, April 1998,
+              <https://www.rfc-editor.org/info/rfc2309>.
+
+   [RFC2460]  Deering, S. and R. Hinden, "Internet Protocol, Version 6
+              (IPv6) Specification", RFC 2460, DOI 10.17487/RFC2460,
+              December 1998, <https://www.rfc-editor.org/info/rfc2460>.
+
+   [RFC2475]  Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z.,
+              and W. Weiss, "An Architecture for Differentiated
+              Services", RFC 2475, DOI 10.17487/RFC2475, December 1998,
+              <https://www.rfc-editor.org/info/rfc2475>.
+
+   [RFC2481]  Ramakrishnan, K. and S. Floyd, "A Proposal to add Explicit
+              Congestion Notification (ECN) to IP", RFC 2481,
+              DOI 10.17487/RFC2481, January 1999,
+              <https://www.rfc-editor.org/info/rfc2481>.
+
+   [RFC3168]  Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
+              of Explicit Congestion Notification (ECN) to IP",
+              RFC 3168, DOI 10.17487/RFC3168, September 2001,
+              <https://www.rfc-editor.org/info/rfc3168>.
+
+   [RFC3697]  Rajahalme, J., Conta, A., Carpenter, B., and S. Deering,
+              "IPv6 Flow Label Specification", RFC 3697,
+              DOI 10.17487/RFC3697, March 2004,
+              <https://www.rfc-editor.org/info/rfc3697>.
+
+   [RFC4094]  Manner, J. and X. Fu, "Analysis of Existing Quality-of-
+              Service Signaling Protocols", RFC 4094,
+              DOI 10.17487/RFC4094, May 2005,
+              <https://www.rfc-editor.org/info/rfc4094>.
+
+   [RFC4443]  Conta, A., Deering, S., and M. Gupta, Ed., "Internet
+              Control Message Protocol (ICMPv6) for the Internet
+              Protocol Version 6 (IPv6) Specification", STD 89,
+              RFC 4443, DOI 10.17487/RFC4443, March 2006,
+              <https://www.rfc-editor.org/info/rfc4443>.
+
+   [RFC4782]  Floyd, S., Allman, M., Jain, A., and P. Sarolahti, "Quick-
+              Start for TCP and IP", RFC 4782, DOI 10.17487/RFC4782,
+              January 2007, <https://www.rfc-editor.org/info/rfc4782>.
+
+   [RFC5082]  Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C.
+              Pignataro, "The Generalized TTL Security Mechanism
+              (GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007,
+              <https://www.rfc-editor.org/info/rfc5082>.
+
+   [RFC5218]  Thaler, D. and B. Aboba, "What Makes for a Successful
+              Protocol?", RFC 5218, DOI 10.17487/RFC5218, July 2008,
+              <https://www.rfc-editor.org/info/rfc5218>.
+
+   [RFC5533]  Nordmark, E. and M. Bagnulo, "Shim6: Level 3 Multihoming
+              Shim Protocol for IPv6", RFC 5533, DOI 10.17487/RFC5533,
+              June 2009, <https://www.rfc-editor.org/info/rfc5533>.
+
+   [RFC5575]  Marques, P., Sheth, N., Raszuk, R., Greene, B., Mauch, J.,
+              and D. McPherson, "Dissemination of Flow Specification
+              Rules", RFC 5575, DOI 10.17487/RFC5575, August 2009,
+              <https://www.rfc-editor.org/info/rfc5575>.
+
+   [RFC5681]  Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
+              Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
+              <https://www.rfc-editor.org/info/rfc5681>.
+
+   [RFC5971]  Schulzrinne, H. and R. Hancock, "GIST: General Internet
+              Signalling Transport", RFC 5971, DOI 10.17487/RFC5971,
+              October 2010, <https://www.rfc-editor.org/info/rfc5971>.
+
+   [RFC5973]  Stiemerling, M., Tschofenig, H., Aoun, C., and E. Davies,
+              "NAT/Firewall NSIS Signaling Layer Protocol (NSLP)",
+              RFC 5973, DOI 10.17487/RFC5973, October 2010,
+              <https://www.rfc-editor.org/info/rfc5973>.
+
+   [RFC5974]  Manner, J., Karagiannis, G., and A. McDonald, "NSIS
+              Signaling Layer Protocol (NSLP) for Quality-of-Service
+              Signaling", RFC 5974, DOI 10.17487/RFC5974, October 2010,
+              <https://www.rfc-editor.org/info/rfc5974>.
+
+   [RFC5981]  Manner, J., Stiemerling, M., Tschofenig, H., and R. Bless,
+              Ed., "Authorization for NSIS Signaling Layer Protocols",
+              RFC 5981, DOI 10.17487/RFC5981, February 2011,
+              <https://www.rfc-editor.org/info/rfc5981>.
+
+   [RFC6294]  Hu, Q. and B. Carpenter, "Survey of Proposed Use Cases for
+              the IPv6 Flow Label", RFC 6294, DOI 10.17487/RFC6294, June
+              2011, <https://www.rfc-editor.org/info/rfc6294>.
+
+   [RFC6398]  Le Faucheur, F., Ed., "IP Router Alert Considerations and
+              Usage", BCP 168, RFC 6398, DOI 10.17487/RFC6398, October
+              2011, <https://www.rfc-editor.org/info/rfc6398>.
+
+   [RFC6437]  Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme,
+              "IPv6 Flow Label Specification", RFC 6437,
+              DOI 10.17487/RFC6437, November 2011,
+              <https://www.rfc-editor.org/info/rfc6437>.
+
+   [RFC6438]  Carpenter, B. and S. Amante, "Using the IPv6 Flow Label
+              for Equal Cost Multipath Routing and Link Aggregation in
+              Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011,
+              <https://www.rfc-editor.org/info/rfc6438>.
+
+   [RFC6633]  Gont, F., "Deprecation of ICMP Source Quench Messages",
+              RFC 6633, DOI 10.17487/RFC6633, May 2012,
+              <https://www.rfc-editor.org/info/rfc6633>.
+
+   [RFC6928]  Chu, J., Dukkipati, N., Cheng, Y., and M. Mathis,
+              "Increasing TCP's Initial Window", RFC 6928,
+              DOI 10.17487/RFC6928, April 2013,
+              <https://www.rfc-editor.org/info/rfc6928>.
+
+   [RFC7305]  Lear, E., Ed., "Report from the IAB Workshop on Internet
+              Technology Adoption and Transition (ITAT)", RFC 7305,
+              DOI 10.17487/RFC7305, July 2014,
+              <https://www.rfc-editor.org/info/rfc7305>.
+
+   [RFC7418]  Dawkins, S., Ed., "An IRTF Primer for IETF Participants",
+              RFC 7418, DOI 10.17487/RFC7418, December 2014,
+              <https://www.rfc-editor.org/info/rfc7418>.
+
+   [RFC8085]  Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage
+              Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085,
+              March 2017, <https://www.rfc-editor.org/info/rfc8085>.
+
+   [RFC8170]  Thaler, D., Ed., "Planning for Protocol Adoption and
+              Subsequent Transitions", RFC 8170, DOI 10.17487/RFC8170,
+              May 2017, <https://www.rfc-editor.org/info/rfc8170>.
+
+   [RFC8655]  Finn, N., Thubert, P., Varga, B., and J. Farkas,
+              "Deterministic Networking Architecture", RFC 8655,
+              DOI 10.17487/RFC8655, October 2019,
+              <https://www.rfc-editor.org/info/rfc8655>.
+
+   [RFC8793]  Wissingh, B., Wood, C., Afanasyev, A., Zhang, L., Oran,
+              D., and C. Tschudin, "Information-Centric Networking
+              (ICN): Content-Centric Networking (CCNx) and Named Data
+              Networking (NDN) Terminology", RFC 8793,
+              DOI 10.17487/RFC8793, June 2020,
+              <https://www.rfc-editor.org/info/rfc8793>.
+
+   [RFC9000]  Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based
+              Multiplexed and Secure Transport", RFC 9000,
+              DOI 10.17487/RFC9000, May 2021,
+              <https://www.rfc-editor.org/info/rfc9000>.
+
+   [SAAG-105-Min]
+              "Security Area Open Meeting - IETF 105 Minutes", July
+              2019, <https://datatracker.ietf.org/meeting/105/materials/
+              minutes-105-saag-00>.
+
+   [SAF07]    Sarolahti, P., Allman, M., and S. Floyd, "Determining an
+              appropriate sending rate over an underutilized network
+              path", Computer Networks: The International Journal of
+              Computer and Telecommunications Networking, Volume 51,
+              Number 7, DOI 10.1016/j.comnet.2006.11.006, May 2007,
+              <https://dl.acm.org/doi/10.1016/j.comnet.2006.11.006>.
+
+   [SallyFloyd]
+              Floyd, S., "ECN (Explicit Congestion Notification) in TCP/
+              IP", June 2009, <https://www.icir.org/floyd/ecn.html>.
+
+   [Sch11]    Scharf, M., "Fast Startup Internet Congestion Control for
+              Broadband Interactive Applications", Ph.D. Thesis,
+              University of Stuttgart, April 2011.
+
+   [Shim6-35] Meyer, D., Huston, G., Schiller, J., and V. Gill, "IAB
+              IPv6 Multihoming Panel at NANOG 35", North American
+              Network Operators' Group (NANOG), October 2005,
+              <https://www.youtube.com/watch?v=ji6Y_rYHAQs>.
+
+   [TRIGTRAN-55]
+              "Triggers for Transport BOF at IETF 55", November 2002,
+              <https://www.ietf.org/proceedings/55/239.htm>.
+
+   [TRIGTRAN-56]
+              "Triggers for Transport BOF at IETF 56", March 2003,
+              <https://www.ietf.org/proceedings/56/251.htm>.
+
+   [vista-impl]
+              Sridharan, M., Bansal, D., and D. Thaler, "Implementation
+              Report on Experiences with Various TCP RFCs", March 2007,
+              <https://www.ietf.org/proceedings/68/slides/tsvarea-3/
+              sld1.htm>.
+
+Acknowledgments
+
+   Initial material for Section 6.1 on ST2 was provided by Gorry
+   Fairhurst.
+
+   Initial material for Section 6.2 on IntServ was provided by Ron
+   Bonica.
+
+   Initial material for Section 6.3 on Quick-Start TCP was provided by
+   Michael Scharf, who also provided suggestions to improve this section
+   after it was edited.
+
+   Initial material for Section 6.4 on ICMP Source Quench was provided
+   by Gorry Fairhurst.
+
+   Initial material for Section 6.5 on Triggers for Transport (TRIGTRAN)
+   was provided by Spencer Dawkins.
+
+   Section 6.6 on Shim6 builds on initial material describing obstacles,
+   which was provided by Erik Nordmark, with background added by Spencer
+   Dawkins.
+
+   Initial material for Section 6.7 on Next Steps in Signaling (NSIS)
+   was provided by Roland Bless and Martin Stiemerling.
+
+   Initial material for Section 6.8 on IPv6 Flow Labels was provided by
+   Gorry Fairhurst.
+
+   Initial material for Section 6.9 on Explicit Congestion Notification
+   was provided by Spencer Dawkins.
+
+   Our thanks to Adrian Farrel, Bob Briscoe, C.M. Heard, David Black,
+   Eric Kinnear, Erik Auerswald, Gorry Fairhurst, Jake Holland, Joe
+   Touch, Joeri de Ruiter, Kireeti Kompella, Mohamed Boucadair, Randy
+   Presuhn, Roland Bless, Ruediger Geib, Theresa Enghardt, and Wes Eddy,
+   who provided review comments on this document as a "work in process".
+
+   Mallory Knodel reviewed this document for the Internet Research
+   Steering Group and provided many helpful suggestions.
+
+   David Oran also provided helpful comments and text suggestions on
+   this document during Internet Research Steering Group balloting.  In
+   particular, Section 5 reflects his review.
+
+   Benjamin Kaduk, Martin Duke, and Rob Wilton provided helpful comments
+   during Internet Engineering Steering Group conflict review.
+
+   Special thanks to Adrian Farrel for helping Spencer navigate the
+   twisty little passages of Flow Specs and Filter Specs in IntServ,
+   RSVP, MPLS, and BGP.  They are all alike, except when they are
+   different [Colossal-Cave].
+
+Author's Address
+
+   Spencer Dawkins (editor)
+   Tencent America
+   United States of America
+
+   Email: spencerdawkins.ietf@gmail.com