doc: Add RFC documents

1 files changed, 1963 insertions, 0 deletions

diff --git a/doc/rfc/rfc7938.txt b/doc/rfc/rfc7938.txt
new file mode 100644
index 0000000..30d544b
--- /dev/null
+++ b/doc/rfc/rfc7938.txt
@@ -0,0 +1,1963 @@
+
+
+
+
+
+
+Internet Engineering Task Force (IETF)                       P. Lapukhov
+Request for Comments: 7938                                      Facebook
+Category: Informational                                        A. Premji
+ISSN: 2070-1721                                          Arista Networks
+                                                        J. Mitchell, Ed.
+                                                             August 2016
+
+
+           Use of BGP for Routing in Large-Scale Data Centers
+
+Abstract
+
+   Some network operators build and operate data centers that support
+   over one hundred thousand servers.  In this document, such data
+   centers are referred to as "large-scale" to differentiate them from
+   smaller infrastructures.  Environments of this scale have a unique
+   set of network requirements with an emphasis on operational
+   simplicity and network stability.  This document summarizes
+   operational experience in designing and operating large-scale data
+   centers using BGP as the only routing protocol.  The intent is to
+   report on a proven and stable routing design that could be leveraged
+   by others in the industry.
+
+Status of This Memo
+
+   This document is not an Internet Standards Track specification; it is
+   published for informational purposes.
+
+   This document is a product of the Internet Engineering Task Force
+   (IETF).  It represents the consensus of the IETF community.  It has
+   received public review and has been approved for publication by the
+   Internet Engineering Steering Group (IESG).  Not all documents
+   approved by the IESG are a candidate for any level of Internet
+   Standard; see Section 2 of RFC 7841.
+
+   Information about the current status of this document, any errata,
+   and how to provide feedback on it may be obtained at
+   http://www.rfc-editor.org/info/rfc7938.
+
+
+
+
+
+
+
+
+
+
+
+
+
+Lapukhov, et al.              Informational                     [Page 1]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+Copyright Notice
+
+   Copyright (c) 2016 IETF Trust and the persons identified as the
+   document authors.  All rights reserved.
+
+   This document is subject to BCP 78 and the IETF Trust's Legal
+   Provisions Relating to IETF Documents
+   (http://trustee.ietf.org/license-info) in effect on the date of
+   publication of this document.  Please review these documents
+   carefully, as they describe your rights and restrictions with respect
+   to this document.  Code Components extracted from this document must
+   include Simplified BSD License text as described in Section 4.e of
+   the Trust Legal Provisions and are provided without warranty as
+   described in the Simplified BSD License.
+
+Table of Contents
+
+   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
+   2.  Network Design Requirements . . . . . . . . . . . . . . . . .   4
+     2.1.  Bandwidth and Traffic Patterns  . . . . . . . . . . . . .   4
+     2.2.  CAPEX Minimization  . . . . . . . . . . . . . . . . . . .   4
+     2.3.  OPEX Minimization . . . . . . . . . . . . . . . . . . . .   5
+     2.4.  Traffic Engineering . . . . . . . . . . . . . . . . . . .   5
+     2.5.  Summarized Requirements . . . . . . . . . . . . . . . . .   6
+   3.  Data Center Topologies Overview . . . . . . . . . . . . . . .   6
+     3.1.  Traditional DC Topology . . . . . . . . . . . . . . . . .   6
+     3.2.  Clos Network Topology . . . . . . . . . . . . . . . . . .   7
+       3.2.1.  Overview  . . . . . . . . . . . . . . . . . . . . . .   7
+       3.2.2.  Clos Topology Properties  . . . . . . . . . . . . . .   8
+       3.2.3.  Scaling the Clos Topology . . . . . . . . . . . . . .   9
+       3.2.4.  Managing the Size of Clos Topology Tiers  . . . . . .  10
+   4.  Data Center Routing Overview  . . . . . . . . . . . . . . . .  11
+     4.1.  L2-Only Designs . . . . . . . . . . . . . . . . . . . . .  11
+     4.2.  Hybrid L2/L3 Designs  . . . . . . . . . . . . . . . . . .  12
+     4.3.  L3-Only Designs . . . . . . . . . . . . . . . . . . . . .  12
+   5.  Routing Protocol Design . . . . . . . . . . . . . . . . . . .  13
+     5.1.  Choosing EBGP as the Routing Protocol . . . . . . . . . .  13
+     5.2.  EBGP Configuration for Clos Topology  . . . . . . . . . .  15
+       5.2.1.  EBGP Configuration Guidelines and Example ASN Scheme   15
+       5.2.2.  Private Use ASNs  . . . . . . . . . . . . . . . . . .  16
+       5.2.3.  Prefix Advertisement  . . . . . . . . . . . . . . . .  17
+       5.2.4.  External Connectivity . . . . . . . . . . . . . . . .  18
+       5.2.5.  Route Summarization at the Edge . . . . . . . . . . .  19
+   6.  ECMP Considerations . . . . . . . . . . . . . . . . . . . . .  20
+     6.1.  Basic ECMP  . . . . . . . . . . . . . . . . . . . . . . .  20
+     6.2.  BGP ECMP over Multiple ASNs . . . . . . . . . . . . . . .  21
+     6.3.  Weighted ECMP . . . . . . . . . . . . . . . . . . . . . .  21
+     6.4.  Consistent Hashing  . . . . . . . . . . . . . . . . . . .  22
+
+
+
+Lapukhov, et al.              Informational                     [Page 2]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   7.  Routing Convergence Properties  . . . . . . . . . . . . . . .  22
+     7.1.  Fault Detection Timing  . . . . . . . . . . . . . . . . .  22
+     7.2.  Event Propagation Timing  . . . . . . . . . . . . . . . .  23
+     7.3.  Impact of Clos Topology Fan-Outs  . . . . . . . . . . . .  24
+     7.4.  Failure Impact Scope  . . . . . . . . . . . . . . . . . .  24
+     7.5.  Routing Micro-Loops . . . . . . . . . . . . . . . . . . .  26
+   8.  Additional Options for Design . . . . . . . . . . . . . . . .  26
+     8.1.  Third-Party Route Injection . . . . . . . . . . . . . . .  26
+     8.2.  Route Summarization within Clos Topology  . . . . . . . .  27
+       8.2.1.  Collapsing Tier 1 Devices Layer . . . . . . . . . . .  27
+       8.2.2.  Simple Virtual Aggregation  . . . . . . . . . . . . .  29
+     8.3.  ICMP Unreachable Message Masquerading . . . . . . . . . .  29
+   9.  Security Considerations . . . . . . . . . . . . . . . . . . .  30
+   10. References  . . . . . . . . . . . . . . . . . . . . . . . . .  30
+     10.1.  Normative References . . . . . . . . . . . . . . . . . .  30
+     10.2.  Informative References . . . . . . . . . . . . . . . . .  31
+   Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  35
+   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  35
+
+1.  Introduction
+
+   This document describes a practical routing design that can be used
+   in a large-scale data center (DC) design.  Such data centers, also
+   known as "hyper-scale" or "warehouse-scale" data centers, have a
+   unique attribute of supporting over a hundred thousand servers.  In
+   order to accommodate networks of this scale, operators are revisiting
+   networking designs and platforms to address this need.
+
+   The design presented in this document is based on operational
+   experience with data centers built to support large-scale distributed
+   software infrastructure, such as a web search engine.  The primary
+   requirements in such an environment are operational simplicity and
+   network stability so that a small group of people can effectively
+   support a significantly sized network.
+
+   Experimentation and extensive testing have shown that External BGP
+   (EBGP) [RFC4271] is well suited as a stand-alone routing protocol for
+   these types of data center applications.  This is in contrast with
+   more traditional DC designs, which may use simple tree topologies and
+   rely on extending Layer 2 (L2) domains across multiple network
+   devices.  This document elaborates on the requirements that led to
+   this design choice and presents details of the EBGP routing design as
+   well as exploring ideas for further enhancements.
+
+   This document first presents an overview of network design
+   requirements and considerations for large-scale data centers.  Then,
+   traditional hierarchical data center network topologies are
+   contrasted with Clos networks [CLOS1953] that are horizontally scaled
+
+
+
+Lapukhov, et al.              Informational                     [Page 3]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   out.  This is followed by arguments for selecting EBGP with a Clos
+   topology as the most appropriate routing protocol to meet the
+   requirements and the proposed design is described in detail.
+   Finally, this document reviews some additional considerations and
+   design options.  A thorough understanding of BGP is assumed by a
+   reader planning on deploying the design described within the
+   document.
+
+2.  Network Design Requirements
+
+   This section describes and summarizes network design requirements for
+   large-scale data centers.
+
+2.1.  Bandwidth and Traffic Patterns
+
+   The primary requirement when building an interconnection network for
+   a large number of servers is to accommodate application bandwidth and
+   latency requirements.  Until recently it was quite common to see the
+   majority of traffic entering and leaving the data center, commonly
+   referred to as "north-south" traffic.  Traditional "tree" topologies
+   were sufficient to accommodate such flows, even with high
+   oversubscription ratios between the layers of the network.  If more
+   bandwidth was required, it was added by "scaling up" the network
+   elements, e.g., by upgrading the device's linecards or fabrics or
+   replacing the device with one with higher port density.
+
+   Today many large-scale data centers host applications generating
+   significant amounts of server-to-server traffic, which does not
+   egress the DC, commonly referred to as "east-west" traffic.  Examples
+   of such applications could be computer clusters such as Hadoop
+   [HADOOP], massive data replication between clusters needed by certain
+   applications, or virtual machine migrations.  Scaling traditional
+   tree topologies to match these bandwidth demands becomes either too
+   expensive or impossible due to physical limitations, e.g., port
+   density in a switch.
+
+2.2.  CAPEX Minimization
+
+   The Capital Expenditures (CAPEX) associated with the network
+   infrastructure alone constitutes about 10-15% of total data center
+   expenditure (see [GREENBERG2009]).  However, the absolute cost is
+   significant, and hence there is a need to constantly drive down the
+   cost of individual network elements.  This can be accomplished in two
+   ways:
+
+   o  Unifying all network elements, preferably using the same hardware
+      type or even the same device.  This allows for volume pricing on
+      bulk purchases and reduced maintenance and inventory costs.
+
+
+
+Lapukhov, et al.              Informational                     [Page 4]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   o  Driving costs down using competitive pressures, by introducing
+      multiple network equipment vendors.
+
+   In order to allow for good vendor diversity, it is important to
+   minimize the software feature requirements for the network elements.
+   This strategy provides maximum flexibility of vendor equipment
+   choices while enforcing interoperability using open standards.
+
+2.3.  OPEX Minimization
+
+   Operating large-scale infrastructure can be expensive as a larger
+   amount of elements will statistically fail more often.  Having a
+   simpler design and operating using a limited software feature set
+   minimizes software issue-related failures.
+
+   An important aspect of Operational Expenditure (OPEX) minimization is
+   reducing the size of failure domains in the network.  Ethernet
+   networks are known to be susceptible to broadcast or unicast traffic
+   storms that can have a dramatic impact on network performance and
+   availability.  The use of a fully routed design significantly reduces
+   the size of the data-plane failure domains, i.e., limits them to the
+   lowest level in the network hierarchy.  However, such designs
+   introduce the problem of distributed control-plane failures.  This
+   observation calls for simpler and less control-plane protocols to
+   reduce protocol interaction issues, reducing the chance of a network
+   meltdown.  Minimizing software feature requirements as described in
+   the CAPEX section above also reduces testing and training
+   requirements.
+
+2.4.  Traffic Engineering
+
+   In any data center, application load balancing is a critical function
+   performed by network devices.  Traditionally, load balancers are
+   deployed as dedicated devices in the traffic forwarding path.  The
+   problem arises in scaling load balancers under growing traffic
+   demand.  A preferable solution would be able to scale the load-
+   balancing layer horizontally, by adding more of the uniform nodes and
+   distributing incoming traffic across these nodes.  In situations like
+   this, an ideal choice would be to use network infrastructure itself
+   to distribute traffic across a group of load balancers.  The
+   combination of anycast prefix advertisement [RFC4786] and Equal Cost
+   Multipath (ECMP) functionality can be used to accomplish this goal.
+   To allow for more granular load distribution, it is beneficial for
+   the network to support the ability to perform controlled per-hop
+   traffic engineering.  For example, it is beneficial to directly
+   control the ECMP next-hop set for anycast prefixes at every level of
+   the network hierarchy.
+
+
+
+
+Lapukhov, et al.              Informational                     [Page 5]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+2.5.  Summarized Requirements
+
+   This section summarizes the list of requirements outlined in the
+   previous sections:
+
+   o  REQ1: Select a topology that can be scaled "horizontally" by
+      adding more links and network devices of the same type without
+      requiring upgrades to the network elements themselves.
+
+   o  REQ2: Define a narrow set of software features/protocols supported
+      by a multitude of networking equipment vendors.
+
+   o  REQ3: Choose a routing protocol that has a simple implementation
+      in terms of programming code complexity and ease of operational
+      support.
+
+   o  REQ4: Minimize the failure domain of equipment or protocol issues
+      as much as possible.
+
+   o  REQ5: Allow for some traffic engineering, preferably via explicit
+      control of the routing prefix next hop using built-in protocol
+      mechanics.
+
+3.  Data Center Topologies Overview
+
+   This section provides an overview of two general types of data center
+   designs -- hierarchical (also known as "tree-based") and Clos-based
+   network designs.
+
+3.1.  Traditional DC Topology
+
+   In the networking industry, a common design choice for data centers
+   typically looks like an (upside down) tree with redundant uplinks and
+   three layers of hierarchy namely; core, aggregation/distribution, and
+   access layers (see Figure 1).  To accommodate bandwidth demands, each
+   higher layer, from the server towards DC egress or WAN, has higher
+   port density and bandwidth capacity where the core functions as the
+   "trunk" of the tree-based design.  To keep terminology uniform and
+   for comparison with other designs, in this document these layers will
+   be referred to as Tier 1, Tier 2 and Tier 3 "tiers", instead of core,
+   aggregation, or access layers.
+
+
+
+
+
+
+
+
+
+
+Lapukhov, et al.              Informational                     [Page 6]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+             +------+  +------+
+             |      |  |      |
+             |      |--|      |           Tier 1
+             |      |  |      |
+             +------+  +------+
+               |  |      |  |
+     +---------+  |      |  +----------+
+     | +-------+--+------+--+-------+  |
+     | |       |  |      |  |       |  |
+   +----+     +----+    +----+     +----+
+   |    |     |    |    |    |     |    |
+   |    |-----|    |    |    |-----|    | Tier 2
+   |    |     |    |    |    |     |    |
+   +----+     +----+    +----+     +----+
+      |         |          |         |
+      |         |          |         |
+      | +-----+ |          | +-----+ |
+      +-|     |-+          +-|     |-+    Tier 3
+        +-----+              +-----+
+         | | |                | | |
+     <- Servers ->        <- Servers ->
+
+                   Figure 1: Typical DC Network Topology
+
+   Unfortunately, as noted previously, it is not possible to scale a
+   tree-based design to a large enough degree for handling large-scale
+   designs due to the inability to be able to acquire Tier 1 devices
+   with a large enough port density to sufficiently scale Tier 2.  Also,
+   continuous upgrades or replacement of the upper-tier devices are
+   required as deployment size or bandwidth requirements increase, which
+   is operationally complex.  For this reason, REQ1 is in place,
+   eliminating this type of design from consideration.
+
+3.2.  Clos Network Topology
+
+   This section describes a common design for horizontally scalable
+   topology in large-scale data centers in order to meet REQ1.
+
+3.2.1.  Overview
+
+   A common choice for a horizontally scalable topology is a folded Clos
+   topology, sometimes called "fat-tree" (for example, [INTERCON] and
+   [ALFARES2008]).  This topology features an odd number of stages
+   (sometimes known as "dimensions") and is commonly made of uniform
+   elements, e.g., network switches with the same port count.
+   Therefore, the choice of folded Clos topology satisfies REQ1 and
+
+
+
+
+
+Lapukhov, et al.              Informational                     [Page 7]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   facilitates REQ2.  See Figure 2 below for an example of a folded
+   3-stage Clos topology (3 stages counting Tier 2 stage twice, when
+   tracing a packet flow):
+
+   +-------+
+   |       |----------------------------+
+   |       |------------------+         |
+   |       |--------+         |         |
+   +-------+        |         |         |
+   +-------+        |         |         |
+   |       |--------+---------+-------+ |
+   |       |--------+-------+ |       | |
+   |       |------+ |       | |       | |
+   +-------+      | |       | |       | |
+   +-------+      | |       | |       | |
+   |       |------+-+-------+-+-----+ | |
+   |       |------+-+-----+ | |     | | |
+   |       |----+ | |     | | |     | | |
+   +-------+    | | |     | | |   ---------> M links
+    Tier 1      | | |     | | |     | | |
+              +-------+ +-------+ +-------+
+              |       | |       | |       |
+              |       | |       | |       | Tier 2
+              |       | |       | |       |
+              +-------+ +-------+ +-------+
+                | | |     | | |     | | |
+                | | |     | | |   ---------> N Links
+                | | |     | | |     | | |
+                O O O     O O O     O O O   Servers
+
+                  Figure 2: 3-Stage Folded Clos Topology
+
+   This topology is often also referred to as a "Leaf and Spine"
+   network, where "Spine" is the name given to the middle stage of the
+   Clos topology (Tier 1) and "Leaf" is the name of input/output stage
+   (Tier 2).  For uniformity, this document will refer to these layers
+   using the "Tier n" notation.
+
+3.2.2.  Clos Topology Properties
+
+   The following are some key properties of the Clos topology:
+
+   o  The topology is fully non-blocking, or more accurately non-
+      interfering, if M >= N and oversubscribed by a factor of N/M
+      otherwise.  Here M and N is the uplink and downlink port count
+      respectively, for a Tier 2 switch as shown in Figure 2.
+
+
+
+
+
+Lapukhov, et al.              Informational                     [Page 8]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   o  Utilizing this topology requires control and data-plane support
+      for ECMP with a fan-out of M or more.
+
+   o  Tier 1 switches have exactly one path to every server in this
+      topology.  This is an important property that makes route
+      summarization dangerous in this topology (see Section 8.2 below).
+
+   o  Traffic flowing from server to server is load balanced over all
+      available paths using ECMP.
+
+3.2.3.  Scaling the Clos Topology
+
+   A Clos topology can be scaled either by increasing network element
+   port density or by adding more stages, e.g., moving to a 5-stage
+   Clos, as illustrated in Figure 3 below:
+
+                                      Tier 1
+                                     +-----+
+          Cluster                    |     |
+ +----------------------------+   +--|     |--+
+ |                            |   |  +-----+  |
+ |                    Tier 2  |   |           |   Tier 2
+ |                   +-----+  |   |  +-----+  |  +-----+
+ |     +-------------| DEV |------+--|     |--+--|     |-------------+
+ |     |       +-----|  C  |------+  |     |  +--|     |-----+       |
+ |     |       |     +-----+  |      +-----+     +-----+     |       |
+ |     |       |              |                              |       |
+ |     |       |     +-----+  |      +-----+     +-----+     |       |
+ |     | +-----------| DEV |------+  |     |  +--|     |-----------+ |
+ |     | |     | +---|  D  |------+--|     |--+--|     |---+ |     | |
+ |     | |     | |   +-----+  |   |  +-----+  |  +-----+   | |     | |
+ |     | |     | |            |   |           |            | |     | |
+ |   +-----+ +-----+          |   |  +-----+  |          +-----+ +-----+
+ |   | DEV | | DEV |          |   +--|     |--+          |     | |     |
+ |   |  A  | |  B  | Tier 3   |      |     |      Tier 3 |     | |     |
+ |   +-----+ +-----+          |      +-----+             +-----+ +-----+
+ |     | |     | |            |                            | |     | |
+ |     O O     O O            |                            O O     O O
+ |       Servers              |                              Servers
+ +----------------------------+
+
+                      Figure 3: 5-Stage Clos Topology
+
+   The small example of topology in Figure 3 is built from devices with
+   a port count of 4.  In this document, one set of directly connected
+   Tier 2 and Tier 3 devices along with their attached servers will be
+   referred to as a "cluster".  For example, DEV A, B, C, D, and the
+   servers that connect to DEV A and B, on Figure 3 form a cluster.  The
+
+
+
+Lapukhov, et al.              Informational                     [Page 9]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   concept of a cluster may also be a useful concept as a single
+   deployment or maintenance unit that can be operated on at a different
+   frequency than the entire topology.
+
+   In practice, Tier 3 of the network, which is typically Top-of-Rack
+   switches (ToRs), is where oversubscription is introduced to allow for
+   packaging of more servers in the data center while meeting the
+   bandwidth requirements for different types of applications.  The main
+   reason to limit oversubscription at a single layer of the network is
+   to simplify application development that would otherwise need to
+   account for multiple bandwidth pools: within rack (Tier 3), between
+   racks (Tier 2), and between clusters (Tier 1).  Since
+   oversubscription does not have a direct relationship to the routing
+   design, it is not discussed further in this document.
+
+3.2.4.  Managing the Size of Clos Topology Tiers
+
+   If a data center network size is small, it is possible to reduce the
+   number of switches in Tier 1 or Tier 2 of a Clos topology by a factor
+   of two.  To understand how this could be done, take Tier 1 as an
+   example.  Every Tier 2 device connects to a single group of Tier 1
+   devices.  If half of the ports on each of the Tier 1 devices are not
+   being used, then it is possible to reduce the number of Tier 1
+   devices by half and simply map two uplinks from a Tier 2 device to
+   the same Tier 1 device that were previously mapped to different Tier
+   1 devices.  This technique maintains the same bandwidth while
+   reducing the number of elements in Tier 1, thus saving on CAPEX.  The
+   tradeoff, in this example, is the reduction of maximum DC size in
+   terms of overall server count by half.
+
+   In this example, Tier 2 devices will be using two parallel links to
+   connect to each Tier 1 device.  If one of these links fails, the
+   other will pick up all traffic of the failed link, possibly resulting
+   in heavy congestion and quality of service degradation if the path
+   determination procedure does not take bandwidth amount into account,
+   since the number of upstream Tier 1 devices is likely wider than two.
+   To avoid this situation, parallel links can be grouped in link
+   aggregation groups (LAGs), e.g., [IEEE8023AD], with widely available
+   implementation settings that take the whole "bundle" down upon a
+   single link failure.  Equivalent techniques that enforce "fate
+   sharing" on the parallel links can be used in place of LAGs to
+   achieve the same effect.  As a result of such fate-sharing, traffic
+   from two or more failed links will be rebalanced over the multitude
+   of remaining paths that equals the number of Tier 1 devices.  This
+   example is using two links for simplicity, having more links in a
+   bundle will have less impact on capacity upon a member-link failure.
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 10]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+4.  Data Center Routing Overview
+
+   This section provides an overview of three general types of data
+   center protocol designs -- Layer 2 only, Hybrid Layer L2/L3, and
+   Layer 3 only.
+
+4.1.  L2-Only Designs
+
+   Originally, most data center designs used Spanning Tree Protocol
+   (STP) originally defined in [IEEE8021D-1990] for loop-free topology
+   creation, typically utilizing variants of the traditional DC topology
+   described in Section 3.1.  At the time, many DC switches either did
+   not support Layer 3 routing protocols or supported them with
+   additional licensing fees, which played a part in the design choice.
+   Although many enhancements have been made through the introduction of
+   Rapid Spanning Tree Protocol (RSTP) in the latest revision of
+   [IEEE8021D-2004] and Multiple Spanning Tree Protocol (MST) specified
+   in [IEEE8021Q] that increase convergence, stability, and load-
+   balancing in larger topologies, many of the fundamentals of the
+   protocol limit its applicability in large-scale DCs.  STP and its
+   newer variants use an active/standby approach to path selection, and
+   are therefore hard to deploy in horizontally scaled topologies as
+   described in Section 3.2.  Further, operators have had many
+   experiences with large failures due to issues caused by improper
+   cabling, misconfiguration, or flawed software on a single device.
+   These failures regularly affected the entire spanning-tree domain and
+   were very hard to troubleshoot due to the nature of the protocol.
+   For these reasons, and since almost all DC traffic is now IP,
+   therefore requiring a Layer 3 routing protocol at the network edge
+   for external connectivity, designs utilizing STP usually fail all of
+   the requirements of large-scale DC operators.  Various enhancements
+   to link-aggregation protocols such as [IEEE8023AD], generally known
+   as Multi-Chassis Link-Aggregation (M-LAG) made it possible to use
+   Layer 2 designs with active-active network paths while relying on STP
+   as the backup for loop prevention.  The major downsides of this
+   approach are the lack of ability to scale linearly past two in most
+   implementations, lack of standards-based implementations, and the
+   added failure domain risk of syncing state between the devices.
+
+   It should be noted that building large, horizontally scalable,
+   L2-only networks without STP is possible recently through the
+   introduction of the Transparent Interconnection of Lots of Links
+   (TRILL) protocol in [RFC6325].  TRILL resolves many of the issues STP
+   has for large-scale DC design however, due to the limited number of
+   implementations, and often the requirement for specific equipment
+   that supports it, this has limited its applicability and increased
+   the cost of such designs.
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 11]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   Finally, neither the base TRILL specification nor the M-LAG approach
+   totally eliminate the problem of the shared broadcast domain that is
+   so detrimental to the operations of any Layer 2, Ethernet-based
+   solution.  Later TRILL extensions have been proposed to solve the
+   this problem statement, primarily based on the approaches outlined in
+   [RFC7067], but this even further limits the number of available
+   interoperable implementations that can be used to build a fabric.
+   Therefore, TRILL-based designs have issues meeting REQ2, REQ3, and
+   REQ4.
+
+4.2.  Hybrid L2/L3 Designs
+
+   Operators have sought to limit the impact of data-plane faults and
+   build large-scale topologies through implementing routing protocols
+   in either the Tier 1 or Tier 2 parts of the network and dividing the
+   Layer 2 domain into numerous, smaller domains.  This design has
+   allowed data centers to scale up, but at the cost of complexity in
+   managing multiple network protocols.  For the following reasons,
+   operators have retained Layer 2 in either the access (Tier 3) or both
+   access and aggregation (Tier 3 and Tier 2) parts of the network:
+
+   o  Supporting legacy applications that may require direct Layer 2
+      adjacency or use non-IP protocols.
+
+   o  Seamless mobility for virtual machines that require the
+      preservation of IP addresses when a virtual machine moves to a
+      different Tier 3 switch.
+
+   o  Simplified IP addressing = less IP subnets are required for the
+      data center.
+
+   o  Application load balancing may require direct Layer 2 reachability
+      to perform certain functions such as Layer 2 Direct Server Return
+      (DSR).  See [L3DSR].
+
+   o  Continued CAPEX differences between L2- and L3-capable switches.
+
+4.3.  L3-Only Designs
+
+   Network designs that leverage IP routing down to Tier 3 of the
+   network have gained popularity as well.  The main benefit of these
+   designs is improved network stability and scalability, as a result of
+   confining L2 broadcast domains.  Commonly, an Interior Gateway
+   Protocol (IGP) such as Open Shortest Path First (OSPF) [RFC2328] is
+   used as the primary routing protocol in such a design.  As data
+   centers grow in scale, and server count exceeds tens of thousands,
+   such fully routed designs have become more attractive.
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 12]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   Choosing a L3-only design greatly simplifies the network,
+   facilitating the meeting of REQ1 and REQ2, and has widespread
+   adoption in networks where large Layer 2 adjacency and larger size
+   Layer 3 subnets are not as critical compared to network scalability
+   and stability.  Application providers and network operators continue
+   to develop new solutions to meet some of the requirements that
+   previously had driven large Layer 2 domains by using various overlay
+   or tunneling techniques.
+
+5.  Routing Protocol Design
+
+   In this section, the motivations for using External BGP (EBGP) as the
+   single routing protocol for data center networks having a Layer 3
+   protocol design and Clos topology are reviewed.  Then, a practical
+   approach for designing an EBGP-based network is provided.
+
+5.1.  Choosing EBGP as the Routing Protocol
+
+   REQ2 would give preference to the selection of a single routing
+   protocol to reduce complexity and interdependencies.  While it is
+   common to rely on an IGP in this situation, sometimes with either the
+   addition of EBGP at the device bordering the WAN or Internal BGP
+   (IBGP) throughout, this document proposes the use of an EBGP-only
+   design.
+
+   Although EBGP is the protocol used for almost all Inter-Domain
+   Routing in the Internet and has wide support from both vendor and
+   service provider communities, it is not generally deployed as the
+   primary routing protocol within the data center for a number of
+   reasons (some of which are interrelated):
+
+   o  BGP is perceived as a "WAN-only, protocol-only" and not often
+      considered for enterprise or data center applications.
+
+   o  BGP is believed to have a "much slower" routing convergence
+      compared to IGPs.
+
+   o  Large-scale BGP deployments typically utilize an IGP for BGP next-
+      hop resolution as all nodes in the IBGP topology are not directly
+      connected.
+
+   o  BGP is perceived to require significant configuration overhead and
+      does not support neighbor auto-discovery.
+
+
+
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 13]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   This document discusses some of these perceptions, especially as
+   applicable to the proposed design, and highlights some of the
+   advantages of using the protocol such as:
+
+   o  BGP has less complexity in parts of its protocol design --
+      internal data structures and state machine are simpler as compared
+      to most link-state IGPs such as OSPF.  For example, instead of
+      implementing adjacency formation, adjacency maintenance and/or
+      flow-control, BGP simply relies on TCP as the underlying
+      transport.  This fulfills REQ2 and REQ3.
+
+   o  BGP information flooding overhead is less when compared to link-
+      state IGPs.  Since every BGP router calculates and propagates only
+      the best-path selected, a network failure is masked as soon as the
+      BGP speaker finds an alternate path, which exists when highly
+      symmetric topologies, such as Clos, are coupled with an EBGP-only
+      design.  In contrast, the event propagation scope of a link-state
+      IGP is an entire area, regardless of the failure type.  In this
+      way, BGP better meets REQ3 and REQ4.  It is also worth mentioning
+      that all widely deployed link-state IGPs feature periodic
+      refreshes of routing information while BGP does not expire routing
+      state, although this rarely impacts modern router control planes.
+
+   o  BGP supports third-party (recursively resolved) next hops.  This
+      allows for manipulating multipath to be non-ECMP-based or
+      forwarding-based on application-defined paths, through
+      establishment of a peering session with an application
+      "controller" that can inject routing information into the system,
+      satisfying REQ5.  OSPF provides similar functionality using
+      concepts such as "Forwarding Address", but with more difficulty in
+      implementation and far less control of information propagation
+      scope.
+
+   o  Using a well-defined Autonomous System Number (ASN) allocation
+      scheme and standard AS_PATH loop detection, "BGP path hunting"
+      (see [JAKMA2008]) can be controlled and complex unwanted paths
+      will be ignored.  See Section 5.2 for an example of a working ASN
+      allocation scheme.  In a link-state IGP, accomplishing the same
+      goal would require multi-(instance/topology/process) support,
+      typically not available in all DC devices and quite complex to
+      configure and troubleshoot.  Using a traditional single flooding
+      domain, which most DC designs utilize, under certain failure
+      conditions may pick up unwanted lengthy paths, e.g., traversing
+      multiple Tier 2 devices.
+
+
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 14]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   o  EBGP configuration that is implemented with minimal routing policy
+      is easier to troubleshoot for network reachability issues.  In
+      most implementations, it is straightforward to view contents of
+      the BGP Loc-RIB and compare it to the router's Routing Information
+      Base (RIB).  Also, in most implementations, an operator can view
+      every BGP neighbors Adj-RIB-In and Adj-RIB-Out structures, and
+      therefore incoming and outgoing Network Layer Reachability
+      Information (NLRI) information can be easily correlated on both
+      sides of a BGP session.  Thus, BGP satisfies REQ3.
+
+5.2.  EBGP Configuration for Clos Topology
+
+   Clos topologies that have more than 5 stages are very uncommon due to
+   the large numbers of interconnects required by such a design.
+   Therefore, the examples below are made with reference to the 5-stage
+   Clos topology (in unfolded state).
+
+5.2.1.  EBGP Configuration Guidelines and Example ASN Scheme
+
+   The diagram below illustrates an example of an ASN allocation scheme.
+   The following is a list of guidelines that can be used:
+
+   o  EBGP single-hop sessions are established over direct point-to-
+      point links interconnecting the network nodes, no multi-hop or
+      loopback sessions are used, even in the case of multiple links
+      between the same pair of nodes.
+
+   o  Private Use ASNs from the range 64512-65534 are used to avoid ASN
+      conflicts.
+
+   o  A single ASN is allocated to all of the Clos topology's Tier 1
+      devices.
+
+   o  A unique ASN is allocated to each set of Tier 2 devices in the
+      same cluster.
+
+   o  A unique ASN is allocated to every Tier 3 device (e.g., ToR) in
+      this topology.
+
+
+
+
+
+
+
+
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 15]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+                                ASN 65534
+                               +---------+
+                               | +-----+ |
+                               | |     | |
+                             +-|-|     |-|-+
+                             | | +-----+ | |
+                  ASN 646XX  | |         | |  ASN 646XX
+                 +---------+ | |         | | +---------+
+                 | +-----+ | | | +-----+ | | | +-----+ |
+     +-----------|-|     |-|-+-|-|     |-|-+-|-|     |-|-----------+
+     |       +---|-|     |-|-+ | |     | | +-|-|     |-|---+       |
+     |       |   | +-----+ |   | +-----+ |   | +-----+ |   |       |
+     |       |   |         |   |         |   |         |   |       |
+     |       |   |         |   |         |   |         |   |       |
+     |       |   | +-----+ |   | +-----+ |   | +-----+ |   |       |
+     | +-----+---|-|     |-|-+ | |     | | +-|-|     |-|---+-----+ |
+     | |     | +-|-|     |-|-+-|-|     |-|-+-|-|     |-|-+ |     | |
+     | |     | | | +-----+ | | | +-----+ | | | +-----+ | | |     | |
+     | |     | | +---------+ | |         | | +---------+ | |     | |
+     | |     | |             | |         | |             | |     | |
+   +-----+ +-----+           | | +-----+ | |           +-----+ +-----+
+   | ASN | |     |           +-|-|     |-|-+           |     | |     |
+   |65YYY| | ... |             | |     | |             | ... | | ... |
+   +-----+ +-----+             | +-----+ |             +-----+ +-----+
+     | |     | |               +---------+               | |     | |
+     O O     O O              <- Servers ->              O O     O O
+
+                 Figure 4: BGP ASN Layout for 5-Stage Clos
+
+5.2.2.  Private Use ASNs
+
+   The original range of Private Use ASNs [RFC6996] limited operators to
+   1023 unique ASNs.  Since it is quite likely that the number of
+   network devices may exceed this number, a workaround is required.
+   One approach is to re-use the ASNs assigned to the Tier 3 devices
+   across different clusters.  For example, Private Use ASNs 65001,
+   65002 ... 65032 could be used within every individual cluster and
+   assigned to Tier 3 devices.
+
+   To avoid route suppression due to the AS_PATH loop detection
+   mechanism in BGP, upstream EBGP sessions on Tier 3 devices must be
+   configured with the "Allowas-in" feature [ALLOWASIN] that allows
+   accepting a device's own ASN in received route advertisements.
+   Although this feature is not standardized, it is widely available
+   across multiple vendors implementations.  Introducing this feature
+   does not make routing loops more likely in the design since the
+   AS_PATH is being added to by routers at each of the topology tiers
+   and AS_PATH length is an early tie breaker in the BGP path selection
+
+
+
+Lapukhov, et al.              Informational                    [Page 16]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   process.  Further loop protection is still in place at the Tier 1
+   device, which will not accept routes with a path including its own
+   ASN.  Tier 2 devices do not have direct connectivity with each other.
+
+   Another solution to this problem would be to use Four-Octet ASNs
+   ([RFC6793]), where there are additional Private Use ASNs available,
+   see [IANA.AS].  Use of Four-Octet ASNs puts additional protocol
+   complexity in the BGP implementation and should be balanced against
+   the complexity of re-use when considering REQ3 and REQ4.  Perhaps
+   more importantly, they are not yet supported by all BGP
+   implementations, which may limit vendor selection of DC equipment.
+   When supported, ensure that deployed implementations are able to
+   remove the Private Use ASNs when external connectivity
+   (Section 5.2.4) to these ASNs is required.
+
+5.2.3.  Prefix Advertisement
+
+   A Clos topology features a large number of point-to-point links and
+   associated prefixes.  Advertising all of these routes into BGP may
+   create Forwarding Information Base (FIB) overload in the network
+   devices.  Advertising these links also puts additional path
+   computation stress on the BGP control plane for little benefit.
+   There are two possible solutions:
+
+   o  Do not advertise any of the point-to-point links into BGP.  Since
+      the EBGP-based design changes the next-hop address at every
+      device, distant networks will automatically be reachable via the
+      advertising EBGP peer and do not require reachability to these
+      prefixes.  However, this may complicate operations or monitoring:
+      e.g., using the popular "traceroute" tool will display IP
+      addresses that are not reachable.
+
+   o  Advertise point-to-point links, but summarize them on every
+      device.  This requires an address allocation scheme such as
+      allocating a consecutive block of IP addresses per Tier 1 and Tier
+      2 device to be used for point-to-point interface addressing to the
+      lower layers (Tier 2 uplinks will be allocated from Tier 1 address
+      blocks and so forth).
+
+   Server subnets on Tier 3 devices must be announced into BGP without
+   using route summarization on Tier 2 and Tier 1 devices.  Summarizing
+   subnets in a Clos topology results in route black-holing under a
+   single link failure (e.g., between Tier 2 and Tier 3 devices), and
+   hence must be avoided.  The use of peer links within the same tier to
+   resolve the black-holing problem by providing "bypass paths" is
+   undesirable due to O(N^2) complexity of the peering-mesh and waste of
+   ports on the devices.  An alternative to the full mesh of peer links
+   would be to use a simpler bypass topology, e.g., a "ring" as
+
+
+
+Lapukhov, et al.              Informational                    [Page 17]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   described in [FB4POST], but such a topology adds extra hops and has
+   limited bandwidth.  It may require special tweaks to make BGP routing
+   work, e.g., splitting every device into an ASN of its own.  Later in
+   this document, Section 8.2 introduces a less intrusive method for
+   performing a limited form of route summarization in Clos networks and
+   discusses its associated tradeoffs.
+
+5.2.4.  External Connectivity
+
+   A dedicated cluster (or clusters) in the Clos topology could be used
+   for the purpose of connecting to the Wide Area Network (WAN) edge
+   devices, or WAN Routers.  Tier 3 devices in such a cluster would be
+   replaced with WAN routers, and EBGP peering would be used again,
+   though WAN routers are likely to belong to a public ASN if Internet
+   connectivity is required in the design.  The Tier 2 devices in such a
+   dedicated cluster will be referred to as "Border Routers" in this
+   document.  These devices have to perform a few special functions:
+
+   o  Hide network topology information when advertising paths to WAN
+      routers, i.e., remove Private Use ASNs [RFC6996] from the AS_PATH
+      attribute.  This is typically done to avoid ASN number collisions
+      between different data centers and also to provide a uniform
+      AS_PATH length to the WAN for purposes of WAN ECMP to anycast
+      prefixes originated in the topology.  An implementation-specific
+      BGP feature typically called "Remove Private AS" is commonly used
+      to accomplish this.  Depending on implementation, the feature
+      should strip a contiguous sequence of Private Use ASNs found in an
+      AS_PATH attribute prior to advertising the path to a neighbor.
+      This assumes that all ASNs used for intra data center numbering
+      are from the Private Use ranges.  The process for stripping the
+      Private Use ASNs is not currently standardized, see [REMOVAL].
+      However, most implementations at least follow the logic described
+      in this vendor's document [VENDOR-REMOVE-PRIVATE-AS], which is
+      enough for the design specified.
+
+   o  Originate a default route to the data center devices.  This is the
+      only place where a default route can be originated, as route
+      summarization is risky for the unmodified Clos topology.
+      Alternatively, Border Routers may simply relay the default route
+      learned from WAN routers.  Advertising the default route from
+      Border Routers requires that all Border Routers be fully connected
+      to the WAN Routers upstream, to provide resistance to a single-
+      link failure causing the black-holing of traffic.  To prevent
+      black-holing in the situation when all of the EBGP sessions to the
+      WAN routers fail simultaneously on a given device, it is more
+      desirable to readvertise the default route rather than originating
+      the default route via complicated conditional route origination
+      schemes provided by some implementations [CONDITIONALROUTE].
+
+
+
+Lapukhov, et al.              Informational                    [Page 18]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+5.2.5.  Route Summarization at the Edge
+
+   It is often desirable to summarize network reachability information
+   prior to advertising it to the WAN network due to the high amount of
+   IP prefixes originated from within the data center in a fully routed
+   network design.  For example, a network with 2000 Tier 3 devices will
+   have at least 2000 servers subnets advertised into BGP, along with
+   the infrastructure prefixes.  However, as discussed in Section 5.2.3,
+   the proposed network design does not allow for route summarization
+   due to the lack of peer links inside every tier.
+
+   However, it is possible to lift this restriction for the Border
+   Routers by devising a different connectivity model for these devices.
+   There are two options possible:
+
+   o  Interconnect the Border Routers using a full-mesh of physical
+      links or using any other "peer-mesh" topology, such as ring or
+      hub-and-spoke.  Configure BGP accordingly on all Border Leafs to
+      exchange network reachability information, e.g., by adding a mesh
+      of IBGP sessions.  The interconnecting peer links need to be
+      appropriately sized for traffic that will be present in the case
+      of a device or link failure in the mesh connecting the Border
+      Routers.
+
+   o  Tier 1 devices may have additional physical links provisioned
+      toward the Border Routers (which are Tier 2 devices from the
+      perspective of Tier 1).  Specifically, if protection from a single
+      link or node failure is desired, each Tier 1 device would have to
+      connect to at least two Border Routers.  This puts additional
+      requirements on the port count for Tier 1 devices and Border
+      Routers, potentially making it a nonuniform, larger port count,
+      device compared with the other devices in the Clos.  This also
+      reduces the number of ports available to "regular" Tier 2
+      switches, and hence the number of clusters that could be
+      interconnected via Tier 1.
+
+   If any of the above options are implemented, it is possible to
+   perform route summarization at the Border Routers toward the WAN
+   network core without risking a routing black-hole condition under a
+   single link failure.  Both of the options would result in nonuniform
+   topology as additional links have to be provisioned on some network
+   devices.
+
+
+
+
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 19]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+6.  ECMP Considerations
+
+   This section covers the Equal Cost Multipath (ECMP) functionality for
+   Clos topology and discusses a few special requirements.
+
+6.1.  Basic ECMP
+
+   ECMP is the fundamental load-sharing mechanism used by a Clos
+   topology.  Effectively, every lower-tier device will use all of its
+   directly attached upper-tier devices to load-share traffic destined
+   to the same IP prefix.  The number of ECMP paths between any two Tier
+   3 devices in Clos topology is equal to the number of the devices in
+   the middle stage (Tier 1).  For example, Figure 5 illustrates a
+   topology where Tier 3 device A has four paths to reach servers X and
+   Y, via Tier 2 devices B and C and then Tier 1 devices 1, 2, 3, and 4,
+   respectively.
+
+                                Tier 1
+                               +-----+
+                               | DEV |
+                            +->|  1  |--+
+                            |  +-----+  |
+                    Tier 2  |           |   Tier 2
+                   +-----+  |  +-----+  |  +-----+
+     +------------>| DEV |--+->| DEV |--+--|     |-------------+
+     |       +-----|  B  |--+  |  2  |  +--|     |-----+       |
+     |       |     +-----+     +-----+     +-----+     |       |
+     |       |                                         |       |
+     |       |     +-----+     +-----+     +-----+     |       |
+     | +-----+---->| DEV |--+  | DEV |  +--|     |-----+-----+ |
+     | |     | +---|  C  |--+->|  3  |--+--|     |---+ |     | |
+     | |     | |   +-----+  |  +-----+  |  +-----+   | |     | |
+     | |     | |            |           |            | |     | |
+   +-----+ +-----+          |  +-----+  |          +-----+ +-----+
+   | DEV | |     | Tier 3   +->| DEV |--+   Tier 3 |     | |     |
+   |  A  | |     |             |  4  |             |     | |     |
+   +-----+ +-----+             +-----+             +-----+ +-----+
+     | |     | |                                     | |     | |
+     O O     O O            <- Servers ->            X Y     O O
+
+               Figure 5: ECMP Fan-Out Tree from A to X and Y
+
+   The ECMP requirement implies that the BGP implementation must support
+   multipath fan-out for up to the maximum number of devices directly
+   attached at any point in the topology in the upstream or downstream
+   direction.  Normally, this number does not exceed half of the ports
+   found on a device in the topology.  For example, an ECMP fan-out of
+   32 would be required when building a Clos network using 64-port
+
+
+
+Lapukhov, et al.              Informational                    [Page 20]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   devices.  The Border Routers may need to have wider fan-out to be
+   able to connect to a multitude of Tier 1 devices if route
+   summarization at Border Router level is implemented as described in
+   Section 5.2.5.  If a device's hardware does not support wider ECMP,
+   logical link-grouping (link-aggregation at Layer 2) could be used to
+   provide "hierarchical" ECMP (Layer 3 ECMP coupled with Layer 2 ECMP)
+   to compensate for fan-out limitations.  However, this approach
+   increases the risk of flow polarization, as less entropy will be
+   available at the second stage of ECMP.
+
+   Most BGP implementations declare paths to be equal from an ECMP
+   perspective if they match up to and including step (e) in
+   Section 9.1.2.2 of [RFC4271].  In the proposed network design there
+   is no underlying IGP, so all IGP costs are assumed to be zero or
+   otherwise the same value across all paths and policies may be applied
+   as necessary to equalize BGP attributes that vary in vendor defaults,
+   such as the MULTI_EXIT_DISC (MED) attribute and origin code.  For
+   historical reasons, it is also useful to not use 0 as the equalized
+   MED value; this and some other useful BGP information is available in
+   [RFC4277].  Routing loops are unlikely due to the BGP best-path
+   selection process (which prefers shorter AS_PATH length), and longer
+   paths through the Tier 1 devices (which don't allow their own ASN in
+   the path) are not possible.
+
+6.2.  BGP ECMP over Multiple ASNs
+
+   For application load-balancing purposes, it is desirable to have the
+   same prefix advertised from multiple Tier 3 devices.  From the
+   perspective of other devices, such a prefix would have BGP paths with
+   different AS_PATH attribute values, while having the same AS_PATH
+   attribute lengths.  Therefore, BGP implementations must support load-
+   sharing over the above-mentioned paths.  This feature is sometimes
+   known as "multipath relax" or "multipath multiple-AS" and effectively
+   allows for ECMP to be done across different neighboring ASNs if all
+   other attributes are equal as already described in the previous
+   section.
+
+6.3.  Weighted ECMP
+
+   It may be desirable for the network devices to implement "weighted"
+   ECMP, to be able to send more traffic over some paths in ECMP fan-
+   out.  This could be helpful to compensate for failures in the network
+   and send more traffic over paths that have more capacity.  The
+   prefixes that require weighted ECMP would have to be injected using
+   remote BGP speaker (central agent) over a multi-hop session as
+   described further in Section 8.1.  If support in implementations is
+   available, weight distribution for multiple BGP paths could be
+   signaled using the technique described in [LINK].
+
+
+
+Lapukhov, et al.              Informational                    [Page 21]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+6.4.  Consistent Hashing
+
+   It is often desirable to have the hashing function used for ECMP to
+   be consistent (see [CONS-HASH]), to minimize the impact on flow to
+   next-hop affinity changes when a next hop is added or removed to an
+   ECMP group.  This could be used if the network device is used as a
+   load balancer, mapping flows toward multiple destinations -- in this
+   case, losing or adding a destination will not have a detrimental
+   effect on currently established flows.  One particular recommendation
+   on implementing consistent hashing is provided in [RFC2992], though
+   other implementations are possible.  This functionality could be
+   naturally combined with weighted ECMP, with the impact of the next
+   hop changes being proportional to the weight of the given next hop.
+   The downside of consistent hashing is increased load on hardware
+   resource utilization, as typically more resources (e.g., Ternary
+   Content-Addressable Memory (TCAM) space) are required to implement a
+   consistent-hashing function.
+
+7.  Routing Convergence Properties
+
+   This section reviews routing convergence properties in the proposed
+   design.  A case is made that sub-second convergence is achievable if
+   the implementation supports fast EBGP peering session deactivation
+   and timely RIB and FIB updates upon failure of the associated link.
+
+7.1.  Fault Detection Timing
+
+   BGP typically relies on an IGP to route around link/node failures
+   inside an AS, and implements either a polling-based or an event-
+   driven mechanism to obtain updates on IGP state changes.  The
+   proposed routing design does not use an IGP, so the remaining
+   mechanisms that could be used for fault detection are BGP keep-alive
+   time-out (or any other type of keep-alive mechanism) and link-failure
+   triggers.
+
+   Relying solely on BGP keep-alive packets may result in high
+   convergence delays, on the order of multiple seconds (on many BGP
+   implementations the minimum configurable BGP hold timer value is
+   three seconds).  However, many BGP implementations can shut down
+   local EBGP peering sessions in response to the "link down" event for
+   the outgoing interface used for BGP peering.  This feature is
+   sometimes called "fast fallover".  Since links in modern data centers
+   are predominantly point-to-point fiber connections, a physical
+   interface failure is often detected in milliseconds and subsequently
+   triggers a BGP reconvergence.
+
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 22]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   Ethernet links may support failure signaling or detection standards
+   such as Connectivity Fault Management (CFM) as described in
+   [IEEE8021Q]; this may make failure detection more robust.
+   Alternatively, some platforms may support Bidirectional Forwarding
+   Detection (BFD) [RFC5880] to allow for sub-second failure detection
+   and fault signaling to the BGP process.  However, the use of either
+   of these presents additional requirements to vendor software and
+   possibly hardware, and may contradict REQ1.  Until recently with
+   [RFC7130], BFD also did not allow detection of a single member link
+   failure on a LAG, which would have limited its usefulness in some
+   designs.
+
+7.2.  Event Propagation Timing
+
+   In the proposed design, the impact of the BGP
+   MinRouteAdvertisementIntervalTimer (MRAI timer), as specified in
+   Section 9.2.1.1 of [RFC4271], should be considered.  Per the
+   standard, it is required for BGP implementations to space out
+   consecutive BGP UPDATE messages by at least MRAI seconds, which is
+   often a configurable value.  The initial BGP UPDATE messages after an
+   event carrying withdrawn routes are commonly not affected by this
+   timer.  The MRAI timer may present significant convergence delays
+   when a BGP speaker "waits" for the new path to be learned from its
+   peers and has no local backup path information.
+
+   In a Clos topology, each EBGP speaker typically has either one path
+   (Tier 2 devices don't accept paths from other Tier 2 in the same
+   cluster due to same ASN) or N paths for the same prefix, where N is a
+   significantly large number, e.g., N=32 (the ECMP fan-out to the next
+   tier).  Therefore, if a link fails to another device from which a
+   path is received there is either no backup path at all (e.g., from
+   the perspective of a Tier 2 switch losing the link to a Tier 3
+   device), or the backup is readily available in BGP Loc-RIB (e.g.,
+   from the perspective of a Tier 2 device losing the link to a Tier 1
+   switch).  In the former case, the BGP withdrawal announcement will
+   propagate without delay and trigger reconvergence on affected
+   devices.  In the latter case, the best path will be re-evaluated, and
+   the local ECMP group corresponding to the new next-hop set will be
+   changed.  If the BGP path was the best path selected previously, an
+   "implicit withdraw" will be sent via a BGP UPDATE message as
+   described as Option b in Section 3.1 of [RFC4271] due to the BGP
+   AS_PATH attribute changing.
+
+
+
+
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 23]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+7.3.  Impact of Clos Topology Fan-Outs
+
+   Clos topology has large fan-outs, which may impact the "Up->Down"
+   convergence in some cases, as described in this section.  In a
+   situation when a link between Tier 3 and Tier 2 device fails, the
+   Tier 2 device will send BGP UPDATE messages to all upstream Tier 1
+   devices, withdrawing the affected prefixes.  The Tier 1 devices, in
+   turn, will relay these messages to all downstream Tier 2 devices
+   (except for the originator).  Tier 2 devices other than the one
+   originating the UPDATE should then wait for ALL upstream Tier 1
+   devices to send an UPDATE message before removing the affected
+   prefixes and sending corresponding UPDATE downstream to connected
+   Tier 3 devices.  If the original Tier 2 device or the relaying Tier 1
+   devices introduce some delay into their UPDATE message announcements,
+   the result could be UPDATE message "dispersion", that could be as
+   long as multiple seconds.  In order to avoid such a behavior, BGP
+   implementations must support "update groups".  The "update group" is
+   defined as a collection of neighbors sharing the same outbound policy
+   -- the local speaker will send BGP updates to the members of the
+   group synchronously.
+
+   The impact of such "dispersion" grows with the size of topology fan-
+   out and could also grow under network convergence churn.  Some
+   operators may be tempted to introduce "route flap dampening" type
+   features that vendors include to reduce the control-plane impact of
+   rapidly flapping prefixes.  However, due to issues described with
+   false positives in these implementations especially under such
+   "dispersion" events, it is not recommended to enable this feature in
+   this design.  More background and issues with "route flap dampening"
+   and possible implementation changes that could affect this are well
+   described in [RFC7196].
+
+7.4.  Failure Impact Scope
+
+   A network is declared to converge in response to a failure once all
+   devices within the failure impact scope are notified of the event and
+   have recalculated their RIBs and consequently updated their FIBs.
+   Larger failure impact scope typically means slower convergence since
+   more devices have to be notified, and results in a less stable
+   network.  In this section, we describe BGP's advantages over link-
+   state routing protocols in reducing failure impact scope for a Clos
+   topology.
+
+   BGP behaves like a distance-vector protocol in the sense that only
+   the best path from the point of view of the local router is sent to
+   neighbors.  As such, some failures are masked if the local node can
+   immediately find a backup path and does not have to send any updates
+   further.  Notice that in the worst case, all devices in a data center
+
+
+
+Lapukhov, et al.              Informational                    [Page 24]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   topology have to either withdraw a prefix completely or update the
+   ECMP groups in their FIBs.  However, many failures will not result in
+   such a wide impact.  There are two main failure types where impact
+   scope is reduced:
+
+   o  Failure of a link between Tier 2 and Tier 1 devices: In this case,
+      a Tier 2 device will update the affected ECMP groups, removing the
+      failed link.  There is no need to send new information to
+      downstream Tier 3 devices, unless the path was selected as best by
+      the BGP process, in which case only an "implicit withdraw" needs
+      to be sent and this should not affect forwarding.  The affected
+      Tier 1 device will lose the only path available to reach a
+      particular cluster and will have to withdraw the associated
+      prefixes.  Such a prefix withdrawal process will only affect Tier
+      2 devices directly connected to the affected Tier 1 device.  The
+      Tier 2 devices receiving the BGP UPDATE messages withdrawing
+      prefixes will simply have to update their ECMP groups.  The Tier 3
+      devices are not involved in the reconvergence process.
+
+   o  Failure of a Tier 1 device: In this case, all Tier 2 devices
+      directly attached to the failed node will have to update their
+      ECMP groups for all IP prefixes from a non-local cluster.  The
+      Tier 3 devices are once again not involved in the reconvergence
+      process, but may receive "implicit withdraws" as described above.
+
+   Even in the case of such failures where multiple IP prefixes will
+   have to be reprogrammed in the FIB, it is worth noting that all of
+   these prefixes share a single ECMP group on a Tier 2 device.
+   Therefore, in the case of implementations with a hierarchical FIB,
+   only a single change has to be made to the FIB.  "Hierarchical FIB"
+   here means FIB structure where the next-hop forwarding information is
+   stored separately from the prefix lookup table, and the latter only
+   stores pointers to the respective forwarding information.  See
+   [BGP-PIC] for discussion of FIB hierarchies and fast convergence.
+
+   Even though BGP offers reduced failure scope for some cases, further
+   reduction of the fault domain using summarization is not always
+   possible with the proposed design, since using this technique may
+   create routing black-holes as mentioned previously.  Therefore, the
+   worst failure impact scope on the control plane is the network as a
+   whole -- for instance, in the case of a link failure between Tier 2
+   and Tier 3 devices.  The amount of impacted prefixes in this case
+   would be much less than in the case of a failure in the upper layers
+   of a Clos network topology.  The property of having such large
+   failure scope is not a result of choosing EBGP in the design but
+   rather a result of using the Clos topology.
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 25]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+7.5.  Routing Micro-Loops
+
+   When a downstream device, e.g., Tier 2 device, loses all paths for a
+   prefix, it normally has the default route pointing toward the
+   upstream device -- in this case, the Tier 1 device.  As a result, it
+   is possible to get in the situation where a Tier 2 switch loses a
+   prefix, but a Tier 1 switch still has the path pointing to the Tier 2
+   device; this results in a transient micro-loop, since the Tier 1
+   switch will keep passing packets to the affected prefix back to the
+   Tier 2 device, and the Tier 2 will bounce them back again using the
+   default route.  This micro-loop will last for the time it takes the
+   upstream device to fully update its forwarding tables.
+
+   To minimize impact of such micro-loops, Tier 2 and Tier 1 switches
+   can be configured with static "discard" or "null" routes that will be
+   more specific than the default route for prefixes missing during
+   network convergence.  For Tier 2 switches, the discard route should
+   be a summary route, covering all server subnets of the underlying
+   Tier 3 devices.  For Tier 1 devices, the discard route should be a
+   summary covering the server IP address subnets allocated for the
+   whole data center.  Those discard routes will only take precedence
+   for the duration of network convergence, until the device learns a
+   more specific prefix via a new path.
+
+8.  Additional Options for Design
+
+8.1.  Third-Party Route Injection
+
+   BGP allows for a "third-party", i.e., a directly attached BGP
+   speaker, to inject routes anywhere in the network topology, meeting
+   REQ5.  This can be achieved by peering via a multi-hop BGP session
+   with some or even all devices in the topology.  Furthermore, BGP
+   diverse path distribution [RFC6774] could be used to inject multiple
+   BGP next hops for the same prefix to facilitate load balancing, or
+   using the BGP ADD-PATH capability [RFC7911] if supported by the
+   implementation.  Unfortunately, in many implementations, ADD-PATH has
+   been found to only support IBGP properly in the use cases for which
+   it was originally optimized; this limits the "third-party" peering to
+   IBGP only.
+
+   To implement route injection in the proposed design, a third-party
+   BGP speaker may peer with Tier 3 and Tier 1 switches, injecting the
+   same prefix, but using a special set of BGP next hops for Tier 1
+   devices.  Those next hops are assumed to resolve recursively via BGP,
+   and could be, for example, IP addresses on Tier 3 devices.  The
+   resulting forwarding table programming could provide desired traffic
+   proportion distribution among different clusters.
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 26]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+8.2.  Route Summarization within Clos Topology
+
+   As mentioned previously, route summarization is not possible within
+   the proposed Clos topology since it makes the network susceptible to
+   route black-holing under single link failures.  The main problem is
+   the limited number of redundant paths between network elements, e.g.,
+   there is only a single path between any pair of Tier 1 and Tier 3
+   devices.  However, some operators may find route aggregation
+   desirable to improve control-plane stability.
+
+   If any technique to summarize within the topology is planned,
+   modeling of the routing behavior and potential for black-holing
+   should be done not only for single or multiple link failures, but
+   also for fiber pathway failures or optical domain failures when the
+   topology extends beyond a physical location.  Simple modeling can be
+   done by checking the reachability on devices doing summarization
+   under the condition of a link or pathway failure between a set of
+   devices in every tier as well as to the WAN routers when external
+   connectivity is present.
+
+   Route summarization would be possible with a small modification to
+   the network topology, though the tradeoff would be reduction of the
+   total size of the network as well as network congestion under
+   specific failures.  This approach is very similar to the technique
+   described above, which allows Border Routers to summarize the entire
+   data center address space.
+
+8.2.1.  Collapsing Tier 1 Devices Layer
+
+   In order to add more paths between Tier 1 and Tier 3 devices, group
+   Tier 2 devices into pairs, and then connect the pairs to the same
+   group of Tier 1 devices.  This is logically equivalent to
+   "collapsing" Tier 1 devices into a group of half the size, merging
+   the links on the "collapsed" devices.  The result is illustrated in
+   Figure 6.  For example, in this topology DEV C and DEV D connect to
+   the same set of Tier 1 devices (DEV 1 and DEV 2), whereas before they
+   were connecting to different groups of Tier 1 devices.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 27]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+                    Tier 2       Tier 1       Tier 2
+                   +-----+      +-----+      +-----+
+     +-------------| DEV |------| DEV |------|     |-------------+
+     |       +-----|  C  |--++--|  1  |--++--|     |-----+       |
+     |       |     +-----+  ||  +-----+  ||  +-----+     |       |
+     |       |              ||           ||              |       |
+     |       |     +-----+  ||  +-----+  ||  +-----+     |       |
+     | +-----+-----| DEV |--++--| DEV |--++--|     |-----+-----+ |
+     | |     | +---|  D  |------|  2  |------|     |---+ |     | |
+     | |     | |   +-----+      +-----+      +-----+   | |     | |
+     | |     | |                                       | |     | |
+   +-----+ +-----+                                   +-----+ +-----+
+   | DEV | | DEV |                                   |     | |     |
+   |  A  | |  B  | Tier 3                     Tier 3 |     | |     |
+   +-----+ +-----+                                   +-----+ +-----+
+     | |     | |                                       | |     | |
+     O O     O O             <- Servers ->             O O     O O
+
+                      Figure 6: 5-Stage Clos Topology
+
+   Having this design in place, Tier 2 devices may be configured to
+   advertise only a default route down to Tier 3 devices.  If a link
+   between Tier 2 and Tier 3 fails, the traffic will be re-routed via
+   the second available path known to a Tier 2 switch.  It is still not
+   possible to advertise a summary route covering prefixes for a single
+   cluster from Tier 2 devices since each of them has only a single path
+   down to this prefix.  It would require dual-homed servers to
+   accomplish that.  Also note that this design is only resilient to
+   single link failures.  It is possible for a double link failure to
+   isolate a Tier 2 device from all paths toward a specific Tier 3
+   device, thus causing a routing black-hole.
+
+   A result of the proposed topology modification would be a reduction
+   of the port capacity of Tier 1 devices.  This limits the maximum
+   number of attached Tier 2 devices, and therefore will limit the
+   maximum DC network size.  A larger network would require different
+   Tier 1 devices that have higher port density to implement this
+   change.
+
+   Another problem is traffic rebalancing under link failures.  Since
+   there are two paths from Tier 1 to Tier 3, a failure of the link
+   between Tier 1 and Tier 2 switch would result in all traffic that was
+   taking the failed link to switch to the remaining path.  This will
+   result in doubling the link utilization on the remaining link.
+
+
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 28]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+8.2.2.  Simple Virtual Aggregation
+
+   A completely different approach to route summarization is possible,
+   provided that the main goal is to reduce the FIB size, while allowing
+   the control plane to disseminate full routing information.  Firstly,
+   it could be easily noted that in many cases multiple prefixes, some
+   of which are less specific, share the same set of the next hops (same
+   ECMP group).  For example, from the perspective of Tier 3 devices,
+   all routes learned from upstream Tier 2 devices, including the
+   default route, will share the same set of BGP next hops, provided
+   that there are no failures in the network.  This makes it possible to
+   use the technique similar to that described in [RFC6769] and only
+   install the least specific route in the FIB, ignoring more specific
+   routes if they share the same next-hop set.  For example, under
+   normal network conditions, only the default route needs to be
+   programmed into the FIB.
+
+   Furthermore, if the Tier 2 devices are configured with summary
+   prefixes covering all of their attached Tier 3 device's prefixes, the
+   same logic could be applied in Tier 1 devices as well and, by
+   induction to Tier 2/Tier 3 switches in different clusters.  These
+   summary routes should still allow for more specific prefixes to leak
+   to Tier 1 devices, to enable detection of mismatches in the next-hop
+   sets if a particular link fails, thus changing the next-hop set for a
+   specific prefix.
+
+   Restating once again, this technique does not reduce the amount of
+   control-plane state (i.e., BGP UPDATEs, BGP Loc-RIB size), but only
+   allows for more efficient FIB utilization, by detecting more specific
+   prefixes that share their next-hop set with a subsuming less specific
+   prefix.
+
+8.3.  ICMP Unreachable Message Masquerading
+
+   This section discusses some operational aspects of not advertising
+   point-to-point link subnets into BGP, as previously identified as an
+   option in Section 5.2.3.  The operational impact of this decision
+   could be seen when using the well-known "traceroute" tool.
+   Specifically, IP addresses displayed by the tool will be the link's
+   point-to-point addresses, and hence will be unreachable for
+   management connectivity.  This makes some troubleshooting more
+   complicated.
+
+   One way to overcome this limitation is by using the DNS subsystem to
+   create the "reverse" entries for these point-to-point IP addresses
+   pointing to the same name as the loopback address.  The connectivity
+   then can be made by resolving this name to the "primary" IP address
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 29]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   of the device, e.g., its Loopback interface, which is always
+   advertised into BGP.  However, this creates a dependency on the DNS
+   subsystem, which may be unavailable during an outage.
+
+   Another option is to make the network device perform IP address
+   masquerading, that is, rewriting the source IP addresses of the
+   appropriate ICMP messages sent by the device with the "primary" IP
+   address of the device.  Specifically, the ICMP Destination
+   Unreachable Message (type 3) code 3 (port unreachable) and ICMP Time
+   Exceeded (type 11) code 0 are required for correct operation of the
+   "traceroute" tool.  With this modification, the "traceroute" probes
+   sent to the devices will always be sent back with the "primary" IP
+   address as the source, allowing the operator to discover the
+   "reachable" IP address of the box.  This has the downside of hiding
+   the address of the "entry point" into the device.  If the devices
+   support [RFC5837], this may allow the best of both worlds by
+   providing the information about the incoming interface even if the
+   return address is the "primary" IP address.
+
+9.  Security Considerations
+
+   The design does not introduce any additional security concerns.
+   General BGP security considerations are discussed in [RFC4271] and
+   [RFC4272].  Since a DC is a single-operator domain, this document
+   assumes that edge filtering is in place to prevent attacks against
+   the BGP sessions themselves from outside the perimeter of the DC.
+   This may be a more feasible option for most deployments than having
+   to deal with key management for TCP MD5 as described in [RFC2385] or
+   dealing with the lack of implementations of the TCP Authentication
+   Option [RFC5925] available at the time of publication of this
+   document.  The Generalized TTL Security Mechanism [RFC5082] could
+   also be used to further reduce the risk of BGP session spoofing.
+
+10.  References
+
+10.1.  Normative References
+
+   [RFC4271]  Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
+              Border Gateway Protocol 4 (BGP-4)", RFC 4271,
+              DOI 10.17487/RFC4271, January 2006,
+              <http://www.rfc-editor.org/info/rfc4271>.
+
+   [RFC6996]  Mitchell, J., "Autonomous System (AS) Reservation for
+              Private Use", BCP 6, RFC 6996, DOI 10.17487/RFC6996, July
+              2013, <http://www.rfc-editor.org/info/rfc6996>.
+
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 30]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+10.2.  Informative References
+
+   [ALFARES2008]
+              Al-Fares, M., Loukissas, A., and A. Vahdat, "A Scalable,
+              Commodity Data Center Network Architecture",
+              DOI 10.1145/1402958.1402967, August 2008,
+              <http://dl.acm.org/citation.cfm?id=1402967>.
+
+   [ALLOWASIN]
+              Cisco Systems, "Allowas-in Feature in BGP Configuration
+              Example", February 2015,
+              <http://www.cisco.com/c/en/us/support/docs/ip/
+              border-gateway-protocol-bgp/112236-allowas-in-bgp-config-
+              example.html>.
+
+   [BGP-PIC]  Bashandy, A., Ed., Filsfils, C., and P. Mohapatra, "BGP
+              Prefix Independent Convergence", Work in Progress,
+              draft-ietf-rtgwg-bgp-pic-02, August 2016.
+
+   [CLOS1953] Clos, C., "A Study of Non-Blocking Switching Networks",
+              The Bell System Technical Journal, Vol. 32(2),
+              DOI 10.1002/j.1538-7305.1953.tb01433.x, March 1953.
+
+   [CONDITIONALROUTE]
+              Cisco Systems, "Configuring and Verifying the BGP
+              Conditional Advertisement Feature", August 2005,
+              <http://www.cisco.com/c/en/us/support/docs/ip/
+              border-gateway-protocol-bgp/16137-cond-adv.html>.
+
+   [CONS-HASH]
+              Wikipedia, "Consistent Hashing", July 2016,
+              <https://en.wikipedia.org/w/
+              index.php?title=Consistent_hashing&oldid=728825684>.
+
+   [FB4POST]  Farrington, N. and A. Andreyev, "Facebook's Data Center
+              Network Architecture", May 2013,
+              <http://nathanfarrington.com/papers/facebook-oic13.pdf>.
+
+   [GREENBERG2009]
+              Greenberg, A., Hamilton, J., and D. Maltz, "The Cost of a
+              Cloud: Research Problems in Data Center Networks",
+              DOI 10.1145/1496091.1496103, January 2009,
+              <http://dl.acm.org/citation.cfm?id=1496103>.
+
+   [HADOOP]   Apache, "Apache Hadoop", April 2016,
+              <https://hadoop.apache.org/>.
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 31]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   [IANA.AS]  IANA, "Autonomous System (AS) Numbers",
+              <http://www.iana.org/assignments/as-numbers>.
+
+   [IEEE8021D-1990]
+              IEEE, "IEEE Standard for Local and Metropolitan Area
+              Networks: Media Access Control (MAC) Bridges", IEEE
+              Std 802.1D, DOI 10.1109/IEEESTD.1991.101050, 1991,
+              <http://ieeexplore.ieee.org/servlet/opac?punumber=2255>.
+
+   [IEEE8021D-2004]
+              IEEE, "IEEE Standard for Local and Metropolitan Area
+              Networks: Media Access Control (MAC) Bridges", IEEE
+              Std 802.1D, DOI 10.1109/IEEESTD.2004.94569, June 2004,
+              <http://ieeexplore.ieee.org/servlet/opac?punumber=9155>.
+
+   [IEEE8021Q]
+              IEEE, "IEEE Standard for Local and Metropolitan Area
+              Networks: Bridges and Bridged Networks", IEEE Std 802.1Q,
+              DOI 10.1109/IEEESTD.2014.6991462,
+              <http://ieeexplore.ieee.org/servlet/
+              opac?punumber=6991460>.
+
+   [IEEE8023AD]
+              IEEE, "Amendment to Carrier Sense Multiple Access With
+              Collision Detection (CSMA/CD) Access Method and Physical
+              Layer Specifications - Aggregation of Multiple Link
+              Segments", IEEE Std 802.3ad,
+              DOI 10.1109/IEEESTD.2000.91610, October 2000,
+              <http://ieeexplore.ieee.org/servlet/opac?punumber=6867>.
+
+   [INTERCON] Dally, W. and B. Towles, "Principles and Practices of
+              Interconnection Networks", ISBN 978-0122007514, January
+              2004, <http://dl.acm.org/citation.cfm?id=995703>.
+
+   [JAKMA2008]
+              Jakma, P., "BGP Path Hunting", 2008,
+              <https://blogs.oracle.com/paulj/entry/bgp_path_hunting>.
+
+   [L3DSR]    Schaumann, J., "L3DSR - Overcoming Layer 2 Limitations of
+              Direct Server Return Load Balancing", 2011,
+              <https://www.nanog.org/meetings/nanog51/presentations/
+              Monday/NANOG51.Talk45.nanog51-Schaumann.pdf>.
+
+   [LINK]     Mohapatra, P. and R. Fernando, "BGP Link Bandwidth
+              Extended Community", Work in Progress, draft-ietf-idr-
+              link-bandwidth-06, January 2013.
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 32]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   [REMOVAL]  Mitchell, J., Rao, D., and R. Raszuk, "Private Autonomous
+              System (AS) Removal Requirements", Work in Progress,
+              draft-mitchell-grow-remove-private-as-04, April 2015.
+
+   [RFC2328]  Moy, J., "OSPF Version 2", STD 54, RFC 2328,
+              DOI 10.17487/RFC2328, April 1998,
+              <http://www.rfc-editor.org/info/rfc2328>.
+
+   [RFC2385]  Heffernan, A., "Protection of BGP Sessions via the TCP MD5
+              Signature Option", RFC 2385, DOI 10.17487/RFC2385, August
+              1998, <http://www.rfc-editor.org/info/rfc2385>.
+
+   [RFC2992]  Hopps, C., "Analysis of an Equal-Cost Multi-Path
+              Algorithm", RFC 2992, DOI 10.17487/RFC2992, November 2000,
+              <http://www.rfc-editor.org/info/rfc2992>.
+
+   [RFC4272]  Murphy, S., "BGP Security Vulnerabilities Analysis",
+              RFC 4272, DOI 10.17487/RFC4272, January 2006,
+              <http://www.rfc-editor.org/info/rfc4272>.
+
+   [RFC4277]  McPherson, D. and K. Patel, "Experience with the BGP-4
+              Protocol", RFC 4277, DOI 10.17487/RFC4277, January 2006,
+              <http://www.rfc-editor.org/info/rfc4277>.
+
+   [RFC4786]  Abley, J. and K. Lindqvist, "Operation of Anycast
+              Services", BCP 126, RFC 4786, DOI 10.17487/RFC4786,
+              December 2006, <http://www.rfc-editor.org/info/rfc4786>.
+
+   [RFC5082]  Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C.
+              Pignataro, "The Generalized TTL Security Mechanism
+              (GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007,
+              <http://www.rfc-editor.org/info/rfc5082>.
+
+   [RFC5837]  Atlas, A., Ed., Bonica, R., Ed., Pignataro, C., Ed., Shen,
+              N., and JR. Rivers, "Extending ICMP for Interface and
+              Next-Hop Identification", RFC 5837, DOI 10.17487/RFC5837,
+              April 2010, <http://www.rfc-editor.org/info/rfc5837>.
+
+   [RFC5880]  Katz, D. and D. Ward, "Bidirectional Forwarding Detection
+              (BFD)", RFC 5880, DOI 10.17487/RFC5880, June 2010,
+              <http://www.rfc-editor.org/info/rfc5880>.
+
+   [RFC5925]  Touch, J., Mankin, A., and R. Bonica, "The TCP
+              Authentication Option", RFC 5925, DOI 10.17487/RFC5925,
+              June 2010, <http://www.rfc-editor.org/info/rfc5925>.
+
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 33]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+   [RFC6325]  Perlman, R., Eastlake 3rd, D., Dutt, D., Gai, S., and A.
+              Ghanwani, "Routing Bridges (RBridges): Base Protocol
+              Specification", RFC 6325, DOI 10.17487/RFC6325, July 2011,
+              <http://www.rfc-editor.org/info/rfc6325>.
+
+   [RFC6769]  Raszuk, R., Heitz, J., Lo, A., Zhang, L., and X. Xu,
+              "Simple Virtual Aggregation (S-VA)", RFC 6769,
+              DOI 10.17487/RFC6769, October 2012,
+              <http://www.rfc-editor.org/info/rfc6769>.
+
+   [RFC6774]  Raszuk, R., Ed., Fernando, R., Patel, K., McPherson, D.,
+              and K. Kumaki, "Distribution of Diverse BGP Paths",
+              RFC 6774, DOI 10.17487/RFC6774, November 2012,
+              <http://www.rfc-editor.org/info/rfc6774>.
+
+   [RFC6793]  Vohra, Q. and E. Chen, "BGP Support for Four-Octet
+              Autonomous System (AS) Number Space", RFC 6793,
+              DOI 10.17487/RFC6793, December 2012,
+              <http://www.rfc-editor.org/info/rfc6793>.
+
+   [RFC7067]  Dunbar, L., Eastlake 3rd, D., Perlman, R., and I.
+              Gashinsky, "Directory Assistance Problem and High-Level
+              Design Proposal", RFC 7067, DOI 10.17487/RFC7067, November
+              2013, <http://www.rfc-editor.org/info/rfc7067>.
+
+   [RFC7130]  Bhatia, M., Ed., Chen, M., Ed., Boutros, S., Ed.,
+              Binderberger, M., Ed., and J. Haas, Ed., "Bidirectional
+              Forwarding Detection (BFD) on Link Aggregation Group (LAG)
+              Interfaces", RFC 7130, DOI 10.17487/RFC7130, February
+              2014, <http://www.rfc-editor.org/info/rfc7130>.
+
+   [RFC7196]  Pelsser, C., Bush, R., Patel, K., Mohapatra, P., and O.
+              Maennel, "Making Route Flap Damping Usable", RFC 7196,
+              DOI 10.17487/RFC7196, May 2014,
+              <http://www.rfc-editor.org/info/rfc7196>.
+
+   [RFC7911]  Walton, D., Retana, A., Chen, E., and J. Scudder,
+              "Advertisement of Multiple Paths in BGP", RFC 7911,
+              DOI 10.17487/RFC7911, July 2016,
+              <http://www.rfc-editor.org/info/rfc7911>.
+
+   [VENDOR-REMOVE-PRIVATE-AS]
+              Cisco Systems, "Removing Private Autonomous System Numbers
+              in BGP", August 2005,
+              <http://www.cisco.com/en/US/tech/tk365/
+              technologies_tech_note09186a0080093f27.shtml>.
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 34]
+
+RFC 7938               BGP Routing in Data Centers           August 2016
+
+
+Acknowledgements
+
+   This publication summarizes the work of many people who participated
+   in developing, testing, and deploying the proposed network design,
+   some of whom were George Chen, Parantap Lahiri, Dave Maltz, Edet
+   Nkposong, Robert Toomey, and Lihua Yuan.  The authors would also like
+   to thank Linda Dunbar, Anoop Ghanwani, Susan Hares, Danny McPherson,
+   Robert Raszuk, and Russ White for reviewing this document and
+   providing valuable feedback, and Mary Mitchell for initial grammar
+   and style suggestions.
+
+Authors' Addresses
+
+   Petr Lapukhov
+   Facebook
+   1 Hacker Way
+   Menlo Park, CA  94025
+   United States of America
+
+   Email: petr@fb.com
+
+
+   Ariff Premji
+   Arista Networks
+   5453 Great America Parkway
+   Santa Clara, CA  95054
+   United States of America
+
+   Email: ariff@arista.com
+   URI:   http://arista.com/
+
+
+   Jon Mitchell (editor)
+
+   Email: jrmitche@puck.nether.net
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Lapukhov, et al.              Informational                    [Page 35]
+