summaryrefslogtreecommitdiff
path: root/doc/rfc/rfc9136.txt
diff options
context:
space:
mode:
authorThomas Voss <mail@thomasvoss.com> 2024-11-27 20:54:24 +0100
committerThomas Voss <mail@thomasvoss.com> 2024-11-27 20:54:24 +0100
commit4bfd864f10b68b71482b35c818559068ef8d5797 (patch)
treee3989f47a7994642eb325063d46e8f08ffa681dc /doc/rfc/rfc9136.txt
parentea76e11061bda059ae9f9ad130a9895cc85607db (diff)
doc: Add RFC documents
Diffstat (limited to 'doc/rfc/rfc9136.txt')
-rw-r--r--doc/rfc/rfc9136.txt1710
1 files changed, 1710 insertions, 0 deletions
diff --git a/doc/rfc/rfc9136.txt b/doc/rfc/rfc9136.txt
new file mode 100644
index 0000000..6953dc3
--- /dev/null
+++ b/doc/rfc/rfc9136.txt
@@ -0,0 +1,1710 @@
+
+
+
+
+Internet Engineering Task Force (IETF) J. Rabadan, Ed.
+Request for Comments: 9136 W. Henderickx
+Category: Standards Track Nokia
+ISSN: 2070-1721 J. Drake
+ W. Lin
+ Juniper
+ A. Sajassi
+ Cisco
+ October 2021
+
+
+ IP Prefix Advertisement in Ethernet VPN (EVPN)
+
+Abstract
+
+ The BGP MPLS-based Ethernet VPN (EVPN) (RFC 7432) mechanism provides
+ a flexible control plane that allows intra-subnet connectivity in an
+ MPLS and/or Network Virtualization Overlay (NVO) (RFC 7365) network.
+ In some networks, there is also a need for dynamic and efficient
+ inter-subnet connectivity across Tenant Systems and end devices that
+ can be physical or virtual and do not necessarily participate in
+ dynamic routing protocols. This document defines a new EVPN route
+ type for the advertisement of IP prefixes and explains some use-case
+ examples where this new route type is used.
+
+Status of This Memo
+
+ This is an Internet Standards Track document.
+
+ This document is a product of the Internet Engineering Task Force
+ (IETF). It represents the consensus of the IETF community. It has
+ received public review and has been approved for publication by the
+ Internet Engineering Steering Group (IESG). Further information on
+ Internet Standards is available in Section 2 of RFC 7841.
+
+ Information about the current status of this document, any errata,
+ and how to provide feedback on it may be obtained at
+ https://www.rfc-editor.org/info/rfc9136.
+
+Copyright Notice
+
+ Copyright (c) 2021 IETF Trust and the persons identified as the
+ document authors. All rights reserved.
+
+ This document is subject to BCP 78 and the IETF Trust's Legal
+ Provisions Relating to IETF Documents
+ (https://trustee.ietf.org/license-info) in effect on the date of
+ publication of this document. Please review these documents
+ carefully, as they describe your rights and restrictions with respect
+ to this document. Code Components extracted from this document must
+ include Simplified BSD License text as described in Section 4.e of
+ the Trust Legal Provisions and are provided without warranty as
+ described in the Simplified BSD License.
+
+Table of Contents
+
+ 1. Introduction
+ 1.1. Terminology
+ 2. Problem Statement
+ 2.1. Inter-Subnet Connectivity Requirements in Data Centers
+ 2.2. The Need for the EVPN IP Prefix Route
+ 3. The BGP EVPN IP Prefix Route
+ 3.1. IP Prefix Route Encoding
+ 3.2. Overlay Indexes and Recursive Lookup Resolution
+ 4. Overlay Index Use Cases
+ 4.1. TS IP Address Overlay Index Use Case
+ 4.2. Floating IP Overlay Index Use Case
+ 4.3. Bump-in-the-Wire Use Case
+ 4.4. IP-VRF-to-IP-VRF Model
+ 4.4.1. Interface-less IP-VRF-to-IP-VRF Model
+ 4.4.2. Interface-ful IP-VRF-to-IP-VRF with SBD IRB
+ 4.4.3. Interface-ful IP-VRF-to-IP-VRF with Unnumbered SBD IRB
+ 5. Security Considerations
+ 6. IANA Considerations
+ 7. References
+ 7.1. Normative References
+ 7.2. Informative References
+ Acknowledgments
+ Contributors
+ Authors' Addresses
+
+1. Introduction
+
+ [RFC7365] provides a framework for Data Center (DC) Network
+ Virtualization over Layer 3 and specifies that the Network
+ Virtualization Edge (NVE) devices must provide Layer 2 and Layer 3
+ virtualized network services in multi-tenant DCs. [RFC8365]
+ discusses the use of EVPN as the technology of choice to provide
+ Layer 2 or intra-subnet services in these DCs. This document, along
+ with [RFC9135], specifies the use of EVPN for Layer 3 or inter-subnet
+ connectivity services.
+
+ [RFC9135] defines some fairly common inter-subnet forwarding
+ scenarios where Tenant Systems (TSs) can exchange packets with TSs
+ located in remote subnets. In order to achieve this, [RFC9135]
+ describes how Media Access Control (MAC) and IPs encoded in TS RT-2
+ routes are not only used to populate MAC Virtual Routing and
+ Forwarding (MAC-VRF) and overlay Address Resolution Protocol (ARP)
+ tables but also IP-VRF tables with the encoded TS host routes (/32 or
+ /128). In some cases, EVPN may advertise IP prefixes and therefore
+ provide aggregation in the IP-VRF tables, as opposed to propagating
+ individual host routes. This document complements the scenarios
+ described in [RFC9135] and defines how EVPN may be used to advertise
+ IP prefixes. Interoperability between EVPN and Layer 3 Virtual
+ Private Network (VPN) [RFC4364] IP Prefix routes is out of the scope
+ of this document.
+
+ Section 2.1 describes the inter-subnet connectivity requirements in
+ DCs. Section 2.2 explains why a new EVPN route type is required for
+ IP prefix advertisements. Sections 3, 4, and 5 will describe this
+ route type and how it is used in some specific use cases.
+
+1.1. Terminology
+
+ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+ "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
+ "OPTIONAL" in this document are to be interpreted as described in BCP
+ 14 [RFC2119] [RFC8174] when, and only when, they appear in all
+ capitals, as shown here.
+
+ AC: Attachment Circuit
+
+ ARP: Address Resolution Protocol
+
+ BD: Broadcast Domain. As per [RFC7432], an EVI consists of a
+ single BD or multiple BDs. In case of VLAN-bundle and
+ VLAN-based service models (see [RFC7432]), a BD is
+ equivalent to an EVI. In case of a VLAN-aware bundle
+ service model, an EVI contains multiple BDs. Also, in this
+ document, "BD" and "subnet" are equivalent terms.
+
+ BD Route Target: Refers to the broadcast-domain-assigned Route
+ Target [RFC4364]. In case of a VLAN-aware bundle service
+ model, all the BD instances in the MAC-VRF share the same
+ Route Target.
+
+ BT: Bridge Table. The instantiation of a BD in a MAC-VRF, as
+ per [RFC7432].
+
+ CE: Customer Edge
+
+ DA: Destination Address
+
+ DGW: Data Center Gateway
+
+ Ethernet A-D Route: Ethernet Auto-Discovery (A-D) route, as per
+ [RFC7432].
+
+ Ethernet NVO Tunnel: Refers to Network Virtualization Overlay
+ tunnels with Ethernet payload. Examples of this type of
+ tunnel are VXLAN or GENEVE.
+
+ EVI: EVPN Instance spanning the NVE/PE devices that are
+ participating on that EVPN, as per [RFC7432].
+
+ EVPN: Ethernet VPN, as per [RFC7432].
+
+ GENEVE: Generic Network Virtualization Encapsulation, as per
+ [RFC8926].
+
+ GRE: Generic Routing Encapsulation
+
+ GW IP: Gateway IP address
+
+ IPL: IP Prefix Length
+
+ IP NVO Tunnel: Refers to Network Virtualization Overlay tunnels with
+ IP payload (no MAC header in the payload).
+
+ IP-VRF: A Virtual Routing and Forwarding table for IP routes on an
+ NVE/PE. The IP routes could be populated by EVPN and IP-
+ VPN address families. An IP-VRF is also an instantiation
+ of a Layer 3 VPN in an NVE/PE.
+
+ IRB: Integrated Routing and Bridging interface. It connects an
+ IP-VRF to a BD (or subnet).
+
+ MAC: Media Access Control
+
+ MAC-VRF: A Virtual Routing and Forwarding table for MAC addresses on
+ an NVE/PE, as per [RFC7432]. A MAC-VRF is also an
+ instantiation of an EVI in an NVE/PE.
+
+ ML: MAC Address Length
+
+ ND: Neighbor Discovery
+
+ NVE: Network Virtualization Edge
+
+ NVO: Network Virtualization Overlay
+
+ PE: Provider Edge
+
+ RT-2: EVPN Route Type 2, i.e., MAC/IP Advertisement route, as
+ defined in [RFC7432].
+
+ RT-5: EVPN Route Type 5, i.e., IP Prefix route, as defined in
+ Section 3.
+
+ SBD: Supplementary Broadcast Domain. A BD that does not have
+ any ACs, only IRB interfaces, and is used to provide
+ connectivity among all the IP-VRFs of the tenant. The SBD
+ is only required in IP-VRF-to-IP-VRF use cases (see
+ Section 4.4).
+
+ SN: Subnet
+
+ TS: Tenant System
+
+ VA: Virtual Appliance
+
+ VM: Virtual Machine
+
+ VNI: Virtual Network Identifier. As in [RFC8365], the term is
+ used as a representation of a 24-bit NVO instance
+ identifier, with the understanding that "VNI" will refer to
+ a VXLAN Network Identifier in VXLAN, or a Virtual Network
+ Identifier in GENEVE, etc., unless it is stated otherwise.
+
+ VSID: Virtual Subnet Identifier
+
+ VTEP: VXLAN Termination End Point, as per [RFC7348].
+
+ VXLAN: Virtual eXtensible Local Area Network, as per [RFC7348].
+
+ This document also assumes familiarity with the terminology of
+ [RFC7365], [RFC7432], and [RFC8365].
+
+2. Problem Statement
+
+ This section describes the inter-subnet connectivity requirements in
+ DCs and why a specific route type to advertise IP prefixes is needed.
+
+2.1. Inter-Subnet Connectivity Requirements in Data Centers
+
+ [RFC7432] is used as the control plane for an NVO solution in DCs,
+ where NVE devices can be located in hypervisors or Top-of-Rack (ToR)
+ switches, as described in [RFC8365].
+
+ The following considerations apply to TSs that are physical or
+ virtual systems identified by MAC (and possibly IP addresses) and are
+ connected to BDs by Attachment Circuits:
+
+ * The Tenant Systems may be VMs that generate traffic from their own
+ MAC and IP.
+
+ * The Tenant Systems may be VA entities that forward traffic to/from
+ IP addresses of different end devices sitting behind them.
+
+ - These VAs can be firewalls, load balancers, NAT devices, other
+ appliances, or virtual gateways with virtual routing instances.
+
+ - These VAs do not necessarily participate in dynamic routing
+ protocols and hence rely on the EVPN NVEs to advertise the
+ routes on their behalf.
+
+ - In all these cases, the VA will forward traffic to other TSs
+ using its own source MAC, but the source IP will be the one
+ associated with the end device sitting behind the VA or a
+ translated IP address (part of a public NAT pool) if the VA is
+ performing NAT.
+
+ - Note that the same IP address and endpoint could exist behind
+ two of these TSs. One example of this would be certain
+ appliance resiliency mechanisms, where a virtual IP or floating
+ IP can be owned by one of the two VAs running the resiliency
+ protocol (the Master VA). The Virtual Router Redundancy
+ Protocol (VRRP) [RFC5798] is one particular example of this.
+ Another example is multihomed subnets, i.e., the same subnet is
+ connected to two VAs.
+
+ - Although these VAs provide IP connectivity to VMs and the
+ subnets behind them, they do not always have their own IP
+ interface connected to the EVPN NVE; Layer 2 firewalls are
+ examples of VAs not supporting IP interfaces.
+
+ Figure 1 illustrates some of the examples described above.
+
+ NVE1
+ +-----------+
+ TS1(VM)--| (BD-10) |-----+
+ M1/IP1 +-----------+ | DGW1
+ +---------+ +-------------+
+ | |----| (BD-10) |
+ SN1---+ NVE2 | | | IRB1\ |
+ | +-----------+ | | | (IP-VRF)|---+
+ SN2---TS2(VA)--| (BD-10) |-| | +-------------+ _|_
+ | M2/IP2 +-----------+ | VXLAN/ | ( )
+ IP4---+ <-+ | GENEVE | DGW2 ( WAN )
+ | | | +-------------+ (___)
+ vIP23 (floating) | |----| (BD-10) | |
+ | +---------+ | IRB2\ | |
+ SN1---+ <-+ NVE3 | | | | (IP-VRF)|---+
+ | M3/IP3 +-----------+ | | | +-------------+
+ SN3---TS3(VA)--| (BD-10) |---+ | |
+ | +-----------+ | |
+ IP5---+ | |
+ | |
+ NVE4 | | NVE5 +--SN5
+ +---------------------+ | | +-----------+ |
+ IP6------| (BD-1) | | +-| (BD-10) |--TS4(VA)--SN6
+ | \ | | +-----------+ |
+ | (IP-VRF) |--+ ESI4 +--SN7
+ | / \IRB3 |
+ |---| (BD-2) (BD-10) |
+ SN4| +---------------------+
+
+
+ Note:
+ ESI4 = Ethernet Segment Identifier 4
+
+ Figure 1: DC Inter-subnet Use Cases
+
+ Where:
+
+ NVE1, NVE2, NVE3, NVE4, NVE5, DGW1, and DGW2 share the same BD for a
+ particular tenant. BD-10 is comprised of the collection of BD
+ instances defined in all the NVEs. All the hosts connected to BD-10
+ belong to the same IP subnet. The hosts connected to BD-10 are
+ listed below:
+
+ * TS1 is a VM that generates/receives traffic to/from IP1, where IP1
+ belongs to the BD-10 subnet.
+
+ * TS2 and TS3 are VAs that send/receive traffic to/from the subnets
+ and hosts sitting behind them (SN1, SN2, SN3, IP4, and IP5).
+ Their IP addresses (IP2 and IP3) belong to the BD-10 subnet, and
+ they can also generate/receive traffic. When these VAs receive
+ packets destined to their own MAC addresses (M2 and M3), they will
+ route the packets to the proper subnet or host. These VAs do not
+ support routing protocols to advertise the subnets connected to
+ them and can move to a different server and NVE when the cloud
+ management system decides to do so. These VAs may also support
+ redundancy mechanisms for some subnets, similar to VRRP, where a
+ floating IP is owned by the Master VA and only the Master VA
+ forwards traffic to a given subnet. For example, vIP23 in
+ Figure 1 is a floating IP that can be owned by TS2 or TS3
+ depending on which system is the Master. Only the Master will
+ forward traffic to SN1.
+
+ * Integrated Routing and Bridging interfaces IRB1, IRB2, and IRB3
+ have their own IP addresses that belong to the BD-10 subnet too.
+ These IRB interfaces connect the BD-10 subnet to Virtual Routing
+ and Forwarding (IP-VRF) instances that can route the traffic to
+ other subnets for the same tenant (within the DC or at the other
+ end of the WAN).
+
+ * TS4 is a Layer 2 VA that provides connectivity to subnets SN5,
+ SN6, and SN7 but does not have an IP address itself in the BD-10.
+ TS4 is connected to a port on NVE5 that is assigned to Ethernet
+ Segment Identifier 4 (ESI4).
+
+ For a BD to which an ingress NVE is attached, "Overlay Index" is
+ defined as an identifier that the ingress EVPN NVE requires in order
+ to forward packets to a subnet or host in a remote subnet. As an
+ example, vIP23 (Figure 1) is an Overlay Index that any NVE attached
+ to BD-10 needs to know in order to forward packets to SN1. The IRB3
+ IP address is an Overlay Index required to get to SN4, and ESI4 is an
+ Overlay Index needed to forward traffic to SN5. In other words, the
+ Overlay Index is a next hop in the overlay address space that can be
+ an IP address, a MAC address, or an ESI. When advertised along with
+ an IP prefix, the Overlay Index requires a recursive resolution to
+ find out the egress NVE to which the EVPN packets need to be sent.
+
+ All the DC use cases in Figure 1 require inter-subnet forwarding;
+ therefore, the individual host routes and subnets:
+
+ a) must be advertised from the NVEs (since VAs and VMs do not
+ participate in dynamic routing protocols) and
+
+ b) may be associated with an Overlay Index that can be a VA IP
+ address, a floating IP address, a MAC address, or an ESI. The
+ Overlay Index is further discussed in Section 3.2.
+
+2.2. The Need for the EVPN IP Prefix Route
+
+ [RFC7432] defines a MAC/IP Advertisement route (also referred to as
+ "RT-2") where a MAC address can be advertised together with an IP
+ address length and IP address (IP). While a variable IP address
+ length might have been used to indicate the presence of an IP prefix
+ in a route type 2, there are several specific use cases in which
+ using this route type to deliver IP prefixes is not suitable.
+
+ One example of such use cases is the "floating IP" example described
+ in Section 2.1. In this example, it is necessary to decouple the
+ advertisement of the prefixes from the advertisement of a MAC address
+ of either M2 or M3; otherwise, the solution gets highly inefficient
+ and does not scale.
+
+ For example, if 1,000 prefixes are advertised from M2 (using RT-2)
+ and the floating IP owner changes from M2 to M3, 1,000 routes would
+ be withdrawn by M2 and readvertised by M3. However, if a separate
+ route type is used, 1,000 routes can be advertised as associated with
+ the floating IP address (vIP23), and only one RT-2 can be used for
+ advertising the ownership of the floating IP, i.e., vIP23 and M2 in
+ the route type 2. When the floating IP owner changes from M2 to M3,
+ a single RT-2 withdrawal/update is required to indicate the change.
+ The remote DGW will not change any of the 1,000 prefixes associated
+ with vIP23 but will only update the ARP resolution entry for vIP23
+ (now pointing at M3).
+
+ An EVPN route (type 5) for the advertisement of IP prefixes is
+ described in this document. This new route type has a differentiated
+ role from the RT-2 route and addresses the inter-subnet connectivity
+ scenarios for DCs (or NVO-based networks in general) described in
+ this document. Using this new RT-5, an IP prefix may be advertised
+ along with an Overlay Index, which can be a GW IP address, a MAC, or
+ an ESI. The IP prefix may also be advertised without an Overlay
+ Index, in which case the BGP next hop will point at the egress NVE,
+ Area Border Router (ABR), or ASBR, and the MAC in the EVPN Router's
+ MAC Extended Community will provide the inner MAC destination address
+ to be used. As discussed throughout the document, the EVPN RT-2 does
+ not meet the requirements for all the DC use cases; therefore, this
+ EVPN route type 5 is required.
+
+ The EVPN route type 5 decouples the IP prefix advertisements from the
+ MAC/IP Advertisement routes in EVPN. Hence:
+
+ a) The clean and clear advertisements of IPv4 or IPv6 prefixes in a
+ Network Layer Reachability Information (NLRI) message without MAC
+ addresses are allowed.
+
+ b) Since the route type is different from the MAC/IP Advertisement
+ route, the current procedures described in [RFC7432] do not need
+ to be modified.
+
+ c) A flexible implementation is allowed where the prefix can be
+ linked to different types of Overlay/Underlay Indexes: overlay IP
+ addresses, overlay MAC addresses, overlay ESIs, underlay BGP next
+ hops, etc.
+
+ d) An EVPN implementation not requiring IP prefixes can simply
+ discard them by looking at the route type value.
+
+ The following sections describe how EVPN is extended with a route
+ type for the advertisement of IP prefixes and how this route is used
+ to address the inter-subnet connectivity requirements existing in the
+ DC.
+
+3. The BGP EVPN IP Prefix Route
+
+ The BGP EVPN NLRI as defined in [RFC7432] is shown below:
+
+ +-----------------------------------+
+ | Route Type (1 octet) |
+ +-----------------------------------+
+ | Length (1 octet) |
+ +-----------------------------------+
+ | Route Type specific (variable) |
+ +-----------------------------------+
+
+ Figure 2: BGP EVPN NLRI
+
+ This document defines an additional route type (RT-5) in the IANA
+ "EVPN Route Types" registry [EVPNRouteTypes] to be used for the
+ advertisement of EVPN routes using IP prefixes:
+
+ Value: 5
+ Description: IP Prefix
+
+ According to Section 5.4 of [RFC7606], a node that doesn't recognize
+ the route type 5 (RT-5) will ignore it. Therefore, an NVE following
+ this document can still be attached to a BD where an NVE ignoring RT-
+ 5s is attached. Regular procedures described in [RFC7432] would
+ apply in that case for both NVEs. In case two or more NVEs are
+ attached to different BDs of the same tenant, they MUST support the
+ RT-5 for the proper inter-subnet forwarding operation of the tenant.
+
+ The detailed encoding of this route and associated procedures are
+ described in the following sections.
+
+3.1. IP Prefix Route Encoding
+
+ An IP Prefix route type for IPv4 has the Length field set to 34 and
+ consists of the following fields:
+
+ +---------------------------------------+
+ | RD (8 octets) |
+ +---------------------------------------+
+ |Ethernet Segment Identifier (10 octets)|
+ +---------------------------------------+
+ | Ethernet Tag ID (4 octets) |
+ +---------------------------------------+
+ | IP Prefix Length (1 octet, 0 to 32) |
+ +---------------------------------------+
+ | IP Prefix (4 octets) |
+ +---------------------------------------+
+ | GW IP Address (4 octets) |
+ +---------------------------------------+
+ | MPLS Label (3 octets) |
+ +---------------------------------------+
+
+ Figure 3: EVPN IP Prefix Route NLRI for IPv4
+
+ An IP Prefix route type for IPv6 has the Length field set to 58 and
+ consists of the following fields:
+
+ +---------------------------------------+
+ | RD (8 octets) |
+ +---------------------------------------+
+ |Ethernet Segment Identifier (10 octets)|
+ +---------------------------------------+
+ | Ethernet Tag ID (4 octets) |
+ +---------------------------------------+
+ | IP Prefix Length (1 octet, 0 to 128) |
+ +---------------------------------------+
+ | IP Prefix (16 octets) |
+ +---------------------------------------+
+ | GW IP Address (16 octets) |
+ +---------------------------------------+
+ | MPLS Label (3 octets) |
+ +---------------------------------------+
+
+ Figure 4: EVPN IP Prefix Route NLRI for IPv6
+
+ Where:
+
+ * The Length field of the BGP EVPN NLRI for an EVPN IP Prefix route
+ MUST be either 34 (if IPv4 addresses are carried) or 58 (if IPv6
+ addresses are carried). The IP prefix and gateway IP address MUST
+ be from the same IP address family.
+
+ * The Route Distinguisher (RD) and Ethernet Tag ID MUST be used as
+ defined in [RFC7432] and [RFC8365]. In particular, the RD is
+ unique per MAC-VRF (or IP-VRF). The MPLS Label field is set to
+ either an MPLS label or a VNI, as described in [RFC8365] for other
+ EVPN route types.
+
+ * The Ethernet Segment Identifier MUST be a non-zero 10-octet
+ identifier if the ESI is used as an Overlay Index (see the
+ definition of "Overlay Index" in Section 3.2). It MUST be all
+ bytes zero otherwise. The ESI format is described in [RFC7432].
+
+ * The IP prefix length can be set to a value between 0 and 32 (bits)
+ for IPv4 and between 0 and 128 for IPv6, and it specifies the
+ number of bits in the prefix. The value MUST NOT be greater than
+ 128.
+
+ * The IP prefix is a 4- or 16-octet field (IPv4 or IPv6).
+
+ * The GW IP Address field is a 4- or 16-octet field (IPv4 or IPv6)
+ and will encode a valid IP address as an Overlay Index for the IP
+ prefixes. The GW IP field MUST be all bytes zero if it is not
+ used as an Overlay Index. Refer to Section 3.2 for the definition
+ and use of the Overlay Index.
+
+ * The MPLS Label field is encoded as 3 octets, where the high-order
+ 20 bits contain the label value, as per [RFC7432]. When sending,
+ the label value SHOULD be zero if a recursive resolution based on
+ an Overlay Index is used. If the received MPLS label value is
+ zero, the route MUST contain an Overlay Index, and the ingress
+ NVE/PE MUST perform a recursive resolution to find the egress NVE/
+ PE. If the received label is zero and the route does not contain
+ an Overlay Index, it MUST be "treat as withdraw" [RFC7606].
+
+ The RD, Ethernet Tag ID, IP prefix length, and IP prefix are part of
+ the route key used by BGP to compare routes. The rest of the fields
+ are not part of the route key.
+
+ An IP Prefix route MAY be sent along with an EVPN Router's MAC
+ Extended Community (defined in [RFC9135]) to carry the MAC address
+ that is used as the Overlay Index. Note that the MAC address may be
+ that of a TS.
+
+ As described in Section 3.2, certain data combinations in a received
+ route would imply a treat-as-withdraw handling of the route
+ [RFC7606].
+
+3.2. Overlay Indexes and Recursive Lookup Resolution
+
+ RT-5 routes support recursive lookup resolution through the use of
+ Overlay Indexes as follows:
+
+ * An Overlay Index can be an ESI or IP address in the address space
+ of the tenant or MAC address, and it is used by an NVE as the next
+ hop for a given IP prefix. An Overlay Index always needs a
+ recursive route resolution on the NVE/PE that installs the RT-5
+ into one of its IP-VRFs so that the NVE knows to which egress NVE/
+ PE it needs to forward the packets. It is important to note that
+ recursive resolution of the Overlay Index applies upon
+ installation into an IP-VRF and not upon BGP propagation (for
+ instance, on an ASBR). Also, as a result of the recursive
+ resolution, the egress NVE/PE is not necessarily the same NVE that
+ originated the RT-5.
+
+ * The Overlay Index is indicated along with the RT-5 in the ESI
+ field, GW IP field, or EVPN Router's MAC Extended Community,
+ depending on whether the IP prefix next hop is an ESI, an IP
+ address, or a MAC address in the tenant space. The Overlay Index
+ for a given IP prefix is set by local policy at the NVE that
+ originates an RT-5 for that IP prefix (typically managed by the
+ cloud management system).
+
+ * In order to enable the recursive lookup resolution at the ingress
+ NVE, an NVE that is a possible egress NVE for a given Overlay
+ Index must originate a route advertising itself as the BGP next
+ hop on the path to the system denoted by the Overlay Index. For
+ instance:
+
+ - If an NVE receives an RT-5 that specifies an Overlay Index, the
+ NVE cannot use the RT-5 in its IP-VRF unless (or until) it can
+ recursively resolve the Overlay Index.
+
+ - If the RT-5 specifies an ESI as the Overlay Index, a recursive
+ resolution can only be done if the NVE has received and
+ installed an RT-1 (auto-discovery per EVI) route specifying
+ that ESI.
+
+ - If the RT-5 specifies a GW IP address as the Overlay Index, a
+ recursive resolution can only be done if the NVE has received
+ and installed an RT-2 (MAC/IP Advertisement route) specifying
+ that IP address in the IP Address field of its NLRI.
+
+ - If the RT-5 specifies a MAC address as the Overlay Index, a
+ recursive resolution can only be done if the NVE has received
+ and installed an RT-2 (MAC/IP Advertisement route) specifying
+ that MAC address in the MAC Address field of its NLRI.
+
+ Note that the RT-1 or RT-2 routes needed for the recursive
+ resolution may arrive before or after the given RT-5 route.
+
+ * Irrespective of the recursive resolution, if there is no IGP or
+ BGP route to the BGP next hop of an RT-5, BGP MUST NOT install the
+ RT-5 even if the Overlay Index can be resolved.
+
+ * The ESI and GW IP fields may both be zero at the same time.
+ However, they MUST NOT both be non-zero at the same time. A route
+ containing a non-zero GW IP and a non-zero ESI (at the same time)
+ SHOULD be treat as withdraw [RFC7606].
+
+ * If either the ESI or the GW IP are non-zero, then the non-zero one
+ is the Overlay Index, regardless of whether the EVPN Router's MAC
+ Extended Community is present or the value of the label. In case
+ the GW IP is the Overlay Index (hence, ESI is zero), the EVPN
+ Router's MAC Extended Community is ignored if present.
+
+ * A route where ESI, GW IP, MAC, and Label are all zero at the same
+ time SHOULD be treat as withdraw.
+
+ The indirection provided by the Overlay Index and its recursive
+ lookup resolution is required to achieve fast convergence in case of
+ a failure of the object represented by the Overlay Index (see the
+ example described in Section 2.2).
+
+ Table 1 shows the different RT-5 field combinations allowed by this
+ specification and what Overlay Index must be used by the receiving
+ NVE/PE in each case. Cases where there is no Overlay Index are
+ indicated as "None" in Table 1. If there is no Overlay Index, the
+ receiving NVE/PE will not perform any recursive resolution, and the
+ actual next hop is given by the RT-5's BGP next hop.
+
+ +==========+==========+==========+============+===============+
+ | ESI | GW IP | MAC* | Label | Overlay Index |
+ +==========+==========+==========+============+===============+
+ | Non-Zero | Zero | Zero | Don't Care | ESI |
+ +----------+----------+----------+------------+---------------+
+ | Non-Zero | Zero | Non-Zero | Don't Care | ESI |
+ +----------+----------+----------+------------+---------------+
+ | Zero | Non-Zero | Zero | Don't Care | GW IP |
+ +----------+----------+----------+------------+---------------+
+ | Zero | Zero | Non-Zero | Zero | MAC |
+ +----------+----------+----------+------------+---------------+
+ | Zero | Zero | Non-Zero | Non-Zero | MAC or None** |
+ +----------+----------+----------+------------+---------------+
+ | Zero | Zero | Zero | Non-Zero | None*** |
+ +----------+----------+----------+------------+---------------+
+
+ Table 1: RT-5 Fields and Indicated Overlay Index
+
+ Table Notes:
+
+ * MAC with "Zero" value means no EVPN Router's MAC Extended
+ Community is present along with the RT-5. "Non-Zero" indicates
+ that the extended community is present and carries a valid MAC
+ address. The encoding of a MAC address MUST be the 6-octet MAC
+ address specified by [IEEE-802.1Q]. Examples of invalid MAC
+ addresses are broadcast or multicast MAC addresses. The route
+ MUST be treat as withdraw in case of an invalid MAC address.
+ The presence of the EVPN Router's MAC Extended Community alone
+ is not enough to indicate the use of the MAC address as the
+ Overlay Index since the extended community can be used for
+ other purposes.
+
+ ** In this case, the Overlay Index may be the RT-5's MAC address
+ or "None", depending on the local policy of the receiving NVE/
+ PE. Note that the advertising NVE/PE that sets the Overlay
+ Index SHOULD advertise an RT-2 for the MAC Overlay Index if
+ there are receiving NVE/PEs configured to use the MAC as the
+ Overlay Index. This case in Table 1 is used in the IP-VRF-to-
+ IP-VRF implementations described in Sections 4.4.1 and 4.4.3.
+ The support of a MAC Overlay Index in this model is OPTIONAL.
+
+ *** The Overlay Index is "None". This is a special case used for
+ IP-VRF-to-IP-VRF where the NVE/PEs are connected by IP NVO
+ tunnels as opposed to Ethernet NVO tunnels.
+
+ If the combination of ESI, GW IP, MAC, and Label in the receiving
+ RT-5 is different than the combinations shown in Table 1, the router
+ will process the route as per the rules described at the beginning of
+ this section (Section 3.2).
+
+ Table 2 shows the different inter-subnet use cases described in this
+ document and the corresponding coding of the Overlay Index in the
+ route type 5 (RT-5).
+
+ +=========+=====================+===========================+
+ | Section | Use Case | Overlay Index in the RT-5 |
+ +=========+=====================+===========================+
+ | 4.1 | TS IP address | GW IP |
+ +---------+---------------------+---------------------------+
+ | 4.2 | Floating IP address | GW IP |
+ +---------+---------------------+---------------------------+
+ | 4.3 | "Bump-in-the-wire" | ESI or MAC |
+ +---------+---------------------+---------------------------+
+ | 4.4 | IP-VRF-to-IP-VRF | GW IP, MAC, or None |
+ +---------+---------------------+---------------------------+
+
+ Table 2: Use Cases and Overlay Indexes for Recursive
+ Resolution
+
+ The above use cases are representative of the different Overlay
+ Indexes supported by the RT-5 (GW IP, ESI, MAC, or None).
+
+4. Overlay Index Use Cases
+
+ This section describes some use cases for the Overlay Index types
+ used with the IP Prefix route. Although the examples use IPv4
+ prefixes and subnets, the descriptions of the RT-5 are valid for the
+ same cases with IPv6, except that IP Prefixes, IPL, and GW IP are
+ replaced by the corresponding IPv6 values.
+
+4.1. TS IP Address Overlay Index Use Case
+
+ Figure 5 illustrates an example of inter-subnet forwarding for
+ subnets sitting behind VAs (on TS2 and TS3).
+
+ IP4---+ NVE2 DGW1
+ | +-----------+ +---------+ +-------------+
+ SN2---TS2(VA)--| (BD-10) |-| |----| (BD-10) |
+ | M2/IP2 +-----------+ | | | IRB1\ |
+ -+---+ | | | (IP-VRF)|---+
+ | | | +-------------+ _|_
+ SN1 | VXLAN/ | ( )
+ | | GENEVE | DGW2 ( WAN )
+ -+---+ NVE3 | | +-------------+ (___)
+ | M3/IP3 +-----------+ | |----| (BD-10) | |
+ SN3---TS3(VA)--| (BD-10) |-| | | IRB2\ | |
+ | +-----------+ +---------+ | (IP-VRF)|---+
+ IP5---+ +-------------+
+
+ Figure 5: TS IP Address Use Case
+
+ An example of inter-subnet forwarding between subnet SN1, which uses
+ a 24-bit IP prefix (written as SN1/24 in the future), and a subnet
+ sitting in the WAN is described below. NVE2, NVE3, DGW1, and DGW2
+ are running BGP EVPN. TS2 and TS3 do not participate in dynamic
+ routing protocols, and they only have a static route to forward the
+ traffic to the WAN. SN1/24 is dual-homed to NVE2 and NVE3.
+
+ In this case, a GW IP is used as an Overlay Index. Although a
+ different Overlay Index type could have been used, this use case
+ assumes that the operator knows the VA's IP addresses beforehand,
+ whereas the VA's MAC address is unknown and the VA's ESI is zero.
+ Because of this, the GW IP is the suitable Overlay Index to be used
+ with the RT-5s. The NVEs know the GW IP to be used for a given
+ prefix by policy.
+
+ (1) NVE2 advertises the following BGP routes on behalf of TS2:
+
+ * Route type 2 (MAC/IP Advertisement route) containing: ML = 48
+ (MAC address length), M = M2 (MAC address), IPL = 32 (IP
+ prefix length), IP = IP2, and BGP Encapsulation Extended
+ Community [RFC9012] with the corresponding tunnel type. The
+ MAC and IP addresses may be learned via ARP snooping.
+
+ * Route type 5 (IP Prefix route) containing: IPL = 24, IP =
+ SN1, ESI = 0, and GW IP address = IP2. The prefix and GW IP
+ are learned by policy.
+
+ (2) Similarly, NVE3 advertises the following BGP routes on behalf of
+ TS3:
+
+ * Route type 2 (MAC/IP Advertisement route) containing: ML =
+ 48, M = M3, IPL = 32, IP = IP3 (and BGP Encapsulation
+ Extended Community).
+
+ * Route type 5 (IP Prefix route) containing: IPL = 24, IP =
+ SN1, ESI = 0, and GW IP address = IP3.
+
+ (3) DGW1 and DGW2 import both received routes based on the Route
+ Targets:
+
+ * Based on the BD-10 Route Target in DGW1 and DGW2, the MAC/IP
+ Advertisement route is imported, and M2 is added to the BD-10
+ along with its corresponding tunnel information. For
+ instance, if VXLAN is used, the VTEP will be derived from the
+ MAC/IP Advertisement route BGP next hop and VNI from the MPLS
+ Label1 field. M2/IP2 is added to the ARP table. Similarly,
+ M3 is added to BD-10, and M3/IP3 is added to the ARP table.
+
+ * Based on the BD-10 Route Target in DGW1 and DGW2, the IP
+ Prefix route is also imported, and SN1/24 is added to the IP-
+ VRF with Overlay Index IP2 pointing at the local BD-10. In
+ this example, it is assumed that the RT-5 from NVE2 is
+ preferred over the RT-5 from NVE3. If both routes were
+ equally preferable and ECMP enabled, SN1/24 would also be
+ added to the routing table with Overlay Index IP3.
+
+ (4) When DGW1 receives a packet from the WAN with destination IPx,
+ where IPx belongs to SN1/24:
+
+ * A destination IP lookup is performed on the DGW1 IP-VRF
+ table, and Overlay Index = IP2 is found. Since IP2 is an
+ Overlay Index, a recursive route resolution is required for
+ IP2.
+
+ * IP2 is resolved to M2 in the ARP table, and M2 is resolved to
+ the tunnel information given by the BD FIB (e.g., remote VTEP
+ and VNI for the VXLAN case).
+
+ * The IP packet destined to IPx is encapsulated with:
+
+ - Inner source MAC = IRB1 MAC.
+
+ - Inner destination MAC = M2.
+
+ - Tunnel information provided by the BD (VNI, VTEP IPs, and
+ MACs for the VXLAN case).
+
+ (5) When the packet arrives at NVE2:
+
+ * Based on the tunnel information (VNI for the VXLAN case), the
+ BD-10 context is identified for a MAC lookup.
+
+ * Encapsulation is stripped off and, based on a MAC lookup
+ (assuming MAC forwarding on the egress NVE), the packet is
+ forwarded to TS2, where it will be properly routed.
+
+ (6) Should TS2 move from NVE2 to NVE3, MAC Mobility procedures will
+ be applied to the MAC route M2/IP2, as defined in [RFC7432].
+ Route type 5 prefixes are not subject to MAC Mobility
+ procedures; hence, no changes in the DGW IP-VRF table will occur
+ for TS2 mobility -- i.e., all the prefixes will still be
+ pointing at IP2 as the Overlay Index. There is an indirection
+ for, e.g., SN1/24, which still points at Overlay Index IP2 in
+ the routing table, but IP2 will be simply resolved to a
+ different tunnel based on the outcome of the MAC Mobility
+ procedures for the MAC/IP Advertisement route M2/IP2.
+
+ Note that in the opposite direction, TS2 will send traffic based on
+ its static-route next-hop information (IRB1 and/or IRB2), and regular
+ EVPN procedures will be applied.
+
+4.2. Floating IP Overlay Index Use Case
+
+ Sometimes TSs work in active/standby mode where an upstream floating
+ IP owned by the active TS is used as the Overlay Index to get to some
+ subnets behind the TS. This redundancy mode, already introduced in
+ Sections 2.1 and 2.2, is illustrated in Figure 6.
+
+ NVE2 DGW1
+ +-----------+ +---------+ +-------------+
+ +---TS2(VA)--| (BD-10) |-| |----| (BD-10) |
+ | M2/IP2 +-----------+ | | | IRB1\ |
+ | <-+ | | | (IP-VRF)|---+
+ | | | | +-------------+ _|_
+ SN1 vIP23 (floating) | VXLAN/ | ( )
+ | | | GENEVE | DGW2 ( WAN )
+ | <-+ NVE3 | | +-------------+ (___)
+ | M3/IP3 +-----------+ | |----| (BD-10) | |
+ +---TS3(VA)--| (BD-10) |-| | | IRB2\ | |
+ +-----------+ +---------+ | (IP-VRF)|---+
+ +-------------+
+
+ Figure 6: Floating IP Overlay Index for Redundant TS
+
+ In this use case, a GW IP is used as an Overlay Index for the same
+ reasons as in Section 4.1. However, this GW IP is a floating IP that
+ belongs to the active TS. Assuming TS2 is the active TS and owns
+ vIP23:
+
+ (1) NVE2 advertises the following BGP routes for TS2:
+
+ * Route type 2 (MAC/IP Advertisement route) containing: ML =
+ 48, M = M2, IPL = 32, and IP = vIP23 (as well as BGP
+ Encapsulation Extended Community). The MAC and IP addresses
+ may be learned via ARP snooping.
+
+ * Route type 5 (IP Prefix route) containing: IPL = 24, IP =
+ SN1, ESI = 0, and GW IP address = vIP23. The prefix and GW
+ IP are learned by policy.
+
+ (2) NVE3 advertises the following BGP route for TS3 (it does not
+ advertise an RT-2 for M3/vIP23):
+
+ * Route type 5 (IP Prefix route) containing: IPL = 24, IP =
+ SN1, ESI = 0, and GW IP address = vIP23. The prefix and GW
+ IP are learned by policy.
+
+ (3) DGW1 and DGW2 import both received routes based on the Route
+ Target:
+
+ * M2 is added to the BD-10 FIB along with its corresponding
+ tunnel information. For the VXLAN use case, the VTEP will be
+ derived from the MAC/IP Advertisement route BGP next hop and
+ VNI from the VNI field. M2/vIP23 is added to the ARP table.
+
+ * SN1/24 is added to the IP-VRF in DGW1 and DGW2 with Overlay
+ Index vIP23 pointing at M2 in the local BD-10.
+
+ (4) When DGW1 receives a packet from the WAN with destination IPx,
+ where IPx belongs to SN1/24:
+
+ * A destination IP lookup is performed on the DGW1 IP-VRF
+ table, and Overlay Index = vIP23 is found. Since vIP23 is an
+ Overlay Index, a recursive route resolution for vIP23 is
+ required.
+
+ * vIP23 is resolved to M2 in the ARP table, and M2 is resolved
+ to the tunnel information given by the BD (remote VTEP and
+ VNI for the VXLAN case).
+
+ * The IP packet destined to IPx is encapsulated with:
+
+ - Inner source MAC = IRB1 MAC.
+
+ - Inner destination MAC = M2.
+
+ - Tunnel information provided by the BD FIB (VNI, VTEP IPs,
+ and MACs for the VXLAN case).
+
+ (5) When the packet arrives at NVE2:
+
+ * Based on the tunnel information (VNI for the VXLAN case), the
+ BD-10 context is identified for a MAC lookup.
+
+ * Encapsulation is stripped off and, based on a MAC lookup
+ (assuming MAC forwarding on the egress NVE), the packet is
+ forwarded to TS2, where it will be properly routed.
+
+ (6) When the redundancy protocol running between TS2 and TS3
+ appoints TS3 as the new active TS for SN1, TS3 will now own the
+ floating vIP23 and will signal this new ownership using a
+ gratuitous ARP REPLY message (explained in [RFC5227]) or
+ similar. Upon receiving the new owner's notification, NVE3 will
+ issue a route type 2 for M3/vIP23, and NVE2 will withdraw the
+ RT-2 for M2/vIP23. DGW1 and DGW2 will update their ARP tables
+ with the new MAC resolving the floating IP. No changes are made
+ in the IP-VRF table.
+
+4.3. Bump-in-the-Wire Use Case
+
+ Figure 7 illustrates an example of inter-subnet forwarding for an IP
+ Prefix route that carries subnet SN1. In this use case, TS2 and TS3
+ are Layer 2 VA devices without any IP addresses that can be included
+ as an Overlay Index in the GW IP field of the IP Prefix route. Their
+ MAC addresses are M2 and M3, respectively, and are connected to BD-
+ 10. Note that IRB1 and IRB2 (in DGW1 and DGW2, respectively) have IP
+ addresses in a subnet different than SN1.
+
+ NVE2 DGW1
+ M2 +-----------+ +---------+ +-------------+
+ +---TS2(VA)--| (BD-10) |-| |----| (BD-10) |
+ | ESI23 +-----------+ | | | IRB1\ |
+ | + | | | (IP-VRF)|---+
+ | | | | +-------------+ _|_
+ SN1 | | VXLAN/ | ( )
+ | | | GENEVE | DGW2 ( WAN )
+ | + NVE3 | | +-------------+ (___)
+ | ESI23 +-----------+ | |----| (BD-10) | |
+ +---TS3(VA)--| (BD-10) |-| | | IRB2\ | |
+ M3 +-----------+ +---------+ | (IP-VRF)|---+
+ +-------------+
+
+ Figure 7: Bump-in-the-Wire Use Case
+
+ Since TS2 and TS3 cannot participate in any dynamic routing protocol
+ and neither has an IP address assigned, there are two potential
+ Overlay Index types that can be used when advertising SN1:
+
+ a) an ESI, i.e., ESI23, that can be provisioned on the attachment
+ ports of NVE2 and NVE3, as shown in Figure 7 or
+
+ b) the VA's MAC address, which can be added to NVE2 and NVE3 by
+ policy.
+
+ The advantage of using an ESI as the Overlay Index as opposed to the
+ VA's MAC address is that the forwarding to the egress NVE can be done
+ purely based on the state of the AC in the Ethernet segment (notified
+ by the Ethernet A-D per EVI route), and all the EVPN multihoming
+ redundancy mechanisms can be reused. For instance, the mass
+ withdrawal mechanism described in [RFC7432] for fast failure
+ detection and propagation can be used. It is assumed per this
+ section that an ESI Overlay Index is used in this use case, but this
+ use case does not preclude the use of the VA's MAC address as an
+ Overlay Index. If a MAC is used as the Overlay Index, the control
+ plane must follow the procedures described in Section 4.4.3.
+
+ The model supports VA redundancy in a similar way to the one
+ described in Section 4.2 for the floating IP Overlay Index use case,
+ except that it uses the EVPN Ethernet A-D per EVI route instead of
+ the MAC advertisement route to advertise the location of the Overlay
+ Index. The procedure is explained below:
+
+ (1) Assuming TS2 is the active TS in ESI23, NVE2 advertises the
+ following BGP routes:
+
+ * Route type 1 (Ethernet A-D route for BD-10) containing: ESI =
+ ESI23 and the corresponding tunnel information (VNI field),
+ as well as the BGP Encapsulation Extended Community as per
+ [RFC8365].
+
+ * Route type 5 (IP Prefix route) containing: IPL = 24, IP =
+ SN1, ESI = ESI23, and GW IP address = 0. The EVPN Router's
+ MAC Extended Community defined in [RFC9135] is added and
+ carries the MAC address (M2) associated with the TS behind
+ which SN1 sits. M2 may be learned by policy; however, the
+ MAC in the Extended Community is preferred if sent with the
+ route.
+
+ (2) NVE3 advertises the following BGP route for TS3 (no AD per EVI
+ route is advertised):
+
+ * Route type 5 (IP Prefix route) containing: IPL = 24, IP =
+ SN1, ESI = 23, and GW IP address = 0. The EVPN Router's MAC
+ Extended Community is added and carries the MAC address (M3)
+ associated with the TS behind which SN1 sits. M3 may be
+ learned by policy; however, the MAC in the Extended Community
+ is preferred if sent with the route.
+
+ (3) DGW1 and DGW2 import the received routes based on the Route
+ Target:
+
+ * The tunnel information to get to ESI23 is installed in DGW1
+ and DGW2. For the VXLAN use case, the VTEP will be derived
+ from the Ethernet A-D route BGP next hop and VNI from the
+ VNI/VSID field (see [RFC8365]).
+
+ * The RT-5 coming from the NVE that advertised the RT-1 is
+ selected, and SN1/24 is added to the IP-VRF in DGW1 and DGW2
+ with Overlay Index ESI23 and MAC = M2.
+
+ (4) When DGW1 receives a packet from the WAN with destination IPx,
+ where IPx belongs to SN1/24:
+
+ * A destination IP lookup is performed on the DGW1 IP-VRF
+ table, and Overlay Index = ESI23 is found. Since ESI23 is an
+ Overlay Index, a recursive route resolution is required to
+ find the egress NVE where ESI23 resides.
+
+ * The IP packet destined to IPx is encapsulated with:
+
+ - Inner source MAC = IRB1 MAC.
+
+ - Inner destination MAC = M2 (this MAC will be obtained from
+ the EVPN Router's MAC Extended Community received along
+ with the RT-5 for SN1). Note that the EVPN Router's MAC
+ Extended Community is used in this case to carry the TS's
+ MAC address, as opposed to the MAC address of the NVE/PE.
+
+ - Tunnel information for the NVO tunnel is provided by the
+ Ethernet A-D route per EVI for ESI23 (VNI and VTEP IP for
+ the VXLAN case).
+
+ (5) When the packet arrives at NVE2:
+
+ * Based on the tunnel demultiplexer information (VNI for the
+ VXLAN case), the BD-10 context is identified for a MAC lookup
+ (assuming a MAC-based disposition model [RFC7432]), or the
+ VNI may directly identify the egress interface (for an MPLS-
+ based disposition model, which in this context is a VNI-based
+ disposition model).
+
+ * Encapsulation is stripped off and, based on a MAC lookup
+ (assuming MAC forwarding on the egress NVE) or a VNI lookup
+ (in case of VNI forwarding), the packet is forwarded to TS2,
+ where it will be forwarded to SN1.
+
+ (6) If the redundancy protocol running between TS2 and TS3 follows
+ an active/standby model and there is a failure, TS3 is appointed
+ as the new active TS for SN1. TS3 will now own the connectivity
+ to SN1 and will signal this new ownership. Upon receiving the
+ new owner's notification, NVE3's AC will become active and issue
+ a route type 1 for ESI23, whereas NVE2 will withdraw its
+ Ethernet A-D route for ESI23. DGW1 and DGW2 will update their
+ tunnel information to resolve ESI23. The inner destination MAC
+ will be changed to M3.
+
+4.4. IP-VRF-to-IP-VRF Model
+
+ This use case is similar to the scenario described in Section 9.1 of
+ [RFC9135]; however, the new requirement here is the advertisement of
+ IP prefixes as opposed to only host routes.
+
+ In the examples described in Sections 4.1, 4.2, and 4.3, the BD
+ instance can connect IRB interfaces and any other Tenant Systems
+ connected to it. EVPN provides connectivity for:
+
+ 1. Traffic destined to the IRB or TS IP interfaces, as well as
+
+ 2. Traffic destined to IP subnets sitting behind the TS, e.g., SN1
+ or SN2.
+
+ In order to provide connectivity for (1), MAC/IP Advertisement routes
+ (RT-2) are needed so that IRB or TS MACs and IPs can be distributed.
+ Connectivity type (2) is accomplished by the exchange of IP Prefix
+ routes (RT-5) for IPs and subnets sitting behind certain Overlay
+ Indexes, e.g., GW IP, ESI, or TS MAC.
+
+ In some cases, IP Prefix routes may be advertised for subnets and IPs
+ sitting behind an IRB. This use case is referred to as the "IP-VRF-
+ to-IP-VRF" model.
+
+ [RFC9135] defines an asymmetric IRB model and a symmetric IRB model
+ based on the required lookups at the ingress and egress NVE. The
+ asymmetric model requires an IP lookup and a MAC lookup at the
+ ingress NVE, whereas only a MAC lookup is needed at the egress NVE;
+ the symmetric model requires IP and MAC lookups at both the ingress
+ and egress NVE. From that perspective, the IP-VRF-to-IP-VRF use case
+ described in this section is a symmetric IRB model.
+
+ Note that in an IP-VRF-to-IP-VRF scenario, out of the many subnets
+ that a tenant may have, it may be the case that only a few are
+ attached to a given IP-VRF of the NVE/PE. In order to provide inter-
+ subnet connectivity among the set of NVE/PEs where the tenant is
+ connected, a new SBD is created on all of them if a recursive
+ resolution is needed. This SBD is instantiated as a regular BD (with
+ no ACs) in each NVE/PE and has an IRB interface that connects the SBD
+ to the IP-VRF. The IRB interface's IP or MAC address is used as the
+ Overlay Index for a recursive resolution.
+
+ Depending on the existence and characteristics of the SBD and IRB
+ interfaces for the IP-VRFs, there are three different IP-VRF-to-IP-
+ VRF scenarios identified and described in this document:
+
+ 1. Interface-less model: no SBD and no Overlay Indexes required.
+
+ 2. Interface-ful with an SBD IRB model: requires SBD as well as GW
+ IP addresses as Overlay Indexes.
+
+ 3. Interface-ful with an unnumbered SBD IRB model: requires SBD as
+ well as MAC addresses as Overlay Indexes.
+
+ Inter-subnet IP multicast is outside the scope of this document.
+
+4.4.1. Interface-less IP-VRF-to-IP-VRF Model
+
+ Figure 8 depicts the Interface-less IP-VRF-to-IP-VRF model.
+
+ NVE1(M1)
+ +------------+
+ IP1+----| (BD-1) | DGW1(M3)
+ | \ | +---------+ +--------+
+ | (IP-VRF)|----| |-|(IP-VRF)|----+
+ | / | | | +--------+ |
+ +---| (BD-2) | | | _+_
+ | +------------+ | | ( )
+ SN1| | VXLAN/ | ( WAN )--H1
+ | NVE2(M2) | GENEVE/| (___)
+ | +------------+ | MPLS | +
+ +---| (BD-2) | | | DGW2(M4) |
+ | \ | | | +--------+ |
+ | (IP-VRF)|----| |-|(IP-VRF)|----+
+ | / | +---------+ +--------+
+ SN2+----| (BD-3) |
+ +------------+
+
+ Figure 8: Interface-less IP-VRF-to-IP-VRF Model
+
+ In this case:
+
+ a) The NVEs and DGWs must provide connectivity between hosts in SN1,
+ SN2, and IP1 and hosts sitting at the other end of the WAN -- for
+ example, H1. It is assumed that the DGWs import/export IP and/or
+ VPN-IP routes to/from the WAN.
+
+ b) The IP-VRF instances in the NVE/DGWs are directly connected
+ through NVO tunnels, and no IRBs and/or BD instances are
+ instantiated to connect the IP-VRFs.
+
+ c) The solution must provide Layer 3 connectivity among the IP-VRFs
+ for Ethernet NVO tunnels -- for instance, VXLAN or GENEVE.
+
+ d) The solution may provide Layer 3 connectivity among the IP-VRFs
+ for IP NVO tunnels -- for example, GENEVE (with IP payload).
+
+ In order to meet the above requirements, the EVPN route type 5 will
+ be used to advertise the IP prefixes, along with the EVPN Router's
+ MAC Extended Community as defined in [RFC9135] if the advertising
+ NVE/DGW uses Ethernet NVO tunnels. Each NVE/DGW will advertise an
+ RT-5 for each of its prefixes with the following fields:
+
+ * RD as per [RFC7432].
+
+ * Ethernet Tag ID = 0.
+
+ * IP prefix length and IP address, as explained in the previous
+ sections.
+
+ * GW IP address = 0.
+
+ * ESI = 0.
+
+ * MPLS label or VNI corresponding to the IP-VRF.
+
+ Each RT-5 will be sent with a Route Target identifying the tenant
+ (IP-VRF) and may be sent with two BGP extended communities:
+
+ * The first one is the BGP Encapsulation Extended Community, as per
+ [RFC9012], identifying the tunnel type.
+
+ * The second one is the EVPN Router's MAC Extended Community, as per
+ [RFC9135], containing the MAC address associated with the NVE
+ advertising the route. This MAC address identifies the NVE/DGW
+ and MAY be reused for all the IP-VRFs in the NVE. The EVPN
+ Router's MAC Extended Community must be sent if the route is
+ associated with an Ethernet NVO tunnel -- for instance, VXLAN. If
+ the route is associated with an IP NVO tunnel -- for instance,
+ GENEVE with an IP payload -- the EVPN Router's MAC Extended
+ Community should not be sent.
+
+ The following example illustrates the procedure to advertise and
+ forward packets to SN1/24 (IPv4 prefix advertised from NVE1):
+
+ (1) NVE1 advertises the following BGP route:
+
+ * Route type 5 (IP Prefix route) containing:
+
+ - IPL = 24, IP = SN1, Label = 10.
+
+ - GW IP = set to 0.
+
+ - BGP Encapsulation Extended Community [RFC9012].
+
+ - EVPN Router's MAC Extended Community that contains M1.
+
+ - Route Target identifying the tenant (IP-VRF).
+
+ (2) DGW1 imports the received routes from NVE1:
+
+ * DGW1 installs SN1/24 in the IP-VRF identified by the RT-5
+ Route Target.
+
+ * Since GW IP = ESI = 0, the label is a non-zero value, and the
+ local policy indicates this interface-less model, DGW1, will
+ use the label and next hop of the RT-5, as well as the MAC
+ address conveyed in the EVPN Router's MAC Extended Community
+ (as the inner destination MAC address) to set up the
+ forwarding state and later encapsulate the routed IP packets.
+
+ (3) When DGW1 receives a packet from the WAN with destination IPx,
+ where IPx belongs to SN1/24:
+
+ * A destination IP lookup is performed on the DGW1 IP-VRF
+ table. The lookup yields SN1/24.
+
+ * Since the RT-5 for SN1/24 had a GW IP = ESI = 0, a non-zero
+ label, and a next hop, and since the model is interface-less,
+ DGW1 will not need a recursive lookup to resolve the route.
+
+ * The IP packet destined to IPx is encapsulated with: inner
+ source MAC = DGW1 MAC, inner destination MAC = M1, outer
+ source IP (tunnel source IP) = DGW1 IP, and outer destination
+ IP (tunnel destination IP) = NVE1 IP. The source and inner
+ destination MAC addresses are not needed if IP NVO tunnels
+ are used.
+
+ (4) When the packet arrives at NVE1:
+
+ * NVE1 will identify the IP-VRF for an IP lookup based on the
+ label (the inner destination MAC is not needed to identify
+ the IP-VRF).
+
+ * An IP lookup is performed in the routing context, where SN1
+ turns out to be a local subnet associated with BD-2. A
+ subsequent lookup in the ARP table and the BD FIB will
+ provide the forwarding information for the packet in BD-2.
+
+ The model described above is called an "interface-less" model since
+ the IP-VRFs are connected directly through tunnels, and they don't
+ require those tunnels to be terminated in SBDs instead, as in
+ Sections 4.4.2 or 4.4.3.
+
+4.4.2. Interface-ful IP-VRF-to-IP-VRF with SBD IRB
+
+ Figure 9 depicts the Interface-ful IP-VRF-to-IP-VRF with SBD IRB
+ model.
+
+ NVE1
+ +------------+ DGW1
+ IP10+---+(BD-1) | +---------------+ +------------+
+ | \ | | | | |
+ |(IP-VRF)-(SBD)| |(SBD)-(IP-VRF)|-----+
+ | / IRB(M1/IP1) IRB(M3/IP3) | |
+ +---+(BD-2) | | | +------------+ _+_
+ | +------------+ | | ( )
+ SN1| | VXLAN/ | ( WAN )--H1
+ | NVE2 | GENEVE/ | (___)
+ | +------------+ | MPLS | DGW2 +
+ +---+(BD-2) | | | +------------+ |
+ | \ | | | | | |
+ |(IP-VRF)-(SBD)| |(SBD)-(IP-VRF)|-----+
+ | / IRB(M2/IP2) IRB(M4/IP4) |
+ SN2+----+(BD-3) | +---------------+ +------------+
+ +------------+
+
+ Figure 9: Interface-ful with SBD IRB Model
+
+ In this model:
+
+ a) As in Section 4.4.1, the NVEs and DGWs must provide connectivity
+ between hosts in SN1, SN2, and IP10 and in hosts sitting at the
+ other end of the WAN.
+
+ b) However, the NVE/DGWs are now connected through Ethernet NVO
+ tunnels terminated in the SBD instance. The IP-VRFs use IRB
+ interfaces for their connectivity to the SBD.
+
+ c) Each SBD IRB has an IP and a MAC address, where the IP address
+ must be reachable from other NVEs or DGWs.
+
+ d) The SBD is attached to all the NVE/DGWs in the tenant domain BDs.
+
+ e) The solution must provide Layer 3 connectivity for Ethernet NVO
+ tunnels -- for instance, VXLAN or GENEVE (with Ethernet payload).
+
+ EVPN type 5 routes will be used to advertise the IP prefixes, whereas
+ EVPN RT-2 routes will advertise the MAC/IP addresses of each SBD IRB
+ interface. Each NVE/DGW will advertise an RT-5 for each of its
+ prefixes with the following fields:
+
+ * RD as per [RFC7432].
+
+ * Ethernet Tag ID = 0.
+
+ * IP prefix length and IP address, as explained in the previous
+ sections.
+
+ * GW IP address = IRB-IP of the SBD (this is the Overlay Index that
+ will be used for the recursive route resolution).
+
+ * ESI = 0.
+
+ * Label value should be zero since the RT-5 route requires a
+ recursive lookup resolution to an RT-2 route. It is ignored on
+ reception, and the MPLS label or VNI from the RT-2's MPLS Label1
+ field is used when forwarding packets.
+
+ Each RT-5 will be sent with a Route Target identifying the tenant
+ (IP-VRF). The EVPN Router's MAC Extended Community should not be
+ sent in this case.
+
+ The following example illustrates the procedure to advertise and
+ forward packets to SN1/24 (IPv4 prefix advertised from NVE1):
+
+ (1) NVE1 advertises the following BGP routes:
+
+ * Route type 5 (IP Prefix route) containing:
+
+ - IPL = 24, IP = SN1, Label = SHOULD be set to 0.
+
+ - GW IP = IP1 (SBD IRB's IP).
+
+ - Route Target identifying the tenant (IP-VRF).
+
+ * Route type 2 (MAC/IP Advertisement route for the SBD IRB)
+ containing:
+
+ - ML = 48, M = M1, IPL = 32, IP = IP1, Label = 10.
+
+ - A BGP Encapsulation Extended Community [RFC9012].
+
+ - Route Target identifying the SBD. This Route Target may
+ be the same as the one used with the RT-5.
+
+ (2) DGW1 imports the received routes from NVE1:
+
+ * DGW1 installs SN1/24 in the IP-VRF identified by the RT-5
+ Route Target.
+
+ - Since GW IP is different from zero, the GW IP (IP1) will
+ be used as the Overlay Index for the recursive route
+ resolution to the RT-2 carrying IP1.
+
+ (3) When DGW1 receives a packet from the WAN with destination IPx,
+ where IPx belongs to SN1/24:
+
+ * A destination IP lookup is performed on the DGW1 IP-VRF
+ table. The lookup yields SN1/24, which is associated with
+ the Overlay Index IP1. The forwarding information is derived
+ from the RT-2 received for IP1.
+
+ * The IP packet destined to IPx is encapsulated with: inner
+ source MAC = M3, inner destination MAC = M1, outer source IP
+ (source VTEP) = DGW1 IP, and outer destination IP
+ (destination VTEP) = NVE1 IP.
+
+ (4) When the packet arrives at NVE1:
+
+ * NVE1 will identify the IP-VRF for an IP lookup based on the
+ label and the inner MAC DA.
+
+ * An IP lookup is performed in the routing context, where SN1
+ turns out to be a local subnet associated with BD-2. A
+ subsequent lookup in the ARP table and the BD FIB will
+ provide the forwarding information for the packet in BD-2.
+
+ The model described above is called an "interface-ful with SBD IRB"
+ model because the tunnels connecting the DGWs and NVEs need to be
+ terminated into the SBD. The SBD is connected to the IP-VRFs via SBD
+ IRB interfaces, and that allows the recursive resolution of RT-5s to
+ GW IP addresses.
+
+4.4.3. Interface-ful IP-VRF-to-IP-VRF with Unnumbered SBD IRB
+
+ Figure 10 depicts the Interface-ful IP-VRF-to-IP-VRF with unnumbered
+ SBD IRB model. Note that this model is similar to the one described
+ in Section 4.4.2, only without IP addresses on the SBD IRB
+ interfaces.
+
+ NVE1
+ +------------+ DGW1
+ IP1+----+(BD-1) | +---------------+ +------------+
+ | \ | | | | |
+ |(IP-VRF)-(SBD)| (SBD)-(IP-VRF) |-----+
+ | / IRB(M1)| | IRB(M3) | |
+ +---+(BD-2) | | | +------------+ _+_
+ | +------------+ | | ( )
+ SN1| | VXLAN/ | ( WAN )--H1
+ | NVE2 | GENEVE/ | (___)
+ | +------------+ | MPLS | DGW2 +
+ +---+(BD-2) | | | +------------+ |
+ | \ | | | | | |
+ |(IP-VRF)-(SBD)| (SBD)-(IP-VRF) |-----+
+ | / IRB(M2)| | IRB(M4) |
+ SN2+----+(BD-3) | +---------------+ +------------+
+ +------------+
+
+ Figure 10: Interface-ful with Unnumbered SBD IRB Model
+
+ In this model:
+
+ a) As in Sections 4.4.1 and 4.4.2, the NVEs and DGWs must provide
+ connectivity between hosts in SN1, SN2, and IP1 and in hosts
+ sitting at the other end of the WAN.
+
+ b) As in Section 4.4.2, the NVE/DGWs are connected through Ethernet
+ NVO tunnels terminated in the SBD instance. The IP-VRFs use IRB
+ interfaces for their connectivity to the SBD.
+
+ c) However, each SBD IRB has a MAC address only and no IP address
+ (which is why the model refers to an "unnumbered" SBD IRB). In
+ this model, there is no need to have IP reachability to the SBD
+ IRB interfaces themselves, and there is a requirement to limit
+ the number of IP addresses used.
+
+ d) As in Section 4.4.2, the SBD is composed of all the NVE/DGW BDs
+ of the tenant that need inter-subnet forwarding.
+
+ e) As in Section 4.4.2, the solution must provide Layer 3
+ connectivity for Ethernet NVO tunnels -- for instance, VXLAN or
+ GENEVE (with Ethernet payload).
+
+ This model will also make use of the RT-5 recursive resolution. EVPN
+ type 5 routes will advertise the IP prefixes along with the EVPN
+ Router's MAC Extended Community used for the recursive lookup,
+ whereas EVPN RT-2 routes will advertise the MAC addresses of each SBD
+ IRB interface (this time without an IP).
+
+ Each NVE/DGW will advertise an RT-5 for each of its prefixes with the
+ same fields as described in Section 4.4.2, except:
+
+ * GW IP address = set to 0.
+
+ Each RT-5 will be sent with a Route Target identifying the tenant
+ (IP-VRF) and the EVPN Router's MAC Extended Community containing the
+ MAC address associated with the SBD IRB interface. This MAC address
+ may be reused for all the IP-VRFs in the NVE.
+
+ The example is similar to the one in Section 4.4.2:
+
+ (1) NVE1 advertises the following BGP routes:
+
+ * Route type 5 (IP Prefix route) containing the same values as
+ in the example in Section 4.4.2, except:
+
+ - GW IP = SHOULD be set to 0.
+
+ - EVPN Router's MAC Extended Community containing M1 (this
+ will be used for the recursive lookup to an RT-2).
+
+ * Route type 2 (MAC route for the SBD IRB) with the same values
+ as in Section 4.4.2, except:
+
+ - ML = 48, M = M1, IPL = 0, Label = 10.
+
+ (2) DGW1 imports the received routes from NVE1:
+
+ * DGW1 installs SN1/24 in the IP-VRF identified by the RT-5
+ Route Target.
+
+ - The MAC contained in the EVPN Router's MAC Extended
+ Community sent along with the RT-5 (M1) will be used as
+ the Overlay Index for the recursive route resolution to
+ the RT-2 carrying M1.
+
+ (3) When DGW1 receives a packet from the WAN with destination IPx,
+ where IPx belongs to SN1/24:
+
+ * A destination IP lookup is performed on the DGW1 IP-VRF
+ table. The lookup yields SN1/24, which is associated with
+ the Overlay Index M1. The forwarding information is derived
+ from the RT-2 received for M1.
+
+ * The IP packet destined to IPx is encapsulated with: inner
+ source MAC = M3, inner destination MAC = M1, outer source IP
+ (source VTEP) = DGW1 IP, and outer destination IP
+ (destination VTEP) = NVE1 IP.
+
+ (4) When the packet arrives at NVE1:
+
+ * NVE1 will identify the IP-VRF for an IP lookup based on the
+ label and the inner MAC DA.
+
+ * An IP lookup is performed in the routing context, where SN1
+ turns out to be a local subnet associated with BD-2. A
+ subsequent lookup in the ARP table and the BD FIB will
+ provide the forwarding information for the packet in BD-2.
+
+ The model described above is called an "interface-ful with unnumbered
+ SBD IRB" model (as in Section 4.4.2) but without the SBD IRB having
+ an IP address.
+
+5. Security Considerations
+
+ This document provides a set of procedures to achieve inter-subnet
+ forwarding across NVEs or PEs attached to a group of BDs that belong
+ to the same tenant (or VPN). The security considerations discussed
+ in [RFC7432] apply to the intra-subnet forwarding or communication
+ within each of those BDs. In addition, the security considerations
+ in [RFC4364] should also be understood, since this document and
+ [RFC4364] may be used in similar applications.
+
+ Contrary to [RFC4364], this document does not describe PE/CE route
+ distribution techniques but rather considers the CEs as TSs or VAs
+ that do not run dynamic routing protocols. This can be considered a
+ security advantage, since dynamic routing protocols can be blocked on
+ the NVE/PE ACs, not allowing the tenant to interact with the
+ infrastructure's dynamic routing protocols.
+
+ In this document, the RT-5 may use a regular BGP next hop for its
+ resolution or an Overlay Index that requires a recursive resolution
+ to a different EVPN route (an RT-2 or an RT-1). In the latter case,
+ it is worth noting that any action that ends up filtering or
+ modifying the RT-2 or RT-1 routes used to convey the Overlay Indexes
+ will modify the resolution of the RT-5 and therefore the forwarding
+ of packets to the remote subnet.
+
+6. IANA Considerations
+
+ IANA has registered value 5 in the "EVPN Route Types" registry
+ [EVPNRouteTypes] defined by [RFC7432] as follows:
+
+ +=======+=============+===========+
+ | Value | Description | Reference |
+ +=======+=============+===========+
+ | 5 | IP Prefix | RFC 9136 |
+ +-------+-------------+-----------+
+
+ Table 3
+
+7. References
+
+7.1. Normative References
+
+ [EVPNRouteTypes]
+ IANA, "EVPN Route Types",
+ <https://www.iana.org/assignments/evpn>.
+
+ [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
+ Requirement Levels", BCP 14, RFC 2119,
+ DOI 10.17487/RFC2119, March 1997,
+ <https://www.rfc-editor.org/info/rfc2119>.
+
+ [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
+ Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
+ Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
+ 2015, <https://www.rfc-editor.org/info/rfc7432>.
+
+ [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
+ 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
+ May 2017, <https://www.rfc-editor.org/info/rfc8174>.
+
+ [RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
+ Uttaro, J., and W. Henderickx, "A Network Virtualization
+ Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
+ DOI 10.17487/RFC8365, March 2018,
+ <https://www.rfc-editor.org/info/rfc8365>.
+
+ [RFC9012] Patel, K., Van de Velde, G., Sangli, S., and J. Scudder,
+ "The BGP Tunnel Encapsulation Attribute", RFC 9012,
+ DOI 10.17487/RFC9012, April 2021,
+ <https://www.rfc-editor.org/info/rfc9012>.
+
+ [RFC9135] Sajassi, A., Salam, S., Thoria, S., Drake, J., and J.
+ Rabadan, "Integrated Routing and Bridging in Ethernet VPN
+ (EVPN)", RFC 9135, DOI 10.17487/RFC9135, October 2021,
+ <https://www.rfc-editor.org/info/rfc9135>.
+
+7.2. Informative References
+
+ [IEEE-802.1Q]
+ IEEE, "IEEE Standard for Local and Metropolitan Area
+ Networks -- Bridges and Bridged Networks",
+ DOI 10.1109/IEEESTD.2018.8403927, IEEE Std 802.1Q, July
+ 2018,
+ <https://standards.ieee.org/standard/802_1Q-2018.html>.
+
+ [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
+ Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
+ 2006, <https://www.rfc-editor.org/info/rfc4364>.
+
+ [RFC5227] Cheshire, S., "IPv4 Address Conflict Detection", RFC 5227,
+ DOI 10.17487/RFC5227, July 2008,
+ <https://www.rfc-editor.org/info/rfc5227>.
+
+ [RFC5798] Nadas, S., Ed., "Virtual Router Redundancy Protocol (VRRP)
+ Version 3 for IPv4 and IPv6", RFC 5798,
+ DOI 10.17487/RFC5798, March 2010,
+ <https://www.rfc-editor.org/info/rfc5798>.
+
+ [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
+ L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
+ eXtensible Local Area Network (VXLAN): A Framework for
+ Overlaying Virtualized Layer 2 Networks over Layer 3
+ Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
+ <https://www.rfc-editor.org/info/rfc7348>.
+
+ [RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y.
+ Rekhter, "Framework for Data Center (DC) Network
+ Virtualization", RFC 7365, DOI 10.17487/RFC7365, October
+ 2014, <https://www.rfc-editor.org/info/rfc7365>.
+
+ [RFC7606] Chen, E., Ed., Scudder, J., Ed., Mohapatra, P., and K.
+ Patel, "Revised Error Handling for BGP UPDATE Messages",
+ RFC 7606, DOI 10.17487/RFC7606, August 2015,
+ <https://www.rfc-editor.org/info/rfc7606>.
+
+ [RFC8926] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed.,
+ "Geneve: Generic Network Virtualization Encapsulation",
+ RFC 8926, DOI 10.17487/RFC8926, November 2020,
+ <https://www.rfc-editor.org/info/rfc8926>.
+
+Acknowledgments
+
+ The authors would like to thank Mukul Katiyar, Jeffrey Zhang, and
+ Alex Nichol for their valuable feedback and contributions. Tony
+ Przygienda and Thomas Morin also helped improve this document with
+ their feedback. Special thanks to Eric Rosen for his detailed
+ review, which really helped improve the readability and clarify the
+ concepts. We also thank Alvaro Retana for his thorough review.
+
+Contributors
+
+ In addition to the authors listed on the front page, the following
+ coauthors have also contributed to this document:
+
+ Senthil Sathappan
+ Florin Balus
+ Aldrin Isaac
+ Senad Palislamovic
+ Samir Thoria
+
+Authors' Addresses
+
+ Jorge Rabadan (editor)
+ Nokia
+ 777 E. Middlefield Road
+ Mountain View, CA 94043
+ United States of America
+
+ Email: jorge.rabadan@nokia.com
+
+
+ Wim Henderickx
+ Nokia
+
+ Email: wim.henderickx@nokia.com
+
+
+ John Drake
+ Juniper
+
+ Email: jdrake@juniper.net
+
+
+ Wen Lin
+ Juniper
+
+ Email: wlin@juniper.net
+
+
+ Ali Sajassi
+ Cisco
+
+ Email: sajassi@cisco.com