doc: Add RFC documents

1 files changed, 1307 insertions, 0 deletions

diff --git a/doc/rfc/rfc9469.txt b/doc/rfc/rfc9469.txt
new file mode 100644
index 0000000..5a47d2e
--- /dev/null
+++ b/doc/rfc/rfc9469.txt
@@ -0,0 +1,1307 @@
+
+
+
+
+Internet Engineering Task Force (IETF)                   J. Rabadan, Ed.
+Request for Comments: 9469                                      M. Bocci
+Category: Informational                                            Nokia
+ISSN: 2070-1721                                               S. Boutros
+                                                                   Ciena
+                                                              A. Sajassi
+                                                                   Cisco
+                                                          September 2023
+
+
+  Applicability of Ethernet Virtual Private Network (EVPN) to Network
+              Virtualization over Layer 3 (NVO3) Networks
+
+Abstract
+
+   An Ethernet Virtual Private Network (EVPN) provides a unified control
+   plane that solves the issues of Network Virtualization Edge (NVE)
+   auto-discovery, tenant Media Access Control (MAC) / IP dissemination,
+   and advanced features in a scablable way as required by Network
+   Virtualization over Layer 3 (NVO3) networks.  EVPN is a scalable
+   solution for NVO3 networks and keeps the independence of the underlay
+   IP Fabric, i.e., there is no need to enable Protocol Independent
+   Multicast (PIM) in the underlay network and maintain multicast states
+   for tenant Broadcast Domains.  This document describes the use of
+   EVPN for NVO3 networks and discusses its applicability to basic Layer
+   2 and Layer 3 connectivity requirements and to advanced features such
+   as MAC Mobility, MAC Protection and Loop Protection, multihoming,
+   Data Center Interconnect (DCI), and much more.  No new EVPN
+   procedures are introduced.
+
+Status of This Memo
+
+   This document is not an Internet Standards Track specification; it is
+   published for informational purposes.
+
+   This document is a product of the Internet Engineering Task Force
+   (IETF).  It represents the consensus of the IETF community.  It has
+   received public review and has been approved for publication by the
+   Internet Engineering Steering Group (IESG).  Not all documents
+   approved by the IESG are candidates for any level of Internet
+   Standard; see Section 2 of RFC 7841.
+
+   Information about the current status of this document, any errata,
+   and how to provide feedback on it may be obtained at
+   https://www.rfc-editor.org/info/rfc9469.
+
+Copyright Notice
+
+   Copyright (c) 2023 IETF Trust and the persons identified as the
+   document authors.  All rights reserved.
+
+   This document is subject to BCP 78 and the IETF Trust's Legal
+   Provisions Relating to IETF Documents
+   (https://trustee.ietf.org/license-info) in effect on the date of
+   publication of this document.  Please review these documents
+   carefully, as they describe your rights and restrictions with respect
+   to this document.  Code Components extracted from this document must
+   include Revised BSD License text as described in Section 4.e of the
+   Trust Legal Provisions and are provided without warranty as described
+   in the Revised BSD License.
+
+Table of Contents
+
+   1.  Introduction
+   2.  EVPN and NVO3 Terminology
+   3.  Why is EVPN Needed in NVO3 Networks?
+   4.  Applicability of EVPN to NVO3 Networks
+     4.1.  EVPN Route Types Used in NVO3 Networks
+     4.2.  EVPN Basic Applicability for Layer 2 Services
+       4.2.1.  Auto-Discovery and Auto-Provisioning
+       4.2.2.  Remote NVE Auto-Discovery
+       4.2.3.  Distribution of Tenant MAC and IP Information
+     4.3.  EVPN Basic Applicability for Layer 3 Services
+     4.4.  EVPN as Control Plane for NVO3 Encapsulations and Geneve
+     4.5.  EVPN OAM and Application to NVO3
+     4.6.  EVPN as the Control Plane for NVO3 Security
+     4.7.  Advanced EVPN Features for NVO3 Networks
+       4.7.1.  Virtual Machine (VM) Mobility
+       4.7.2.  MAC Protection, Duplication Detection, and Loop
+               Protection
+       4.7.3.  Reduction/Optimization of BUM Traffic in Layer 2
+               Services
+       4.7.4.  Ingress Replication (IR) Optimization for BUM Traffic
+       4.7.5.  EVPN Multihoming
+       4.7.6.  EVPN Recursive Resolution for Inter-subnet Unicast
+               Forwarding
+       4.7.7.  EVPN Optimized Inter-subnet Multicast Forwarding
+       4.7.8.  Data Center Interconnect (DCI)
+   5.  Security Considerations
+   6.  IANA Considerations
+   7.  References
+     7.1.  Normative References
+     7.2.  Informative References
+   Acknowledgments
+   Authors' Addresses
+
+1.  Introduction
+
+   In Network Virtualization over Layer 3 (NVO3) networks, Network
+   Virtualization Edge (NVE) devices sit at the edge of the underlay
+   network and provide Layer 2 and Layer 3 connectivity among Tenant
+   Systems (TSes) of the same tenant.  The NVEs need to build and
+   maintain mapping tables so they can deliver encapsulated packets to
+   their intended destination NVE(s).  While there are different options
+   to create and disseminate the mapping table entries, NVEs may
+   exchange that information directly among themselves via a control
+   plane protocol, such as Ethernet Virtual Private Network (EVPN).
+   EVPN provides an efficient, flexible, and unified control plane
+   option that can be used for Layer 2 and Layer 3 Virtual Network (VN)
+   service connectivity.  This document does not introduce any new
+   procedures in EVPN.
+
+   In this document, we assume that the EVPN control plane module
+   resides in the NVEs.  The NVEs can be virtual switches in
+   hypervisors, Top-of-Rack (ToR) switches or Leaf switches, or Data
+   Center Gateways.  As described in [RFC7365], Network Virtualization
+   Authorities (NVAs) may be used to provide the forwarding information
+   to the NVEs, and in that case, EVPN could be used to disseminate the
+   information across multiple federated NVAs.  The applicability of
+   EVPN would then be similar to the one described in this document.
+   However, for simplicity, the description assumes control plane
+   communication among NVE(s).
+
+2.  EVPN and NVO3 Terminology
+
+   This document uses the terminology of [RFC7365] in addition to the
+   terms that follow.
+
+   AC:  Attachment Circuit or logical interface associated with a given
+      BT.  To determine the AC on which a packet arrived, the NVE will
+      examine the physical/logical port and/or VLAN tags (where the VLAN
+      tags can be individual c-tags, s-tags, or ranges of both).
+
+   ARP and NDP:  Address Resolution Protocol (IPv4) and Neighbor
+      Discovery Protocol (IPv6), respectively.
+
+   BD:  Broadcast Domain that corresponds to a tenant IP subnet.  If no
+      suppression techniques are used, a BUM frame that is injected in a
+      Broadcast Domain will reach all the NVEs that are attached to that
+      Broadcast Domain.  An EVI may contain one or multiple Broadcast
+      Domains depending on the service model [RFC7432].  This document
+      will use the term Broadcast Domain to refer to a tenant subnet.
+
+   BT:  Bridge Table, as defined in [RFC7432].  A BT is the
+      instantiation of a Broadcast Domain in an NVE.  When there is a
+      single Broadcast Domain on a given EVI, the MAC-VRF is equivalent
+      to the BT on that NVE.  Although a Broadcast Domain spans multiple
+      NVEs and a BT is really the instantiation of a Broadcast Domain in
+      an NVE, this document uses BT and Broadcast Domain
+      interchangeably.
+
+   BUM:  Broadcast, Unknown Unicast, and Multicast frames
+
+   Clos:  A multistage network topology described in [CLOS1953], where
+      all the edge switches (or Leafs) are connected to all the core
+      switches (or Spines).  Typically used in Data Centers.
+
+   DF and NDF:  Designated Forwarder and Non-Designated Forwarder,
+      respectively.  These are the roles that a given PE can have in a
+      given ES.
+
+   ECMP:  Equal-Cost Multipath
+
+   ES:  Ethernet Segment.  When a Tenant System (TS) is connected to one
+      or more NVEs via a set of Ethernet links, that set of links is
+      referred to as an "Ethernet Segment".  Each ES is represented by a
+      unique Ethernet Segment Identifier (ESI) in the NVO3 network, and
+      the ESI is used in EVPN routes that are specific to that ES.
+
+   Ethernet Tag:  Used to represent a Broadcast Domain that is
+      configured on a given ES for the purpose of Designated Forwarder
+      election.  Note that any of the following may be used to represent
+      a Broadcast Domain: VIDs (including Q-in-Q tags), configured IDs,
+      VNIs, normalized VIDs, Service Instance Identifiers (I-SIDs),
+      etc., as long as the representation of the Broadcast Domains is
+      configured consistently across the multihomed PEs attached to that
+      ES.
+
+   EVI or EVPN Instance:  A Layer 2 Virtual Network that uses an EVPN
+      control plane to exchange reachability information among the
+      member NVEs.  It corresponds to a set of MAC-VRFs of the same
+      tenant.  See MAC-VRF in this section.
+
+   EVPN:  Ethernet Virtual Private Network, as described in [RFC7432].
+
+   EVPN VLAN-Aware Bundle Service Interface:  Similar to the VLAN-bundle
+      interface but each individual VLAN value is mapped to a different
+      Broadcast Domain.  In this interface, there are multiple Broadcast
+      Domains per EVI for a given tenant.  Each Broadcast Domain is
+      identified by an "Ethernet Tag", which is a control plane value
+      that identifies the routes for the Broadcast Domain within the
+      EVI.
+
+   EVPN VLAN-Based Service Interface:  One of the three service
+      interfaces defined in [RFC7432].  It is characterized as a
+      Broadcast Domain that uses a single VLAN per physical access port
+      to attach tenant traffic to the Broadcast Domain.  In this service
+      interface, there is only one Broadcast Domain per EVI.
+
+   EVPN VLAN-Bundle Service Interface:  Similar to the VLAN-based
+      interface but uses a bundle of VLANs per physical port to attach
+      tenant traffic to the Broadcast Domain.  Like the VLAN-based
+      interface, there is only one Broadcast Domain per EVI.
+
+   Geneve:  Generic Network Virtualization Encapsulation.  An NVO3
+      encapsulation defined in [RFC8926].
+
+   IP-VRF:  IP Virtual Routing and Forwarding table, as defined in
+      [RFC4364].  It stores IP Prefixes that are part of the tenant's IP
+      space and are distributed among NVEs of the same tenant by EVPN.
+      A Route Distinguisher (RD) and one or more Route Targets (RTs) are
+      required properties of an IP-VRF.  An IP-VRF is instantiated in an
+      NVE for a given tenant if the NVE is attached to multiple subnets
+      of the tenant and local inter-subnet forwarding is required across
+      those subnets.
+
+   IRB:  Integrated Routing and Bridging.  It refers to the logical
+      interface that connects a Broadcast Domain instance (or a BT) to
+      an IP-VRF and forwards packets with a destination in a different
+      subnet.
+
+   MAC-VRF:  A MAC Virtual Routing and Forwarding table, as defined in
+      [RFC7432].  The instantiation of an EVI (EVPN Instance) in an NVE.
+      A Route Distinguisher (RD) and one or more RTs are required
+      properties of a MAC-VRF, and they are normally different from the
+      ones defined in the associated IP-VRF (if the MAC-VRF has an IRB
+      interface).
+
+   NVE:  Network Virtualization Edge.  A network entity that sits at the
+      edge of an underlay network and implements Layer 2 and/or Layer 3
+      network virtualization functions.  The network-facing side of the
+      NVE uses the underlying Layer 3 network to tunnel tenant frames to
+      and from other NVEs.  The tenant-facing side of the NVE sends and
+      receives Ethernet frames to and from individual Tenant Systems.
+      In this document, an NVE could be implemented as a virtual switch
+      within a hypervisor, a switch, or a router, and it runs EVPN in
+      the control plane.
+
+   NVO3 tunnels:  Network Virtualization over Layer 3 tunnels.  In this
+      document, NVO3 tunnels refer to a way to encapsulate tenant frames
+      or packets into IP packets, whose IP Source Addresses (SAs) or
+      Destination Addresses (DAs) belong to the underlay IP address
+      space, and identify NVEs connected to the same underlay network.
+      Examples of NVO3 tunnel encapsulations are VXLAN [RFC7348], Geneve
+      [RFC8926], or MPLSoUDP [RFC7510].
+
+   PE:  Provider Edge
+
+   PMSI:  Provider Multicast Service Interface
+
+   PTA:  PMSI Tunnel Attribute
+
+   RT and RD:  Route Target and Route Distinguisher, respectively.
+
+   RT-1, RT-2, RT-3, etc.:  These refer to the Route Types followed by
+      the type numbers as defined in the "EVPN Route Types" IANA
+      registry (see <https://www.iana.org/assignments/evpn/>).
+
+   SA and DA:  Source Address and Destination Address, respectively.
+      They are used along with MAC or IP, e.g., IP SA or MAC DA.
+
+   SBD:  Supplementary Broadcast Domain, as defined in [RFC9136].  It is
+      a Broadcast Domain that does not have any Attachment Circuits,
+      only has IRB interfaces, and provides connectivity among all the
+      IP-VRFs of a tenant in the Interface-ful IP-VRF-to-IP-VRF models.
+
+   TS:  Tenant System.  A physical or virtual system that can play the
+      role of a host or a forwarding element, such as a router, switch,
+      firewall, etc.  It belongs to a single tenant and connects to one
+      or more Broadcast Domains of that tenant.
+
+   VID:  Virtual Local Area Network Identifier
+
+   VNI:  Virtual Network Identifier.  Irrespective of the NVO3
+      encapsulation, the tunnel header always includes a VNI that is
+      added at the ingress NVE (based on the mapping table lookup) and
+      identifies the BT at the egress NVE.  This VNI is called VNI in
+      VXLAN or Geneve, Virtual Subnet ID (VSID) in nvGRE, or Label in
+      MPLSoGRE or MPLSoUDP.  This document refers to VNI as a generic
+      VNI for any NVO3 encapsulation.
+
+   VXLAN:  Virtual eXtensible Local Area Network.  An NVO3 encapsulation
+      defined in [RFC7348].
+
+3.  Why is EVPN Needed in NVO3 Networks?
+
+   Data Centers have adopted NVO3 architectures mostly due to the issues
+   discussed in [RFC7364].  The architecture of a Data Center is
+   nowadays based on a Clos design, where every Leaf is connected to a
+   layer of Spines and there is a number of ECMPs between any two Leaf
+   nodes.  All the links between Leaf and Spine nodes are routed links,
+   forming what we also know as an underlay IP Fabric.  The underlay IP
+   Fabric does not have issues with loops or flooding (like old Spanning
+   Tree Data Center designs did), convergence is fast, and ECMP
+   generally distributes utilization well across all the links.
+
+   On this architecture, and as discussed by [RFC7364], multi-tenant
+   intra-subnet and inter-subnet connectivity services are provided by
+   NVO3 tunnels.  VXLAN [RFC7348] and Geneve [RFC8926] are two examples
+   of such NVO3 tunnels.
+
+   Why is a control plane protocol along with NVO3 tunnels helpful?
+   There are three main reasons:
+
+   a.  Auto-discovery of the remote NVEs that are attached to the same
+       VPN instance (Layer 2 and/or Layer 3) as the ingress NVE is.
+
+   b.  Dissemination of the MAC/IP host information so that mapping
+       tables can be populated on the remote NVEs.
+
+   c.  Advanced features such as MAC Mobility, MAC Protection, BUM and
+       ARP/ND traffic reduction/suppression, multihoming, functionality
+       similar to Prefix Independent Convergence (PIC) [BGP-PIC], fast
+       convergence, etc.
+
+   "Flood and learn" is a possible approach to achieve points (a) and
+   (b) above for multipoint Ethernet services.  "Flood and learn" refers
+   to "flooding" BUM traffic from the ingress NVE to all the egress NVEs
+   attached to the same Broadcast Domain instead of using a specific
+   control plane on the NVEs.  The egress NVEs may then use data path
+   source MAC address "learning" on the frames received over the NVO3
+   tunnels.  When the destination host replies and the frames arrive at
+   the NVE that initially flooded BUM frames, the NVE will also "learn"
+   the source MAC address of the frame encapsulated on the NVO3 tunnel.
+   This approach has the following drawbacks:
+
+   *  In order to flood a given BUM frame, the ingress NVE must know the
+      IP addresses of the remote NVEs attached to the same Broadcast
+      Domain.  This may be done as follows:
+
+      -  The remote tunnel IP addresses can be statically provisioned on
+         the ingress NVE.  If the ingress NVE receives a BUM frame for
+         the Broadcast Domain on an ingress Attachment Circuit, it will
+         do ingress replication and will send the frame to all the
+         configured egress NVE destination IP addresses in the Broadcast
+         Domain.
+
+      -  All the NVEs attached to the same Broadcast Domain can
+         subscribe to an underlay IP multicast group that is dedicated
+         to that Broadcast Domain.  When an ingress NVE receives a BUM
+         frame on an ingress Attachment Circuit, it will send a single
+         copy of the frame encapsulated into an NVO3 tunnel using the
+         multicast address as the destination IP address of the tunnel.
+         This solution requires PIM in the underlay network and the
+         association of individual Broadcast Domains to underlay IP
+         multicast groups.
+
+   *  "Flood and learn" solves the issues of auto-discovery and the
+      learning of the MAC to VNI/tunnel IP mapping on the NVEs for a
+      given Broadcast Domain.  However, it does not provide a solution
+      for advanced features, and it does not scale well (mostly due to
+      the need for constant flooding and the underlay PIM states that
+      must be maintained).
+
+   EVPN provides a unified control plane that solves the issues of NVE
+   auto-discovery, tenant MAC/IP dissemination, and advanced features in
+   a scalable way and keeps the independence of the underlay IP Fabric;
+   i.e., there is no need to enable PIM in the underlay network and
+   maintain multicast states for tenant Broadcast Domains.
+
+   Section 4 describes how EVPN can be used to meet the control plane
+   requirements in an NVO3 network.
+
+4.  Applicability of EVPN to NVO3 Networks
+
+   This section discusses the applicability of EVPN to NVO3 networks.
+   The intent is not to provide a comprehensive explanation of the
+   protocol itself, but to give an introduction and point at the
+   corresponding reference document so the reader can easily find more
+   details if needed.
+
+4.1.  EVPN Route Types Used in NVO3 Networks
+
+   EVPN supports multiple Route Types, and each type has a different
+   function.  For convenience, Table 1 shows a summary of all the
+   existing EVPN Route Types and their usages.  In this document, we may
+   refer to these route types as RT-x routes, where x is the type number
+   included in the first column of Table 1.
+
+     +======+================+=======================================+
+     | Type | Description    | Usage                                 |
+     +======+================+=======================================+
+     | 1    | Ethernet Auto- | Multihoming: Used for MAC mass-       |
+     |      | Discovery      | withdraw when advertised per Ethernet |
+     |      |                | Segment and for aliasing/backup       |
+     |      |                | functions when advertised per EVI.    |
+     +------+----------------+---------------------------------------+
+     | 2    | MAC/IP         | Host MAC/IP dissemination; supports   |
+     |      | Advertisement  | MAC Mobility and protection.          |
+     +------+----------------+---------------------------------------+
+     | 3    | Inclusive      | NVE discovery and BUM flooding tree   |
+     |      | Multicast      | setup.                                |
+     |      | Ethernet Tag   |                                       |
+     +------+----------------+---------------------------------------+
+     | 4    | Ethernet       | Multihoming: ES auto-discovery and DF |
+     |      | Segment        | election.                             |
+     +------+----------------+---------------------------------------+
+     | 5    | IP Prefix      | IP Prefix dissemination.              |
+     +------+----------------+---------------------------------------+
+     | 6    | Selective      | Indicate interest for a multicast S,G |
+     |      | Multicast      | or *,G.                               |
+     |      | Ethernet Tag   |                                       |
+     +------+----------------+---------------------------------------+
+     | 7    | Multicast Join | Multihoming: S,G or *,G state synch.  |
+     |      | Synch          |                                       |
+     +------+----------------+---------------------------------------+
+     | 8    | Multicast      | Multihoming: S,G or *,G leave synch.  |
+     |      | Leave Synch    |                                       |
+     +------+----------------+---------------------------------------+
+     | 9    | Per-Region     | BUM tree creation across regions.     |
+     |      | I-PMSI A-D     |                                       |
+     +------+----------------+---------------------------------------+
+     | 10   | S-PMSI A-D     | Multicast tree for S,G or *,G states. |
+     +------+----------------+---------------------------------------+
+     | 11   | Leaf A-D       | Used for responses to explicit        |
+     |      |                | tracking.                             |
+     +------+----------------+---------------------------------------+
+
+                         Table 1: EVPN Route Types
+
+4.2.  EVPN Basic Applicability for Layer 2 Services
+
+   Although the applicability of EVPN to NVO3 networks spans multiple
+   documents, EVPN's baseline specification is [RFC7432].  [RFC7432]
+   allows multipoint Layer 2 VPNs to be operated as IP VPNs [RFC4364],
+   where MACs and the information to set up flooding trees are
+   distributed by Multiprotocol BGP (MP-BGP) [RFC4760].  Based on
+   [RFC7432], [RFC8365] describes how to use EVPN to deliver Layer 2
+   services specifically in NVO3 networks.
+
+   Figure 1 represents a Layer 2 service deployed with an EVPN Broadcast
+   Domain in an NVO3 network.
+
+                                +--TS2---+
+                                *        | Single-Active
+                                *        |   ESI-1
+                              +----+  +----+
+                              |BD1 |  |BD1 |
+                +-------------|    |--|    |-----------+
+                |             +----+  +----+           |
+                |              NVE2    NVE3          NVE4
+                |           EVPN NVO3 Network       +----+
+           NVE1(IP-A)                               | BD1|-----+
+          +-------------+      RT-2                 |    |     |
+          |             |    +-------+              +----+     |
+          |   +----+    |    |MAC1   |               NVE5     TS3
+   TS1--------|BD1 |    |    |IP1    |              +----+     |
+   MAC1   |   +----+    |    |Label L|--->          | BD1|-----+
+   IP1    |             |    |NH IP-A|              |    | All-Active
+          | Hypervisor  |    +-------+              +----+  ESI-2
+          +-------------+                              |
+                +--------------------------------------+
+
+             Figure 1: EVPN for L2 in an NVO3 Network - Example
+
+   In a simple NVO3 network, such as the example of Figure 1, these are
+   the basic constructs that EVPN uses for Layer 2 services (or Layer 2
+   Virtual Networks):
+
+   *  BD1 is an EVPN Broadcast Domain for a given tenant and TS1, TS2,
+      and TS3 are connected to it.  The five represented NVEs are
+      attached to BD1 and are connected to the same underlay IP network.
+      That is, each NVE learns the remote NVEs' loopback addresses via
+      underlay routing protocol.
+
+   *  NVE1 is deployed as a virtual switch in a hypervisor with IP-A as
+      underlay loopback IP address.  The rest of the NVEs in Figure 1
+      are physical switches and TS2/TS3 are multihomed to them.  TS1 is
+      a virtual machine, identified by MAC1 and IP1.  TS2 and TS3 are
+      physically dual-connected to NVEs; hence, they are normally not
+      considered virtual machines.
+
+   *  The terms Single-Active and All-Active in Figure 1 refer to the
+      mode in which the TS2 and TS3 are multihomed to the NVEs in BD1.
+      In All-Active mode, all the multihoming links are active and can
+      send or receive traffic.  In Single-Active mode, only one link (of
+      the set of links connected to the NVEs) is active.
+
+4.2.1.  Auto-Discovery and Auto-Provisioning
+
+   Auto-discovery is one of the basic capabilities of EVPN.  The
+   provisioning of EVPN components in NVEs is significantly automated,
+   simplifying the deployment of services and minimizing manual
+   operations that are prone to human error.
+
+   These are some of the auto-discovery and auto-provisioning
+   capabilities available in EVPN:
+
+   *  Automation on Ethernet Segments (ESes): An Ethernet Segment is
+      defined as a group of NVEs that are attached to the same Tenant
+      System or network.  An Ethernet Segment is identified by an
+      Ethernet Segment Identifier (ESI) in the control plane, but
+      neither the ESI nor the NVEs that share the same Ethernet Segment
+      are required to be manually provisioned in the local NVE.
+
+      -  If the multihomed Tenant System or network is running
+         protocols, such as the Link Aggregation Control Protocol (LACP)
+         [IEEE.802.1AX_2014], the Multiple Spanning Tree Protocol
+         (MSTP), G.8032, etc., and all the NVEs in the Ethernet Segment
+         can listen to the protocol PDUs to uniquely identify the
+         multihomed Tenant System/network, then the ESI can be "auto-
+         sensed" or "auto-provisioned" following the guidelines in
+         Section 5 of [RFC7432].  The ESI can also be auto-derived out
+         of other parameters that are common to all NVEs attached to the
+         same Ethernet Segment.
+
+      -  As described in [RFC7432], EVPN can also auto-derive the BGP
+         parameters required to advertise the presence of a local
+         Ethernet Segment in the control plane (RT and RD).  Local
+         Ethernet Segments are advertised using Ethernet Segment routes,
+         and the ESI-import Route Target used by Ethernet Segment routes
+         can be auto-derived based on the procedures of Section 7.6 of
+         [RFC7432].
+
+      -  By listening to other Ethernet Segment routes that match the
+         local ESI and import Route Target, an NVE can also auto-
+         discover the other NVEs participating in the multihoming for
+         the Ethernet Segment.
+
+      -  Once the NVE has auto-discovered all the NVEs attached to the
+         same Ethernet Segment, the NVE can automatically perform the
+         Designated Forwarder election algorithm (which determines the
+         NVE that will forward traffic to the multihomed Tenant System/
+         network).  EVPN guarantees that all the NVEs in the Ethernet
+         Segment have a consistent Designated Forwarder election.
+
+   *  Auto-provisioning of services: When deploying a Layer 2 service
+      for a tenant in an NVO3 network, all the NVEs attached to the same
+      subnet must be configured with a MAC-VRF and the Broadcast Domain
+      for the subnet, as well as certain parameters for them.  Note that
+      if the EVPN service interfaces are VLAN-based or VLAN-bundle,
+      implementations do not normally have a specific provisioning for
+      the Broadcast Domain since, in this case, it is the same construct
+      as the MAC-VRF.  EVPN allows auto-deriving as many MAC-VRF
+      parameters as possible.  As an example, the MAC-VRF's Route Target
+      and Route Distinguisher for the EVPN routes may be auto-derived.
+      Section 5.1.2.1 of [RFC8365] specifies how to auto-derive a MAC-
+      VRF's Route Target as long as a VLAN-based service interface is
+      implemented.  [RFC7432] specifies how to auto-derive the Route
+      Distinguisher.
+
+4.2.2.  Remote NVE Auto-Discovery
+
+   Auto-discovery via MP-BGP [RFC4760] is used to discover the remote
+   NVEs attached to a given Broadcast Domain, the NVEs participating in
+   a given redundancy group, the tunnel encapsulation types supported by
+   an NVE, etc.
+
+   In particular, when a new MAC-VRF and Broadcast Domain are enabled,
+   the NVE will advertise a new Inclusive Multicast Ethernet Tag route.
+   Besides other fields, the Inclusive Multicast Ethernet Tag route will
+   encode the IP address of the advertising NVE, the Ethernet Tag (which
+   is zero in the case of VLAN-based and VLAN-bundle interfaces), and a
+   PMSI Tunnel Attribute (PTA) that indicates the information about the
+   intended way to deliver BUM traffic for the Broadcast Domain.
+
+   When BD1 is enabled in the example of Figure 1, NVE1 will send an
+   Inclusive Multicast Ethernet Tag route including its own IP address,
+   an Ethernet-Tag for BD1, and the PMSI Tunnel Attribute to the remote
+   NVEs.  Assuming Ingress Replication (IR) is used, the Inclusive
+   Multicast Ethernet Tag route will include an identification for
+   Ingress Replication in the PMSI Tunnel Attribute and the VNI that the
+   other NVEs in the Broadcast Domain must use to send BUM traffic to
+   the advertising NVE.  The other NVEs in the Broadcast Domain will
+   import the Inclusive Multicast Ethernet Tag route and will add NVE1's
+   IP address to the flooding list for BD1.  Note that the Inclusive
+   Multicast Ethernet Tag route is also sent with a BGP encapsulation
+   attribute [RFC9012] that indicates what NVO3 encapsulation the remote
+   NVEs should use when sending BUM traffic to NVE1.
+
+   Refer to [RFC7432] for more information about the Inclusive Multicast
+   Ethernet Tag route and forwarding of BUM traffic.  See [RFC8365] for
+   its considerations on NVO3 networks.
+
+4.2.3.  Distribution of Tenant MAC and IP Information
+
+   Tenant MAC/IP information is advertised to remote NVEs using MAC/IP
+   Advertisement routes.  Following the example of Figure 1:
+
+   *  In a given EVPN Broadcast Domain, the MAC addresses of TSes are
+      first learned at the NVE they are attached to via data path or
+      management plane learning.  In Figure 1, we assume NVE1 learns
+      MAC1/IP1 in the management plane (for instance, via Cloud
+      Management System) since the NVE is a virtual switch.  NVE2, NVE3,
+      NVE4, and NVE5 are ToR/Leaf switches, and they normally learn MAC
+      addresses via data path.
+
+   *  Once NVE1's BD1 learns MAC1/IP1, NVE1 advertises that information
+      along with a VNI and Next-Hop IP-A in a MAC/IP Advertisement
+      route.  The EVPN routes are advertised using the Route
+      Distinguisher / Route Targets of the MAC-VRF where the Broadcast
+      Domain belongs.  Similarly, all the NVEs in BD1 learn local MAC/IP
+      addresses and advertise them in MAC/IP Advertisement routes.
+
+   *  The remote NVEs can then add MAC1 to their mapping table for BD1
+      (BT).  For instance, when TS3 sends frames to NVE4 with the
+      destination MAC address = MAC1, NVE4 does a MAC lookup on the
+      Bridge Table that yields IP-A and Label L.  NVE4 can then
+      encapsulate the frame into an NVO3 tunnel with IP-A as the tunnel
+      destination IP address and L as the VNI.  Note that the MAC/IP
+      Advertisement route may also contain the host's IP address (as
+      shown in the example of Figure 1).  While the MAC of the received
+      MAC/IP Advertisement route is installed in the Bridge Table, the
+      IP address may be installed in the Proxy ARP/ND table (if enabled)
+      or in the ARP/IP-VRF tables if the Broadcast Domain has an IRB.
+      See Section 4.7.3 for more information about Proxy ARP/ND and
+      Section 4.3 for more details about IRB and Layer 3 services.
+
+   Refer to [RFC7432] and [RFC8365] for more information about the MAC/
+   IP Advertisement route and the forwarding of known unicast traffic.
+
+4.3.  EVPN Basic Applicability for Layer 3 Services
+
+   [RFC9136] and [RFC9135] are the reference documents that describe how
+   EVPN can be used for Layer 3 services.  Inter-subnet forwarding in
+   EVPN networks is implemented via IRB interfaces between Broadcast
+   Domains and IP-VRFs.  An EVPN Broadcast Domain corresponds to an IP
+   subnet.  When IP packets generated in a Broadcast Domain are destined
+   to a different subnet (different Broadcast Domain) of the same
+   tenant, the packets are sent to the IRB attached to the local
+   Broadcast Domain in the source NVE.  As discussed in [RFC9135],
+   depending on how the IP packets are forwarded between the ingress NVE
+   and the egress NVE, there are two forwarding models: Asymmetric and
+   Symmetric.
+
+   The Asymmetric model is illustrated in the example of Figure 2, and
+   it requires the configuration of all the Broadcast Domains of the
+   tenant in all the NVEs attached to the same tenant.  That way, there
+   is no need to advertise IP Prefixes between NVEs since all the NVEs
+   are attached to all the subnets.  It is called "Asymmetric" because
+   the ingress and egress NVEs do not perform the same number of lookups
+   in the data plane.  In Figure 2, if TS1 and TS2 are in different
+   subnets and TS1 sends IP packets to TS2, the following lookups are
+   required in the data path: a MAC lookup at BD1's table, an IP lookup
+   at the IP-VRF, a MAC lookup at BD2's table at the ingress NVE1, and
+   only a MAC lookup at the egress NVE.  The two IP-VRFs in Figure 2 are
+   not connected by tunnels, and all the connectivity between the NVEs
+   is done based on tunnels between the Broadcast Domains.
+
+                 +-------------------------------------+
+                 |             EVPN NVO3               |
+                 |                                     |
+               NVE1                                 NVE2
+         +--------------------+            +--------------------+
+         | +---+IRB +------+  |            |  +------+IRB +---+ |
+   TS1-----|BD1|----|IP-VRF|  |            |  |IP-VRF|----|BD1| |
+         | +---+    |      |  |            |  |      |    +---+ |
+         | +---+    |      |  |            |  |      |    +---+ |
+         | |BD2|----|      |  |            |  |      |----|BD2|----TS2
+         | +---+IRB +------+  |            |  +------+IRB +---+ |
+         +--------------------+            +--------------------+
+                 |                                     |
+                 +-------------------------------------+
+
+        Figure 2: EVPN for L3 in an NVO3 Network - Asymmetric Model
+
+   In the Symmetric model, depicted in Figure 3, the same number of data
+   path lookups is needed at the ingress and egress NVEs.  For example,
+   if TS1 sends IP packets to TS3, the following data path lookups are
+   required: a MAC lookup at NVE1's BD1 table, an IP lookup at NVE1's
+   IP-VRF, and an IP lookup and MAC lookup at NVE2's IP-VRF and BD3,
+   respectively.  In the Symmetric model, the inter-subnet connectivity
+   between NVEs is done based on tunnels between the IP-VRFs.
+
+                 +-------------------------------------+
+                 |             EVPN NVO3               |
+                 |                                     |
+               NVE1                                 NVE2
+         +--------------------+            +--------------------+
+         | +---+IRB +------+  |            |  +------+IRB +---+ |
+   TS1-----|BD1|----|IP-VRF|  |            |  |IP-VRF|----|BD3|-----TS3
+         | +---+    |      |  |            |  |      |    +---+ |
+         | +---+IRB |      |  |            |  +------+          |
+   TS2-----|BD2|----|      |  |            +--------------------+
+         | +---+    +------+  |                        |
+         +--------------------+                        |
+                 |                                     |
+                 +-------------------------------------+
+
+         Figure 3: EVPN for L3 in an NVO3 Network - Symmetric Model
+
+   The Symmetric model scales better than the Asymmetric model because
+   it does not require the NVEs to be attached to all the tenant's
+   subnets.  However, it requires the use of NVO3 tunnels on the IP-VRFs
+   and the exchange of IP Prefixes between the NVEs in the control
+   plane.  EVPN uses MAC/IP Advertisement routes for the exchange of
+   host IP routes and IP Prefix routes for the exchange of prefixes of
+   any length, including host routes.  As an example, in Figure 3, NVE2
+   needs to advertise TS3's host route and/or TS3's subnet so that the
+   IP lookup on NVE1's IP-VRF succeeds.
+
+   [RFC9135] specifies the use of MAC/IP Advertisement routes for the
+   advertisement of host routes.  Section 4.4.1 of [RFC9136] specifies
+   the use of IP Prefix routes for the advertisement of IP Prefixes in
+   an "Interface-less IP-VRF-to-IP-VRF Model".  The Symmetric model for
+   host routes can be implemented following either approach:
+
+   a.  [RFC9135] uses MAC/IP Advertisement routes to convey the
+       information to populate Layer 2, ARP/ND, and Layer 3 Forwarding
+       Information Base tables in the remote NVE.  For instance, in
+       Figure 3, NVE2 would advertise a MAC/IP Advertisement route with
+       TS3's IP and MAC addresses and include two labels / VNIs: a
+       label-3/VNI-3 that identifies BD3 for MAC lookup (that would be
+       used for Layer 2 traffic in case NVE1 was attached to BD3 too)
+       and a label-1/VNI-1 that identifies the IP-VRF for IP lookup
+       (that would be used for Layer 3 traffic).  NVE1 imports the MAC/
+       IP Advertisement route and installs TS3's IP in the IP-VRF route
+       table with label-1/VNI-1.  Traffic, e.g., from TS2 to TS3, would
+       be encapsulated with label-1/VNI-1 and forwarded to NVE2.
+
+   b.  [RFC9136] uses MAC/IP Advertisement routes to convey the
+       information to populate the Layer 2 Forwarding Information Base,
+       ARP/ND tables, and IP Prefix routes to populate the IP-VRF Layer
+       3 Forwarding Information Base table.  For instance, in Figure 3,
+       NVE2 would advertise a MAC/IP Advertisement route including TS3's
+       MAC and IP addresses with a single label-3/VNI-3.  In this
+       example, this MAC/IP Advertisement route wouldn't be imported by
+       NVE1 because NVE1 is not attached to BD3.  In addition, NVE2
+       would advertise an IP Prefix route with TS3's IP address and
+       label-1/VNI-1.  This IP Prefix route would be imported by NVE1's
+       IP-VRF and the host route installed in the Layer 3 Forwarding
+       Information Base associated with label-1/VNI-1.  Traffic from TS2
+       to TS3 would be encapsulated with label-1/VNI-1.
+
+4.4.  EVPN as Control Plane for NVO3 Encapsulations and Geneve
+
+   [RFC8365] describes how to use EVPN for NVO3 encapsulations, such us
+   VXLAN, nvGRE, or MPLSoGRE.  The procedures can be easily applicable
+   to any other NVO3 encapsulation, particularly Geneve.
+
+   Geneve [RFC8926] is the proposed standard encapsulation specified in
+   the IETF Network Virtualization Overlays Working Group.  The EVPN
+   control plane can signal the Geneve encapsulation type in the BGP
+   Tunnel Encapsulation Extended Community (see [RFC9012]).
+
+   Geneve requires a control plane [NVO3-ENCAP] to:
+
+   *  Negotiate a subset of Geneve option TLVs that can be carried on a
+      Geneve tunnel,
+
+   *  Enforce an order for Geneve option TLVs, and
+
+   *  Limit the total number of options that could be carried on a
+      Geneve tunnel.
+
+   The EVPN control plane can easily extend the BGP Tunnel Encapsulation
+   attribute sub-TLV [RFC9012] to specify the Geneve tunnel options that
+   can be received or transmitted over a Geneve tunnel by a given NVE.
+   [EVPN-GENEVE] describes the EVPN control plane extensions to support
+   Geneve.
+
+4.5.  EVPN OAM and Application to NVO3
+
+   EVPN Operations, Administration, and Maintenance (OAM), as described
+   in [EVPN-LSP-PING], defines mechanisms to detect data plane failures
+   in an EVPN deployment over an MPLS network.  These mechanisms detect
+   failures related to point-to-point (P2P) and Point-to-Multipoint
+   (P2MP) connectivity, for multi-tenant unicast and multicast Layer 2
+   traffic, between multi-tenant access nodes connected to EVPN PE(s),
+   and in a single-homed, Single-Active, or All-Active redundancy model.
+
+   In general, EVPN OAM mechanisms defined for EVPN deployed in MPLS
+   networks are equally applicable for EVPN in NVO3 networks.
+
+4.6.  EVPN as the Control Plane for NVO3 Security
+
+   EVPN can be used to signal the security protection capabilities of a
+   sender NVE, as well as what portion of an NVO3 packet (taking a
+   Geneve packet as an example) can be protected by the sender NVE, to
+   ensure the privacy and integrity of tenant traffic carried over the
+   NVO3 tunnels [SECURE-EVPN].
+
+4.7.  Advanced EVPN Features for NVO3 Networks
+
+   This section describes how EVPN can be used to deliver advanced
+   capabilities in NVO3 networks.
+
+4.7.1.  Virtual Machine (VM) Mobility
+
+   [RFC7432] replaces the classic Ethernet "flood and learn" behavior
+   among NVEs with BGP-based MAC learning.  In return, this provides
+   more control over the location of MAC addresses in the Broadcast
+   Domain and consequently advanced features, such as MAC Mobility.  If
+   we assume that Virtual Machine (VM) Mobility means the VM's MAC and
+   IP addresses move with the VM, EVPN's MAC Mobility is the required
+   procedure that facilitates VM Mobility.  According to Section 15 of
+   [RFC7432], when a MAC is advertised for the first time in a Broadcast
+   Domain, all the NVEs attached to the Broadcast Domain will store
+   Sequence Number zero for that MAC.  When the MAC "moves" to a remote
+   NVE within the same Broadcast Domain, the NVE that just learned the
+   MAC locally increases the Sequence Number in the MAC/IP Advertisement
+   route's MAC Mobility extended community to indicate that it owns the
+   MAC now.  That makes all the NVEs in the Broadcast Domain change
+   their tables immediately with no need to wait for any aging timer.
+   EVPN guarantees a fast MAC Mobility without flooding or packet drops
+   in the Broadcast Domain.
+
+4.7.2.  MAC Protection, Duplication Detection, and Loop Protection
+
+   The advertisement of MACs in the control plane allows advanced
+   features such as MAC Protection, Duplication Detection, and Loop
+   Protection.
+
+   In a MAC/IP Advertisement route, MAC Protection refers to EVPN's
+   ability to indicate that a MAC must be protected by the NVE receiving
+   the route [RFC7432].  The Protection is indicated in the "Sticky bit"
+   of the MAC Mobility extended community sent along the MAC/IP
+   Advertisement route for a MAC.  NVEs' Attachment Circuits that are
+   connected to subject-to-be-protected servers or VMs may set the
+   Sticky bit on the MAC/IP Advertisement routes sent for the MACs
+   associated with the Attachment Circuits.  Also, statically configured
+   MAC addresses should be advertised as Protected MAC addresses since
+   they are not subject to MAC Mobility procedures.
+
+   MAC Duplication Detection refers to EVPN's ability to detect
+   duplicate MAC addresses [RFC7432].  A "MAC move" is a relearn event
+   that happens at an access Attachment Circuit or through a MAC/IP
+   Advertisement route with a Sequence Number that is higher than the
+   stored one for the MAC.  When a MAC moves a number of times (N)
+   within an M-second window between two NVEs, the MAC is declared as a
+   duplicate and the detecting NVE does not re-advertise the MAC
+   anymore.
+
+   [RFC7432] provides MAC Duplication Detection, and with an extension,
+   it can protect the Broadcast Domain against loops created by backdoor
+   links between NVEs.  The same principle (based on the Sequence
+   Number) may be extended to protect the Broadcast Domain against
+   loops.  When a MAC is detected as a duplicate, the NVE may install it
+   as a drop-MAC and discard received frames with source MAC address or
+   the destination MAC address matching that duplicate MAC.  The MAC
+   Duplication extension to support Loop Protection is described in
+   Section 15.3 of [RFC7432BIS].
+
+4.7.3.  Reduction/Optimization of BUM Traffic in Layer 2 Services
+
+   In Broadcast Domains with a significant amount of flooding due to
+   Unknown Unicast and broadcast frames, EVPN may help reduce and
+   sometimes even suppress the flooding.
+
+   In Broadcast Domains where most of the broadcast traffic is caused by
+   the Address Resolution Protocol (ARP) and the Neighbor Discovery
+   Protocol (NDP) on the Tenant Systems, EVPN's Proxy ARP and Proxy ND
+   capabilities may reduce the flooding drastically.  The use of Proxy
+   ARP/ND is specified in [RFC9161].
+
+   Proxy ARP/ND procedures, along with the assumption that Tenant
+   Systems always issue a Gratuitous ARP (GARP) or an unsolicited
+   Neighbor Advertisement message when they come up in the Broadcast
+   Domain, may drastically reduce the Unknown Unicast flooding in the
+   Broadcast Domain.
+
+   The flooding caused by Tenant Systems' IGMP / Multicast Listener
+   Discovery (MLD) or PIM messages in the Broadcast Domain may also be
+   suppressed by the use of IGMP/MLD and PIM Proxy functions, as
+   specified in [RFC9251] and [EVPN-PIM-PROXY].  These two documents
+   also specify how to forward IP multicast traffic efficiently within
+   the same Broadcast Domain, translate soft state IGMP/MLD/PIM messages
+   into hard state BGP routes, and provide fast convergence redundancy
+   for IP multicast on multihomed ESes.
+
+4.7.4.  Ingress Replication (IR) Optimization for BUM Traffic
+
+   When an NVE attached to a given Broadcast Domain needs to send BUM
+   traffic for the Broadcast Domain to the remote NVEs attached to the
+   same Broadcast Domain, Ingress Replication is a very common option in
+   NVO3 networks since it is completely independent of the multicast
+   capabilities of the underlay network.  Also, if the optimization
+   procedures to reduce/suppress the flooding in the Broadcast Domain
+   are enabled (Section 4.7.3) in spite of creating multiple copies of
+   the same frame at the ingress NVE, Ingress Replication may be good
+   enough.  However, in Broadcast Domains where Multicast (or Broadcast)
+   traffic is significant, Ingress Replication may be very inefficient
+   and cause performance issues on virtual switch-based NVEs.
+
+   [EVPN-OPT-IR] specifies the use of Assisted Replication (AR) NVO3
+   tunnels in EVPN Broadcast Domains.  AR retains the independence of
+   the underlay network while providing a way to forward Broadcast and
+   multicast traffic efficiently.  AR uses AR-REPLICATORs that can
+   replicate the broadcast/multicast traffic on behalf of the AR-LEAF
+   NVEs.  The AR-LEAF NVEs are typically virtual switches or NVEs with
+   limited replication capabilities.  AR can work in a single-stage
+   replication mode (Non-Selective Mode) or in a dual-stage replication
+   mode (Selective Mode).  Both modes are detailed in [EVPN-OPT-IR].
+
+   In addition, [EVPN-OPT-IR] describes a procedure to avoid sending BUM
+   to certain NVEs that do not need that type of traffic.  This is done
+   by enabling Pruned Flood Lists (PFLs) on a given Broadcast Domain.
+   For instance, a virtual switch NVE that learns all its local MAC
+   addresses for a Broadcast Domain via a Cloud Management System does
+   not need to receive the Broadcast Domain's Unknown Unicast traffic.
+   PFLs help optimize the BUM flooding in the Broadcast Domain.
+
+4.7.5.  EVPN Multihoming
+
+   Another fundamental concept in EVPN is multihoming.  A given Tenant
+   System can be multihomed to two or more NVEs for a given Broadcast
+   Domain, and the set of links connected to the same Tenant System is
+   defined as an ES.  EVPN supports Single-Active and All-Active
+   multihoming.  In Single-Active multihoming, only one link in the
+   Ethernet Segment is active.  In All-Active multihoming, all the links
+   in the Ethernet Segment are active for unicast traffic.  Both modes
+   support load-balancing:
+
+   *  Single-Active multihoming means per-service load-balancing to/from
+      the Tenant System.  For example, in Figure 1 for BD1, only one of
+      the NVEs can forward traffic from/to TS2.  For a different
+      Broadcast Domain, the other NVE may forward traffic.
+
+   *  All-active multihoming means per-flow load-balancing for unicast
+      frames to/from the Tenant System.  That is, in Figure 1 and for
+      BD1, both NVE4 and NVE5 can forward known unicast traffic to/from
+      TS3.  For BUM traffic, only one of the two NVEs can forward
+      traffic to TS3, and both can forward traffic from TS3.
+
+   There are two key aspects in the EVPN multihoming procedures:
+
+   Designated Forwarder (DF) election:
+      The Designated Forwarder is the NVE that forwards the traffic to
+      the Ethernet Segment in Single-Active mode.  In the case of All-
+      Active mode, the Designated Forwarder is the NVE that forwards the
+      BUM traffic to the Ethernet Segment.
+
+   Split-horizon function:
+      Prevents the Tenant System from receiving echoed BUM frames that
+      the Tenant System itself sent to the Ethernet Segment.  This is
+      especially relevant in All-Active ESes where the TS may forward
+      BUM frames to a Non-Designated Forwarder NVE that can flood the
+      BUM frames back to the Designated Forwarder NVE and then back to
+      the TS.  As an example, assuming NVE4 is the Designated Forwarder
+      for ESI-2 in BD1, Figure 1 shows that BUM frames sent from TS3 to
+      NVE5 will be received at NVE4.  NVE4 will forward them back to TS3
+      since NVE4 is the Designated Forwarder for BD1.  Split-horizon
+      allows NVE4 (and any multihomed NVE for that matter) to identify
+      if an EVPN BUM frame is coming from the same Ethernet Segment or a
+      different one.  If the frame belongs to the same ESI-2, NVE4 will
+      not forward the BUM frame to TS3 in spite of being the Designated
+      Forwarder.
+
+   While [RFC7432] describes the default algorithm for the Designated
+   Forwarder election, [RFC8584] and [EVPN-PREF-DF] specify other
+   algorithms and procedures that optimize the Designated Forwarder
+   election.
+
+   The split-horizon function is specified in [RFC7432], and it is
+   carried out by using a special ESI-label that it identifies in the
+   data path with all the BUM frames originating from a given NVE and
+   Ethernet Segment.  Since the ESI-label is an MPLS label, it cannot be
+   used in all the non-MPLS NVO3 encapsulations.  Therefore, [RFC8365]
+   defines a modified split-horizon procedure that is based on the
+   source IP address of the NVO3 tunnel; it is known as "Local-Bias".
+   It is worth noting that Local-Bias only works for All-Active
+   multihoming, and not for Single-Active multihoming.
+
+4.7.6.  EVPN Recursive Resolution for Inter-subnet Unicast Forwarding
+
+   Section 4.3 describes how EVPN can be used for inter-subnet
+   forwarding among subnets of the same tenant.  MAC/IP Advertisement
+   routes and IP Prefix routes allow the advertisement of host routes
+   and IP Prefixes (IP Prefix route) of any length.  The procedures
+   outlined by Section 4.3 are similar to the ones in [RFC4364], but
+   they are only for NVO3 tunnels.  However, [RFC9136] also defines
+   advanced inter-subnet forwarding procedures that allow the resolution
+   of IP Prefix routes not only to BGP next hops but also to "overlay
+   indexes" that can be a MAC, a Gateway IP (GW-IP), or an ESI, all of
+   them in the tenant space.
+
+   Figure 4 illustrates an example that uses Recursive Resolution to a
+   GW-IP as per Section 4.4.2 of [RFC9136].  In this example, IP-VRFs in
+   NVE1 and NVE2 are connected by a Supplementary Broadcast Domain
+   (SBD).  An SBD is a Broadcast Domain that connects all the IP-VRFs of
+   the same tenant via IRB and has no Attachment Circuits.  NVE1
+   advertises the host route TS2-IP/L (IP address and Prefix Length of
+   TS2) in an IP Prefix route with overlay index GW-IP=IP1.  Also, IP1
+   is advertised in a MAC/IP Advertisement route associated with M1,
+   VNI-S, and BGP next-hop NVE1.  Upon importing the two routes, NVE2
+   installs TS2-IP/L in the IP-VRF with a next hop that is the GW-IP
+   IP1.  NVE2 also installs M1 in the Supplementary Broadcast Domain,
+   with VNI-S and NVE1 as next hop.  If TS3 sends a packet with IP
+   DA=TS2, NVE2 will perform a Recursive Resolution of the IP Prefix
+   route prefix information to the forwarding information of the
+   correlated MAC/IP Advertisement route.  The IP Prefix route's
+   Recursive Resolution has several advantages, such as better
+   convergence in scaled networks (since multiple IP Prefix routes can
+   be invalidated with a single withdrawal of the overlay index route)
+   or the ability to advertise multiple IP Prefix routes from an overlay
+   index that can move or change dynamically.  [RFC9136] describes a few
+   use cases.
+
+                 +-------------------------------------+
+                 |             EVPN NVO3               |
+                 |                                     +
+               NVE1                                 NVE2
+         +--------------------+            +--------------------+
+         | +---+IRB +------+  |            |  +------+IRB +---+ |
+   TS1-----|BD1|----|IP-VRF|  |            |  |IP-VRF|----|BD3|-----TS3
+         | +---+    |      |-(SBD)------(SBD)-|      |    +---+ |
+         | +---+IRB |      |IRB(IP1/M1)    IRB+------+          |
+   TS2-----|BD2|----|      |  |            +-----------+--------+
+         | +---+    +------+  |                        |
+         +--------------------+                        |
+                 |   RT-2(M1,IP1,VNI-S,NVE1)-->        |
+                 |     RT-5(TS2-IP/L,GW-IP=IP1)-->     |
+                 +-------------------------------------+
+
+            Figure 4: EVPN for L3 - Recursive Resolution Example
+
+4.7.7.  EVPN Optimized Inter-subnet Multicast Forwarding
+
+   The concept of the Supplementary Broadcast Domain described in
+   Section 4.7.6 is also used in [EVPN-IRB-MCAST] for the procedures
+   related to inter-subnet multicast forwarding across Broadcast Domains
+   of the same tenant.  For instance, [EVPN-IRB-MCAST] allows the
+   efficient forwarding of IP multicast traffic from any Broadcast
+   Domain to any other Broadcast Domain (or even to the same Broadcast
+   Domain where the source resides).  The [EVPN-IRB-MCAST] procedures
+   are supported along with EVPN multihoming and for any tree allowed on
+   NVO3 networks, including IR or AR.  [EVPN-IRB-MCAST] also describes
+   the interoperability between EVPN and other multicast technologies
+   such as Multicast VPN (MVPN) and PIM for inter-subnet multicast.
+
+   [EVPN-MVPN-SEAMLESS] describes another potential solution to support
+   EVPN to MVPN interoperability.
+
+4.7.8.  Data Center Interconnect (DCI)
+
+   Tenant Layer 2 and Layer 3 services deployed on NVO3 networks must
+   often be extended to remote NVO3 networks that are connected via non-
+   NOV3 Wide Area Networks (WANs) (mostly MPLS-based WANs).  [RFC9014]
+   defines some architectural models that can be used to interconnect
+   NVO3 networks via MPLS WANs.
+
+   When NVO3 networks are connected by MPLS WANs, [RFC9014] specifies
+   how EVPN can be used end to end in spite of using a different
+   encapsulation in the WAN.  [RFC9014] also supports the use of NVO3 or
+   Segment Routing (encoding 32-bit or 128-bit Segment Identifiers into
+   labels or IPv6 addresses, respectively) transport tunnels in the WAN.
+
+   Even if EVPN can also be used in the WAN for Layer 2 and Layer 3
+   services, there may be a need to provide a Gateway function between
+   EVPN for NVO3 encapsulations and IP VPN for MPLS tunnels if the
+   operator uses IP VPN in the WAN.  [EVPN-IPVPN-INTERWORK] specifies
+   the interworking function between EVPN and IP VPN for unicast inter-
+   subnet forwarding.  If inter-subnet multicast forwarding is also
+   needed across an IP VPN WAN, [EVPN-IRB-MCAST] describes the required
+   interworking between EVPN and MVPNs.
+
+5.  Security Considerations
+
+   This document does not introduce any new procedure or additional
+   signaling in EVPN and relies on the security considerations of the
+   individual specifications used as a reference throughout the
+   document.  In particular, and as mentioned in [RFC7432], control
+   plane and forwarding path protection are aspects to secure in any
+   EVPN domain when applied to NVO3 networks.
+
+   [RFC7432] mentions security techniques such as those discussed in
+   [RFC5925] to authenticate BGP messages, and those included in
+   [RFC4271], [RFC4272], and [RFC6952] to secure BGP are relevant for
+   EVPN in NVO3 networks as well.
+
+6.  IANA Considerations
+
+   This document has no IANA actions.
+
+7.  References
+
+7.1.  Normative References
+
+   [RFC7364]  Narten, T., Ed., Gray, E., Ed., Black, D., Fang, L.,
+              Kreeger, L., and M. Napierala, "Problem Statement:
+              Overlays for Network Virtualization", RFC 7364,
+              DOI 10.17487/RFC7364, October 2014,
+              <https://www.rfc-editor.org/info/rfc7364>.
+
+   [RFC7365]  Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y.
+              Rekhter, "Framework for Data Center (DC) Network
+              Virtualization", RFC 7365, DOI 10.17487/RFC7365, October
+              2014, <https://www.rfc-editor.org/info/rfc7365>.
+
+   [RFC7432]  Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
+              Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
+              Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
+              2015, <https://www.rfc-editor.org/info/rfc7432>.
+
+7.2.  Informative References
+
+   [BGP-PIC]  Bashandy, A., Ed., Filsfils, C., and P. Mohapatra, "BGP
+              Prefix Independent Convergence", Work in Progress,
+              Internet-Draft, draft-ietf-rtgwg-bgp-pic-19, 1 April 2023,
+              <https://datatracker.ietf.org/doc/html/draft-ietf-rtgwg-
+              bgp-pic-19>.
+
+   [CLOS1953] Clos, C., "A study of non-blocking switching networks",
+              The Bell System Technical Journal, Vol. 32, Issue 2,
+              DOI 10.1002/j.1538-7305.1953.tb01433.x, March 1953,
+              <https://ieeexplore.ieee.org/document/6770468>.
+
+   [EVPN-GENEVE]
+              Boutros, S., Ed., Sajassi, A., Drake, J., Rabadan, J., and
+              S. Aldrin, "EVPN control plane for Geneve", Work in
+              Progress, Internet-Draft, draft-ietf-bess-evpn-geneve-06,
+              26 May 2023, <https://datatracker.ietf.org/doc/html/draft-
+              ietf-bess-evpn-geneve-06>.
+
+   [EVPN-IPVPN-INTERWORK]
+              Rabadan, J., Ed., Sajassi, A., Ed., Rosen, E., Drake, J.,
+              Lin, W., Uttaro, J., and A. Simpson, "EVPN Interworking
+              with IPVPN", Work in Progress, Internet-Draft, draft-ietf-
+              bess-evpn-ipvpn-interworking-08, 5 July 2023,
+              <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
+              evpn-ipvpn-interworking-08>.
+
+   [EVPN-IRB-MCAST]
+              Lin, W., Zhang, Z., Drake, J., Rosen, E., Ed., Rabadan,
+              J., and A. Sajassi, "EVPN Optimized Inter-Subnet Multicast
+              (OISM) Forwarding", Work in Progress, Internet-Draft,
+              draft-ietf-bess-evpn-irb-mcast-09, 21 February 2023,
+              <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
+              evpn-irb-mcast-09>.
+
+   [EVPN-LSP-PING]
+              Jain, P., Sajassi, A., Salam, S., Boutros, S., and G.
+              Mirsky, "LSP-Ping Mechanisms for EVPN and PBB-EVPN", Work
+              in Progress, Internet-Draft, draft-ietf-bess-evpn-lsp-
+              ping-11, 29 May 2023,
+              <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
+              evpn-lsp-ping-11>.
+
+   [EVPN-MVPN-SEAMLESS]
+              Sajassi, A., Thiruvenkatasamy, K., Thoria, S., Gupta, A.,
+              and L. Jalil, "Seamless Multicast Interoperability between
+              EVPN and MVPN PEs", Work in Progress, Internet-Draft,
+              draft-ietf-bess-evpn-mvpn-seamless-interop-05, 13 March
+              2023, <https://datatracker.ietf.org/doc/html/draft-ietf-
+              bess-evpn-mvpn-seamless-interop-05>.
+
+   [EVPN-OPT-IR]
+              Rabadan, J., Ed., Sathappan, S., Lin, W., Katiyar, M., and
+              A. Sajassi, "Optimized Ingress Replication Solution for
+              Ethernet VPN (EVPN)", Work in Progress, Internet-Draft,
+              draft-ietf-bess-evpn-optimized-ir-12, 25 January 2022,
+              <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
+              evpn-optimized-ir-12>.
+
+   [EVPN-PIM-PROXY]
+              Rabadan, J., Ed., Kotalwar, J., Sathappan, S., Zhang, Z.,
+              and A. Sajassi, "PIM Proxy in EVPN Networks", Work in
+              Progress, Internet-Draft, draft-skr-bess-evpn-pim-proxy-
+              01, 30 October 2017,
+              <https://datatracker.ietf.org/doc/html/draft-skr-bess-
+              evpn-pim-proxy-01>.
+
+   [EVPN-PREF-DF]
+              Rabadan, J., Ed., Sathappan, S., Lin, W., Drake, J., and
+              A. Sajassi, "Preference-based EVPN DF Election", Work in
+              Progress, Internet-Draft, draft-ietf-bess-evpn-pref-df-11,
+              6 July 2023, <https://datatracker.ietf.org/doc/html/draft-
+              ietf-bess-evpn-pref-df-11>.
+
+   [IEEE.802.1AX_2014]
+              IEEE, "IEEE Standard for Local and metropolitan area
+              networks -- Link Aggregation", IEEE Std 802.1AX-2014,
+              DOI 10.1109/IEEESTD.2014.7055197, December 2014,
+              <https://doi.org/10.1109/IEEESTD.2014.7055197>.
+
+   [NVO3-ENCAP]
+              Boutros, S., Ed. and D. Eastlake 3rd, Ed., "Network
+              Virtualization Overlays (NVO3) Encapsulation
+              Considerations", Work in Progress, Internet-Draft, draft-
+              ietf-nvo3-encap-09, 7 October 2022,
+              <https://datatracker.ietf.org/doc/html/draft-ietf-nvo3-
+              encap-09>.
+
+   [RFC4271]  Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
+              Border Gateway Protocol 4 (BGP-4)", RFC 4271,
+              DOI 10.17487/RFC4271, January 2006,
+              <https://www.rfc-editor.org/info/rfc4271>.
+
+   [RFC4272]  Murphy, S., "BGP Security Vulnerabilities Analysis",
+              RFC 4272, DOI 10.17487/RFC4272, January 2006,
+              <https://www.rfc-editor.org/info/rfc4272>.
+
+   [RFC4364]  Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
+              Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
+              2006, <https://www.rfc-editor.org/info/rfc4364>.
+
+   [RFC4760]  Bates, T., Chandra, R., Katz, D., and Y. Rekhter,
+              "Multiprotocol Extensions for BGP-4", RFC 4760,
+              DOI 10.17487/RFC4760, January 2007,
+              <https://www.rfc-editor.org/info/rfc4760>.
+
+   [RFC5925]  Touch, J., Mankin, A., and R. Bonica, "The TCP
+              Authentication Option", RFC 5925, DOI 10.17487/RFC5925,
+              June 2010, <https://www.rfc-editor.org/info/rfc5925>.
+
+   [RFC6952]  Jethanandani, M., Patel, K., and L. Zheng, "Analysis of
+              BGP, LDP, PCEP, and MSDP Issues According to the Keying
+              and Authentication for Routing Protocols (KARP) Design
+              Guide", RFC 6952, DOI 10.17487/RFC6952, May 2013,
+              <https://www.rfc-editor.org/info/rfc6952>.
+
+   [RFC7348]  Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
+              L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
+              eXtensible Local Area Network (VXLAN): A Framework for
+              Overlaying Virtualized Layer 2 Networks over Layer 3
+              Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
+              <https://www.rfc-editor.org/info/rfc7348>.
+
+   [RFC7432BIS]
+              Sajassi, A., Burdet, L., Drake, J., and J. Rabadan, "BGP
+              MPLS-Based Ethernet VPN", Work in Progress, Internet-
+              Draft, draft-ietf-bess-rfc7432bis-07, 13 March 2023,
+              <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
+              rfc7432bis-07>.
+
+   [RFC7510]  Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black,
+              "Encapsulating MPLS in UDP", RFC 7510,
+              DOI 10.17487/RFC7510, April 2015,
+              <https://www.rfc-editor.org/info/rfc7510>.
+
+   [RFC8365]  Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
+              Uttaro, J., and W. Henderickx, "A Network Virtualization
+              Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
+              DOI 10.17487/RFC8365, March 2018,
+              <https://www.rfc-editor.org/info/rfc8365>.
+
+   [RFC8584]  Rabadan, J., Ed., Mohanty, S., Ed., Sajassi, A., Drake,
+              J., Nagaraj, K., and S. Sathappan, "Framework for Ethernet
+              VPN Designated Forwarder Election Extensibility",
+              RFC 8584, DOI 10.17487/RFC8584, April 2019,
+              <https://www.rfc-editor.org/info/rfc8584>.
+
+   [RFC8926]  Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed.,
+              "Geneve: Generic Network Virtualization Encapsulation",
+              RFC 8926, DOI 10.17487/RFC8926, November 2020,
+              <https://www.rfc-editor.org/info/rfc8926>.
+
+   [RFC9012]  Patel, K., Van de Velde, G., Sangli, S., and J. Scudder,
+              "The BGP Tunnel Encapsulation Attribute", RFC 9012,
+              DOI 10.17487/RFC9012, April 2021,
+              <https://www.rfc-editor.org/info/rfc9012>.
+
+   [RFC9014]  Rabadan, J., Ed., Sathappan, S., Henderickx, W., Sajassi,
+              A., and J. Drake, "Interconnect Solution for Ethernet VPN
+              (EVPN) Overlay Networks", RFC 9014, DOI 10.17487/RFC9014,
+              May 2021, <https://www.rfc-editor.org/info/rfc9014>.
+
+   [RFC9135]  Sajassi, A., Salam, S., Thoria, S., Drake, J., and J.
+              Rabadan, "Integrated Routing and Bridging in Ethernet VPN
+              (EVPN)", RFC 9135, DOI 10.17487/RFC9135, October 2021,
+              <https://www.rfc-editor.org/info/rfc9135>.
+
+   [RFC9136]  Rabadan, J., Ed., Henderickx, W., Drake, J., Lin, W., and
+              A. Sajassi, "IP Prefix Advertisement in Ethernet VPN
+              (EVPN)", RFC 9136, DOI 10.17487/RFC9136, October 2021,
+              <https://www.rfc-editor.org/info/rfc9136>.
+
+   [RFC9161]  Rabadan, J., Ed., Sathappan, S., Nagaraj, K., Hankins, G.,
+              and T. King, "Operational Aspects of Proxy ARP/ND in
+              Ethernet Virtual Private Networks", RFC 9161,
+              DOI 10.17487/RFC9161, January 2022,
+              <https://www.rfc-editor.org/info/rfc9161>.
+
+   [RFC9251]  Sajassi, A., Thoria, S., Mishra, M., Patel, K., Drake, J.,
+              and W. Lin, "Internet Group Management Protocol (IGMP) and
+              Multicast Listener Discovery (MLD) Proxies for Ethernet
+              VPN (EVPN)", RFC 9251, DOI 10.17487/RFC9251, June 2022,
+              <https://www.rfc-editor.org/info/rfc9251>.
+
+   [SECURE-EVPN]
+              Sajassi, A., Banerjee, A., Thoria, S., Carrel, D., Weis,
+              B., and J. Drake, "Secure EVPN", Work in Progress,
+              Internet-Draft, draft-ietf-bess-secure-evpn-00, 20 June
+              2023, <https://datatracker.ietf.org/doc/html/draft-ietf-
+              bess-secure-evpn-00>.
+
+Acknowledgments
+
+   The authors thank Aldrin Isaac for his comments.
+
+Authors' Addresses
+
+   Jorge Rabadan (editor)
+   Nokia
+   520 Almanor Ave
+   Sunnyvale, CA 94085
+   United States of America
+   Email: jorge.rabadan@nokia.com
+
+
+   Matthew Bocci
+   Nokia
+   Email: matthew.bocci@nokia.com
+
+
+   Sami Boutros
+   Ciena
+   Email: sboutros@ciena.com
+
+
+   Ali Sajassi
+   Cisco Systems, Inc.
+   Email: sajassi@cisco.com