diff options
Diffstat (limited to 'doc/rfc/rfc8014.txt')
| -rw-r--r-- | doc/rfc/rfc8014.txt | 1963 |
1 files changed, 1963 insertions, 0 deletions
diff --git a/doc/rfc/rfc8014.txt b/doc/rfc/rfc8014.txt new file mode 100644 index 0000000..aa89ee2 --- /dev/null +++ b/doc/rfc/rfc8014.txt @@ -0,0 +1,1963 @@ + + + + + + +Internet Engineering Task Force (IETF) D. Black +Request for Comments: 8014 Dell EMC +Category: Informational J. Hudson +ISSN: 2070-1721 L. Kreeger + M. Lasserre + Independent + T. Narten + IBM + December 2016 + + + An Architecture for + Data-Center Network Virtualization over Layer 3 (NVO3) + +Abstract + + This document presents a high-level overview architecture for + building data-center Network Virtualization over Layer 3 (NVO3) + networks. The architecture is given at a high level, showing the + major components of an overall system. An important goal is to + divide the space into individual smaller components that can be + implemented independently with clear inter-component interfaces and + interactions. It should be possible to build and implement + individual components in isolation and have them interoperate with + other independently implemented components. That way, implementers + have flexibility in implementing individual components and can + optimize and innovate within their respective components without + requiring changes to other components. + +Status of This Memo + + This document is not an Internet Standards Track specification; it is + published for informational purposes. + + This document is a product of the Internet Engineering Task Force + (IETF). It represents the consensus of the IETF community. It has + received public review and has been approved for publication by the + Internet Engineering Steering Group (IESG). Not all documents + approved by the IESG are a candidate for any level of Internet + Standard; see Section 2 of RFC 7841. + + Information about the current status of this document, any errata, + and how to provide feedback on it may be obtained at + http://www.rfc-editor.org/info/rfc8014. + + + + + + + +Black, et al. Informational [Page 1] + +RFC 8014 NVO3 Architecture December 2016 + + +Copyright Notice + + Copyright (c) 2016 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (http://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the Simplified BSD License. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Black, et al. Informational [Page 2] + +RFC 8014 NVO3 Architecture December 2016 + + +Table of Contents + + 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 + 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 + 3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5 + 3.1. VN Service (L2 and L3) . . . . . . . . . . . . . . . . . 7 + 3.1.1. VLAN Tags in L2 Service . . . . . . . . . . . . . . . 8 + 3.1.2. Packet Lifetime Considerations . . . . . . . . . . . 8 + 3.2. Network Virtualization Edge (NVE) Background . . . . . . 9 + 3.3. Network Virtualization Authority (NVA) Background . . . . 10 + 3.4. VM Orchestration Systems . . . . . . . . . . . . . . . . 11 + 4. Network Virtualization Edge (NVE) . . . . . . . . . . . . . . 12 + 4.1. NVE Co-located with Server Hypervisor . . . . . . . . . . 12 + 4.2. Split-NVE . . . . . . . . . . . . . . . . . . . . . . . . 13 + 4.2.1. Tenant VLAN Handling in Split-NVE Case . . . . . . . 14 + 4.3. NVE State . . . . . . . . . . . . . . . . . . . . . . . . 14 + 4.4. Multihoming of NVEs . . . . . . . . . . . . . . . . . . . 15 + 4.5. Virtual Access Point (VAP) . . . . . . . . . . . . . . . 16 + 5. Tenant System Types . . . . . . . . . . . . . . . . . . . . . 16 + 5.1. Overlay-Aware Network Service Appliances . . . . . . . . 16 + 5.2. Bare Metal Servers . . . . . . . . . . . . . . . . . . . 17 + 5.3. Gateways . . . . . . . . . . . . . . . . . . . . . . . . 17 + 5.3.1. Gateway Taxonomy . . . . . . . . . . . . . . . . . . 18 + 5.3.1.1. L2 Gateways (Bridging) . . . . . . . . . . . . . 18 + 5.3.1.2. L3 Gateways (Only IP Packets) . . . . . . . . . . 18 + 5.4. Distributed Inter-VN Gateways . . . . . . . . . . . . . . 19 + 5.5. ARP and Neighbor Discovery . . . . . . . . . . . . . . . 20 + 6. NVE-NVE Interaction . . . . . . . . . . . . . . . . . . . . . 20 + 7. Network Virtualization Authority (NVA) . . . . . . . . . . . 21 + 7.1. How an NVA Obtains Information . . . . . . . . . . . . . 21 + 7.2. Internal NVA Architecture . . . . . . . . . . . . . . . . 22 + 7.3. NVA External Interface . . . . . . . . . . . . . . . . . 22 + 8. NVE-NVA Protocol . . . . . . . . . . . . . . . . . . . . . . 24 + 8.1. NVE-NVA Interaction Models . . . . . . . . . . . . . . . 24 + 8.2. Direct NVE-NVA Protocol . . . . . . . . . . . . . . . . . 25 + 8.3. Propagating Information Between NVEs and NVAs . . . . . . 25 + 9. Federated NVAs . . . . . . . . . . . . . . . . . . . . . . . 26 + 9.1. Inter-NVA Peering . . . . . . . . . . . . . . . . . . . . 29 + 10. Control Protocol Work Areas . . . . . . . . . . . . . . . . . 29 + 11. NVO3 Data-Plane Encapsulation . . . . . . . . . . . . . . . . 29 + 12. Operations, Administration, and Maintenance (OAM) . . . . . . 30 + 13. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 + 14. Security Considerations . . . . . . . . . . . . . . . . . . . 31 + 15. Informative References . . . . . . . . . . . . . . . . . . . 32 + Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 34 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 35 + + + + + +Black, et al. Informational [Page 3] + +RFC 8014 NVO3 Architecture December 2016 + + +1. Introduction + + This document presents a high-level architecture for building data- + center Network Virtualization over Layer 3 (NVO3) networks. The + architecture is given at a high level, which shows the major + components of an overall system. An important goal is to divide the + space into smaller individual components that can be implemented + independently with clear inter-component interfaces and interactions. + It should be possible to build and implement individual components in + isolation and have them interoperate with other independently + implemented components. That way, implementers have flexibility in + implementing individual components and can optimize and innovate + within their respective components without requiring changes to other + components. + + The motivation for overlay networks is given in "Problem Statement: + Overlays for Network Virtualization" [RFC7364]. "Framework for Data + Center (DC) Network Virtualization" [RFC7365] provides a framework + for discussing overlay networks generally and the various components + that must work together in building such systems. This document + differs from the framework document in that it doesn't attempt to + cover all possible approaches within the general design space. + Rather, it describes one particular approach that the NVO3 WG has + focused on. + +2. Terminology + + This document uses the same terminology as [RFC7365]. In addition, + the following terms are used: + + NV Domain: A Network Virtualization Domain is an administrative + construct that defines a Network Virtualization Authority (NVA), + the set of Network Virtualization Edges (NVEs) associated with + that NVA, and the set of virtual networks the NVA manages and + supports. NVEs are associated with a (logically centralized) NVA, + and an NVE supports communication for any of the virtual networks + in the domain. + + NV Region: A region over which information about a set of virtual + networks is shared. The degenerate case of a single NV Domain + corresponds to an NV Region corresponding to that domain. The + more interesting case occurs when two or more NV Domains share + information about part or all of a set of virtual networks that + they manage. Two NVAs share information about particular virtual + networks for the purpose of supporting connectivity between + tenants located in different NV Domains. NVAs can share + information about an entire NV Domain, or just individual virtual + networks. + + + +Black, et al. Informational [Page 4] + +RFC 8014 NVO3 Architecture December 2016 + + + Tenant System Interface (TSI): The interface to a Virtual Network + (VN) as presented to a Tenant System (TS, see [RFC7365]). The TSI + logically connects to the NVE via a Virtual Access Point (VAP). + To the Tenant System, the TSI is like a Network Interface Card + (NIC); the TSI presents itself to a Tenant System as a normal + network interface. + + VLAN: Unless stated otherwise, the terms "VLAN" and "VLAN Tag" are + used in this document to denote a Customer VLAN (C-VLAN) + [IEEE.802.1Q]; the terms are used interchangeably to improve + readability. + +3. Background + + Overlay networks are an approach for providing network virtualization + services to a set of Tenant Systems (TSs) [RFC7365]. With overlays, + data traffic between tenants is tunneled across the underlying data + center's IP network. The use of tunnels provides a number of + benefits by decoupling the network as viewed by tenants from the + underlying physical network across which they communicate. + Additional discussion of some NVO3 use cases can be found in + [USECASES]. + + Tenant Systems connect to Virtual Networks (VNs), with each VN having + associated attributes defining properties of the network (such as the + set of members that connect to it). Tenant Systems connected to a + virtual network typically communicate freely with other Tenant + Systems on the same VN, but communication between Tenant Systems on + one VN and those external to the VN (whether on another VN or + connected to the Internet) is carefully controlled and governed by + policy. The NVO3 architecture does not impose any restrictions to + the application of policy controls even within a VN. + + A Network Virtualization Edge (NVE) [RFC7365] is the entity that + implements the overlay functionality. An NVE resides at the boundary + between a Tenant System and the overlay network as shown in Figure 1. + An NVE creates and maintains local state about each VN for which it + is providing service on behalf of a Tenant System. + + + + + + + + + + + + + +Black, et al. Informational [Page 5] + +RFC 8014 NVO3 Architecture December 2016 + + + +--------+ +--------+ + | Tenant +--+ +----| Tenant | + | System | | (') | System | + +--------+ | ................ ( ) +--------+ + | +-+--+ . . +--+-+ (_) + | | NVE|--. .--| NVE| | + +--| | . . | |---+ + +-+--+ . . +--+-+ + / . . + / . L3 Overlay . +--+-++--------+ + +--------+ / . Network . | NVE|| Tenant | + | Tenant +--+ . .- -| || System | + | System | . . +--+-++--------+ + +--------+ ................ + | + +----+ + | NVE| + | | + +----+ + | + | + ===================== + | | + +--------+ +--------+ + | Tenant | | Tenant | + | System | | System | + +--------+ +--------+ + + + Figure 1: NVO3 Generic Reference Model + + The following subsections describe key aspects of an overlay system + in more detail. Section 3.1 describes the service model (Ethernet + vs. IP) provided to Tenant Systems. Section 3.2 describes NVEs in + more detail. Section 3.3 introduces the Network Virtualization + Authority, from which NVEs obtain information about virtual networks. + Section 3.4 provides background on Virtual Machine (VM) orchestration + systems and their use of virtual networks. + + + + + + + + + + + + + +Black, et al. Informational [Page 6] + +RFC 8014 NVO3 Architecture December 2016 + + +3.1. VN Service (L2 and L3) + + A VN provides either Layer 2 (L2) or Layer 3 (L3) service to + connected tenants. For L2 service, VNs transport Ethernet frames, + and a Tenant System is provided with a service that is analogous to + being connected to a specific L2 C-VLAN. L2 broadcast frames are + generally delivered to all (and multicast frames delivered to a + subset of) the other Tenant Systems on the VN. To a Tenant System, + it appears as if they are connected to a regular L2 Ethernet link. + Within the NVO3 architecture, tenant frames are tunneled to remote + NVEs based on the Media Access Control (MAC) addresses of the frame + headers as originated by the Tenant System. On the underlay, NVO3 + packets are forwarded between NVEs based on the outer addresses of + tunneled packets. + + For L3 service, VNs are routed networks that transport IP datagrams, + and a Tenant System is provided with a service that supports only IP + traffic. Within the NVO3 architecture, tenant frames are tunneled to + remote NVEs based on the IP addresses of the packet originated by the + Tenant System; any L2 destination addresses provided by Tenant + Systems are effectively ignored by the NVEs and overlay network. For + L3 service, the Tenant System will be configured with an IP subnet + that is effectively a point-to-point link, i.e., having only the + Tenant System and a next-hop router address on it. + + L2 service is intended for systems that need native L2 Ethernet + service and the ability to run protocols directly over Ethernet + (i.e., not based on IP). L3 service is intended for systems in which + all the traffic can safely be assumed to be IP. It is important to + note that whether or not an NVO3 network provides L2 or L3 service to + a Tenant System, the Tenant System does not generally need to be + aware of the distinction. In both cases, the virtual network + presents itself to the Tenant System as an L2 Ethernet interface. An + Ethernet interface is used in both cases simply as a widely supported + interface type that essentially all Tenant Systems already support. + Consequently, no special software is needed on Tenant Systems to use + an L3 vs. an L2 overlay service. + + NVO3 can also provide a combined L2 and L3 service to tenants. A + combined service provides L2 service for intra-VN communication but + also provides L3 service for L3 traffic entering or leaving the VN. + Architecturally, the handling of a combined L2/L3 service within the + NVO3 architecture is intended to match what is commonly done today in + non-overlay environments by devices providing a combined bridge/ + router service. With combined service, the virtual network itself + retains the semantics of L2 service, and all traffic is processed + according to its L2 semantics. In addition, however, traffic + requiring IP processing is also processed at the IP level. + + + +Black, et al. Informational [Page 7] + +RFC 8014 NVO3 Architecture December 2016 + + + The IP processing for a combined service can be implemented on a + standalone device attached to the virtual network (e.g., an IP + router) or implemented locally on the NVE (see Section 5.4 on + Distributed Inter-VN Gateways). For unicast traffic, NVE + implementation of a combined service may result in a packet being + delivered to another Tenant System attached to the same NVE (on + either the same or a different VN), tunneled to a remote NVE, or even + forwarded outside the NV Domain. For multicast or broadcast packets, + the combination of NVE L2 and L3 processing may result in copies of + the packet receiving both L2 and L3 treatments to realize delivery to + all of the destinations involved. This distributed NVE + implementation of IP routing results in the same network delivery + behavior as if the L2 processing of the packet included delivery of + the packet to an IP router attached to the L2 VN as a Tenant System, + with the router having additional network attachments to other + networks, either virtual or not. + +3.1.1. VLAN Tags in L2 Service + + An NVO3 L2 virtual network service may include encapsulated L2 VLAN + tags provided by a Tenant System but does not use encapsulated tags + in deciding where and how to forward traffic. Such VLAN tags can be + passed through so that Tenant Systems that send or expect to receive + them can be supported as appropriate. + + The processing of VLAN tags that an NVE receives from a TS is + controlled by settings associated with the VAP. Just as in the case + with ports on Ethernet switches, a number of settings are possible. + For example, Customer VLAN Tags (C-TAGs) can be passed through + transparently, could always be stripped upon receipt from a Tenant + System, could be compared against a list of explicitly configured + tags, etc. + + Note that there are additional considerations when VLAN tags are used + to identify both the VN and a Tenant System VLAN within that VN, as + described in Section 4.2.1. + +3.1.2. Packet Lifetime Considerations + + For L3 service, Tenant Systems should expect the IPv4 Time to Live + (TTL) or IPv6 Hop Limit in the packets they send to be decremented by + at least 1. For L2 service, neither the TTL nor the Hop Limit (when + the packet is IP) is modified. The underlay network manages TTLs and + Hop Limits in the outer IP encapsulation -- the values in these + fields could be independent from or related to the values in the same + fields of tenant IP packets. + + + + + +Black, et al. Informational [Page 8] + +RFC 8014 NVO3 Architecture December 2016 + + +3.2. Network Virtualization Edge (NVE) Background + + Tenant Systems connect to NVEs via a Tenant System Interface (TSI). + The TSI logically connects to the NVE via a Virtual Access Point + (VAP), and each VAP is associated with one VN as shown in Figure 2. + To the Tenant System, the TSI is like a NIC; the TSI presents itself + to a Tenant System as a normal network interface. On the NVE side, a + VAP is a logical network port (virtual or physical) into a specific + virtual network. Note that two different Tenant Systems (and TSIs) + attached to a common NVE can share a VAP (e.g., TS1 and TS2 in + Figure 2) so long as they connect to the same VN. + + | Data-Center Network (IP) | + | | + +-----------------------------------------+ + | | + | Tunnel Overlay | + +------------+---------+ +---------+------------+ + | +----------+-------+ | | +-------+----------+ | + | | Overlay Module | | | | Overlay Module | | + | +---------+--------+ | | +---------+--------+ | + | | | | | | + NVE1 | | | | | | NVE2 + | +--------+-------+ | | +--------+-------+ | + | | VNI1 VNI2 | | | | VNI1 VNI2 | | + | +-+----------+---+ | | +-+-----------+--+ | + | | VAP1 | VAP2 | | | VAP1 | VAP2| + +----+----------+------+ +----+-----------+-----+ + | | | | + |\ | | | + | \ | | /| + -------+--\-------+-------------------+---------/-+------- + | \ | Tenant | / | + TSI1 |TSI2\ | TSI3 TSI1 TSI2/ TSI3 + +---+ +---+ +---+ +---+ +---+ +---+ + |TS1| |TS2| |TS3| |TS4| |TS5| |TS6| + +---+ +---+ +---+ +---+ +---+ +---+ + + Figure 2: NVE Reference Model + + The Overlay Module performs the actual encapsulation and + decapsulation of tunneled packets. The NVE maintains state about the + virtual networks it is a part of so that it can provide the Overlay + Module with information such as the destination address of the NVE to + tunnel a packet to and the Context ID that should be placed in the + encapsulation header to identify the virtual network that a tunneled + packet belongs to. + + + + +Black, et al. Informational [Page 9] + +RFC 8014 NVO3 Architecture December 2016 + + + On the side facing the data-center network, the NVE sends and + receives native IP traffic. When ingressing traffic from a Tenant + System, the NVE identifies the egress NVE to which the packet should + be sent, adds an overlay encapsulation header, and sends the packet + on the underlay network. When receiving traffic from a remote NVE, + an NVE strips off the encapsulation header and delivers the + (original) packet to the appropriate Tenant System. When the source + and destination Tenant System are on the same NVE, no encapsulation + is needed and the NVE forwards traffic directly. + + Conceptually, the NVE is a single entity implementing the NVO3 + functionality. In practice, there are a number of different + implementation scenarios, as described in detail in Section 4. + +3.3. Network Virtualization Authority (NVA) Background + + Address dissemination refers to the process of learning, building, + and distributing the mapping/forwarding information that NVEs need in + order to tunnel traffic to each other on behalf of communicating + Tenant Systems. For example, in order to send traffic to a remote + Tenant System, the sending NVE must know the destination NVE for that + Tenant System. + + One way to build and maintain mapping tables is to use learning, as + 802.1 bridges do [IEEE.802.1Q]. When forwarding traffic to multicast + or unknown unicast destinations, an NVE could simply flood traffic. + While flooding works, it can lead to traffic hot spots and to + problems in larger networks (e.g., excessive amounts of flooded + traffic). + + Alternatively, to reduce the scope of where flooding must take place, + or to eliminate it all together, NVEs can make use of a Network + Virtualization Authority (NVA). An NVA is the entity that provides + address mapping and other information to NVEs. NVEs interact with an + NVA to obtain any required address-mapping information they need in + order to properly forward traffic on behalf of tenants. The term + "NVA" refers to the overall system, without regard to its scope or + how it is implemented. NVAs provide a service, and NVEs access that + service via an NVE-NVA protocol as discussed in Section 8. + + Even when an NVA is present, Ethernet bridge MAC address learning + could be used as a fallback mechanism, should the NVA be unable to + provide an answer or for other reasons. This document does not + consider flooding approaches in detail, as there are a number of + benefits in using an approach that depends on the presence of an NVA. + + For the rest of this document, it is assumed that an NVA exists and + will be used. NVAs are discussed in more detail in Section 7. + + + +Black, et al. Informational [Page 10] + +RFC 8014 NVO3 Architecture December 2016 + + +3.4. VM Orchestration Systems + + VM orchestration systems manage server virtualization across a set of + servers. Although VM management is a separate topic from network + virtualization, the two areas are closely related. Managing the + creation, placement, and movement of VMs also involves creating, + attaching to, and detaching from virtual networks. A number of + existing VM orchestration systems have incorporated aspects of + virtual network management into their systems. + + Note also that although this section uses the terms "VM" and + "hypervisor" throughout, the same issues apply to other + virtualization approaches, including Linux Containers (LXC), BSD + Jails, Network Service Appliances as discussed in Section 5.1, etc. + From an NVO3 perspective, it should be assumed that where the + document uses the term "VM" and "hypervisor", the intention is that + the discussion also applies to other systems, where, e.g., the host + operating system plays the role of the hypervisor in supporting + virtualization, and a container plays the equivalent role as a VM. + + When a new VM image is started, the VM orchestration system + determines where the VM should be placed, interacts with the + hypervisor on the target server to load and start the VM, and + controls when a VM should be shut down or migrated elsewhere. VM + orchestration systems also have knowledge about how a VM should + connect to a network, possibly including the name of the virtual + network to which a VM is to connect. The VM orchestration system can + pass such information to the hypervisor when a VM is instantiated. + VM orchestration systems have significant (and sometimes global) + knowledge over the domain they manage. They typically know on what + servers a VM is running, and metadata associated with VM images can + be useful from a network virtualization perspective. For example, + the metadata may include the addresses (MAC and IP) the VMs will use + and the name(s) of the virtual network(s) they connect to. + + VM orchestration systems run a protocol with an agent running on the + hypervisor of the servers they manage. That protocol can also carry + information about what virtual network a VM is associated with. When + the orchestrator instantiates a VM on a hypervisor, the hypervisor + interacts with the NVE in order to attach the VM to the virtual + networks it has access to. In general, the hypervisor will need to + communicate significant VM state changes to the NVE. In the reverse + direction, the NVE may need to communicate network connectivity + information back to the hypervisor. Examples of deployed VM + orchestration systems include VMware's vCenter Server, Microsoft's + System Center Virtual Machine Manager, and systems based on OpenStack + and its associated plugins (e.g., Nova and Neutron). Each can pass + information about what virtual networks a VM connects to down to the + + + +Black, et al. Informational [Page 11] + +RFC 8014 NVO3 Architecture December 2016 + + + hypervisor. The protocol used between the VM orchestration system + and hypervisors is generally proprietary. + + It should be noted that VM orchestration systems may not have direct + access to all networking-related information a VM uses. For example, + a VM may make use of additional IP or MAC addresses that the VM + management system is not aware of. + +4. Network Virtualization Edge (NVE) + + As introduced in Section 3.2, an NVE is the entity that implements + the overlay functionality. This section describes NVEs in more + detail. An NVE will have two external interfaces: + + Facing the Tenant System: On the side facing the Tenant System, an + NVE interacts with the hypervisor (or equivalent entity) to + provide the NVO3 service. An NVE will need to be notified when a + Tenant System "attaches" to a virtual network (so it can validate + the request and set up any state needed to send and receive + traffic on behalf of the Tenant System on that VN). Likewise, an + NVE will need to be informed when the Tenant System "detaches" + from the virtual network so that it can reclaim state and + resources appropriately. + + Facing the Data-Center Network: On the side facing the data-center + network, an NVE interfaces with the data-center underlay network, + sending and receiving tunneled packets to and from the underlay. + The NVE may also run a control protocol with other entities on the + network, such as the Network Virtualization Authority. + +4.1. NVE Co-located with Server Hypervisor + + When server virtualization is used, the entire NVE functionality will + typically be implemented as part of the hypervisor and/or virtual + switch on the server. In such cases, the Tenant System interacts + with the hypervisor, and the hypervisor interacts with the NVE. + Because the interaction between the hypervisor and NVE is implemented + entirely in software on the server, there is no "on-the-wire" + protocol between Tenant Systems (or the hypervisor) and the NVE that + needs to be standardized. While there may be APIs between the NVE + and hypervisor to support necessary interaction, the details of such + APIs are not in scope for the NVO3 WG at the time of publication of + this memo. + + Implementing NVE functionality entirely on a server has the + disadvantage that server CPU resources must be spent implementing the + NVO3 functionality. Experimentation with overlay approaches and + previous experience with TCP and checksum adapter offloads suggest + + + +Black, et al. Informational [Page 12] + +RFC 8014 NVO3 Architecture December 2016 + + + that offloading certain NVE operations (e.g., encapsulation and + decapsulation operations) onto the physical network adapter can + produce performance advantages. As has been done with checksum and/ + or TCP server offload and other optimization approaches, there may be + benefits to offloading common operations onto adapters where + possible. Just as important, the addition of an overlay header can + disable existing adapter offload capabilities that are generally not + prepared to handle the addition of a new header or other operations + associated with an NVE. + + While the exact details of how to split the implementation of + specific NVE functionality between a server and its network adapters + are an implementation matter and outside the scope of IETF + standardization, the NVO3 architecture should be cognizant of and + support such separation. Ideally, it may even be possible to bypass + the hypervisor completely on critical data-path operations so that + packets between a Tenant System and its VN can be sent and received + without having the hypervisor involved in each individual packet + operation. + +4.2. Split-NVE + + Another possible scenario leads to the need for a split-NVE + implementation. An NVE running on a server (e.g., within a + hypervisor) could support NVO3 service towards the tenant but not + perform all NVE functions (e.g., encapsulation) directly on the + server; some of the actual NVO3 functionality could be implemented on + (i.e., offloaded to) an adjacent switch to which the server is + attached. While one could imagine a number of link types between a + server and the NVE, one simple deployment scenario would involve a + server and NVE separated by a simple L2 Ethernet link. A more + complicated scenario would have the server and NVE separated by a + bridged access network, such as when the NVE resides on a Top of Rack + (ToR) switch, with an embedded switch residing between servers and + the ToR switch. + + For the split-NVE case, protocols will be needed that allow the + hypervisor and NVE to negotiate and set up the necessary state so + that traffic sent across the access link between a server and the NVE + can be associated with the correct virtual network instance. + Specifically, on the access link, traffic belonging to a specific + Tenant System would be tagged with a specific VLAN C-TAG that + identifies which specific NVO3 virtual network instance it connects + to. The hypervisor-NVE protocol would negotiate which VLAN C-TAG to + use for a particular virtual network instance. More details of the + protocol requirements for functionality between hypervisors and NVEs + can be found in [NVE-NVA]. + + + + +Black, et al. Informational [Page 13] + +RFC 8014 NVO3 Architecture December 2016 + + +4.2.1. Tenant VLAN Handling in Split-NVE Case + + Preserving tenant VLAN tags across an NVO3 VN, as described in + Section 3.1.1, poses additional complications in the split-NVE case. + The portion of the NVE that performs the encapsulation function needs + access to the specific VLAN tags that the Tenant System is using in + order to include them in the encapsulated packet. When an NVE is + implemented entirely within the hypervisor, the NVE has access to the + complete original packet (including any VLAN tags) sent by the + tenant. In the split-NVE case, however, the VLAN tag used between + the hypervisor and offloaded portions of the NVE normally only + identifies the specific VN that traffic belongs to. In order to + allow a tenant to preserve VLAN information from end to end between + Tenant Systems in the split-NVE case, additional mechanisms would be + needed (e.g., carry an additional VLAN tag by carrying both a C-TAG + and a Service VLAN Tag (S-TAG) as specified in [IEEE.802.1Q] where + the C-TAG identifies the tenant VLAN end to end and the S-TAG + identifies the VN locally between each Tenant System and the + corresponding NVE). + +4.3. NVE State + + NVEs maintain internal data structures and state to support the + sending and receiving of tenant traffic. An NVE may need some or all + of the following information: + + 1. An NVE keeps track of which attached Tenant Systems are connected + to which virtual networks. When a Tenant System attaches to a + virtual network, the NVE will need to create or update the local + state for that virtual network. When the last Tenant System + detaches from a given VN, the NVE can reclaim state associated + with that VN. + + 2. For tenant unicast traffic, an NVE maintains a per-VN table of + mappings from Tenant System (inner) addresses to remote NVE + (outer) addresses. + + 3. For tenant multicast (or broadcast) traffic, an NVE maintains a + per-VN table of mappings and other information on how to deliver + tenant multicast (or broadcast) traffic. If the underlying + network supports IP multicast, the NVE could use IP multicast to + deliver tenant traffic. In such a case, the NVE would need to + know what IP underlay multicast address to use for a given VN. + Alternatively, if the underlying network does not support + multicast, a source NVE could use unicast replication to deliver + traffic. In such a case, an NVE would need to know which remote + NVEs are participating in the VN. An NVE could use both + approaches, switching from one mode to the other depending on + + + +Black, et al. Informational [Page 14] + +RFC 8014 NVO3 Architecture December 2016 + + + factors such as bandwidth efficiency and group membership + sparseness. [FRAMEWORK-MCAST] discusses the subject of multicast + handling in NVO3 in further detail. + + 4. An NVE maintains necessary information to encapsulate outgoing + traffic, including what type of encapsulation and what value to + use for a Context ID to identify the VN within the encapsulation + header. + + 5. In order to deliver incoming encapsulated packets to the correct + Tenant Systems, an NVE maintains the necessary information to map + incoming traffic to the appropriate VAP (i.e., TSI). + + 6. An NVE may find it convenient to maintain additional per-VN + information such as QoS settings, Path MTU information, Access + Control Lists (ACLs), etc. + +4.4. Multihoming of NVEs + + NVEs may be multihomed. That is, an NVE may have more than one IP + address associated with it on the underlay network. Multihoming + happens in two different scenarios. First, an NVE may have multiple + interfaces connecting it to the underlay. Each of those interfaces + will typically have a different IP address, resulting in a specific + Tenant Address (on a specific VN) being reachable through the same + NVE but through more than one underlay IP address. Second, a + specific Tenant System may be reachable through more than one NVE, + each having one or more underlay addresses. In both cases, NVE + address-mapping functionality needs to support one-to-many mappings + and enable a sending NVE to (at a minimum) be able to fail over from + one IP address to another, e.g., should a specific NVE underlay + address become unreachable. + + Finally, multihomed NVEs introduce complexities when source unicast + replication is used to implement tenant multicast as described in + Section 4.3. Specifically, an NVE should only receive one copy of a + replicated packet. + + Multihoming is needed to support important use cases. First, a bare + metal server may have multiple uplink connections to either the same + or different NVEs. Having only a single physical path to an upstream + NVE, or indeed, having all traffic flow through a single NVE would be + considered unacceptable in highly resilient deployment scenarios that + seek to avoid single points of failure. Moreover, in today's + networks, the availability of multiple paths would require that they + be usable in an active-active fashion (e.g., for load balancing). + + + + + +Black, et al. Informational [Page 15] + +RFC 8014 NVO3 Architecture December 2016 + + +4.5. Virtual Access Point (VAP) + + The VAP is the NVE side of the interface between the NVE and the TS. + Traffic to and from the tenant flows through the VAP. If an NVE runs + into difficulties sending traffic received on the VAP, it may need to + signal such errors back to the VAP. Because the VAP is an emulation + of a physical port, its ability to signal NVE errors is limited and + lacks sufficient granularity to reflect all possible errors an NVE + may encounter (e.g., inability to reach a particular destination). + Some errors, such as an NVE losing all of its connections to the + underlay, could be reflected back to the VAP by effectively disabling + it. This state change would reflect itself on the TS as an interface + going down, allowing the TS to implement interface error handling + (e.g., failover) in the same manner as when a physical interface + becomes disabled. + +5. Tenant System Types + + This section describes a number of special Tenant System types and + how they fit into an NVO3 system. + +5.1. Overlay-Aware Network Service Appliances + + Some Network Service Appliances [NVE-NVA] (virtual or physical) + provide tenant-aware services. That is, the specific service they + provide depends on the identity of the tenant making use of the + service. For example, firewalls are now becoming available that + support multitenancy where a single firewall provides virtual + firewall service on a per-tenant basis, using per-tenant + configuration rules and maintaining per-tenant state. Such + appliances will be aware of the VN an activity corresponds to while + processing requests. Unlike server virtualization, which shields VMs + from needing to know about multitenancy, a Network Service Appliance + may explicitly support multitenancy. In such cases, the Network + Service Appliance itself will be aware of network virtualization and + either embed an NVE directly or implement a split-NVE as described in + Section 4.2. Unlike server virtualization, however, the Network + Service Appliance may not be running a hypervisor, and the VM + orchestration system may not interact with the Network Service + Appliance. The NVE on such appliances will need to support a control + plane to obtain the necessary information needed to fully participate + in an NV Domain. + + + + + + + + + +Black, et al. Informational [Page 16] + +RFC 8014 NVO3 Architecture December 2016 + + +5.2. Bare Metal Servers + + Many data centers will continue to have at least some servers + operating as non-virtualized (or "bare metal") machines running a + traditional operating system and workload. In such systems, there + will be no NVE functionality on the server, and the server will have + no knowledge of NVO3 (including whether overlays are even in use). + In such environments, the NVE functionality can reside on the first- + hop physical switch. In such a case, the network administrator would + (manually) configure the switch to enable the appropriate NVO3 + functionality on the switch port connecting the server and associate + that port with a specific virtual network. Such configuration would + typically be static, since the server is not virtualized and, once + configured, is unlikely to change frequently. Consequently, this + scenario does not require any protocol or standards work. + +5.3. Gateways + + Gateways on VNs relay traffic onto and off of a virtual network. + Tenant Systems use gateways to reach destinations outside of the + local VN. Gateways receive encapsulated traffic from one VN, remove + the encapsulation header, and send the native packet out onto the + data-center network for delivery. Outside traffic enters a VN in a + reverse manner. + + Gateways can be either virtual (i.e., implemented as a VM) or + physical (i.e., a standalone physical device). For performance + reasons, standalone hardware gateways may be desirable in some cases. + Such gateways could consist of a simple switch forwarding traffic + from a VN onto the local data-center network or could embed router + functionality. On such gateways, network interfaces connecting to + virtual networks will (at least conceptually) embed NVE (or split- + NVE) functionality within them. As in the case with Network Service + Appliances, gateways may not support a hypervisor and will need an + appropriate control-plane protocol to obtain the information needed + to provide NVO3 service. + + Gateways handle several different use cases. For example, one use + case consists of systems supporting overlays together with systems + that do not (e.g., bare metal servers). Gateways could be used to + connect legacy systems supporting, e.g., L2 VLANs, to specific + virtual networks, effectively making them part of the same virtual + network. Gateways could also forward traffic between a virtual + network and other hosts on the data-center network or relay traffic + between different VNs. Finally, gateways can provide external + connectivity such as Internet or VPN access. + + + + + +Black, et al. Informational [Page 17] + +RFC 8014 NVO3 Architecture December 2016 + + +5.3.1. Gateway Taxonomy + + As can be seen from the discussion above, there are several types of + gateways that can exist in an NVO3 environment. This section breaks + them down into the various types that could be supported. Note that + each of the types below could be either implemented in a centralized + manner or distributed to coexist with the NVEs. + +5.3.1.1. L2 Gateways (Bridging) + + L2 Gateways act as Layer 2 bridges to forward Ethernet frames based + on the MAC addresses present in them. + + L2 VN to Legacy L2: This type of gateway bridges traffic between L2 + VNs and other legacy L2 networks such as VLANs or L2 VPNs. + + L2 VN to L2 VN: The main motivation for this type of gateway is to + create separate groups of Tenant Systems using L2 VNs such that + the gateway can enforce network policies between each L2 VN. + +5.3.1.2. L3 Gateways (Only IP Packets) + + L3 Gateways forward IP packets based on the IP addresses present in + the packets. + + L3 VN to Legacy L2: This type of gateway forwards packets between L3 + VNs and legacy L2 networks such as VLANs or L2 VPNs. The original + sender's destination MAC address in any frames that the gateway + forwards from a legacy L2 network would be the MAC address of the + gateway. + + L3 VN to Legacy L3: This type of gateway forwards packets between L3 + VNs and legacy L3 networks. These legacy L3 networks could be + local to the data center, be in the WAN, or be an L3 VPN. + + L3 VN to L2 VN: This type of gateway forwards packets between L3 VNs + and L2 VNs. The original sender's destination MAC address in any + frames that the gateway forwards from a L2 VN would be the MAC + address of the gateway. + + L2 VN to L2 VN: This type of gateway acts similar to a traditional + router that forwards between L2 interfaces. The original sender's + destination MAC address in any frames that the gateway forwards + from any of the L2 VNs would be the MAC address of the gateway. + + L3 VN to L3 VN: The main motivation for this type of gateway is to + create separate groups of Tenant Systems using L3 VNs such that + the gateway can enforce network policies between each L3 VN. + + + +Black, et al. Informational [Page 18] + +RFC 8014 NVO3 Architecture December 2016 + + +5.4. Distributed Inter-VN Gateways + + The relaying of traffic from one VN to another deserves special + consideration. Whether traffic is permitted to flow from one VN to + another is a matter of policy and would not (by default) be allowed + unless explicitly enabled. In addition, NVAs are the logical place + to maintain policy information about allowed inter-VN communication. + Policy enforcement for inter-VN communication can be handled in (at + least) two different ways. Explicit gateways could be the central + point for such enforcement, with all inter-VN traffic forwarded to + such gateways for processing. Alternatively, the NVA can provide + such information directly to NVEs by either providing a mapping for a + target Tenant System (TS) on another VN or indicating that such + communication is disallowed by policy. + + When inter-VN gateways are centralized, traffic between TSs on + different VNs can take suboptimal paths, i.e., triangular routing + results in paths that always traverse the gateway. In the worst + case, traffic between two TSs connected to the same NVE can be hair- + pinned through an external gateway. As an optimization, individual + NVEs can be part of a distributed gateway that performs such + relaying, reducing or completely eliminating triangular routing. In + a distributed gateway, each ingress NVE can perform such relaying + activity directly so long as it has access to the policy information + needed to determine whether cross-VN communication is allowed. + Having individual NVEs be part of a distributed gateway allows them + to tunnel traffic directly to the destination NVE without the need to + take suboptimal paths. + + The NVO3 architecture supports distributed gateways for the case of + inter-VN communication. Such support requires that NVO3 control + protocols include mechanisms for the maintenance and distribution of + policy information about what type of cross-VN communication is + allowed so that NVEs acting as distributed gateways can tunnel + traffic from one VN to another as appropriate. + + Distributed gateways could also be used to distribute other + traditional router services to individual NVEs. The NVO3 + architecture does not preclude such implementations but does not + define or require them as they are outside the scope of the NVO3 + architecture. + + + + + + + + + + +Black, et al. Informational [Page 19] + +RFC 8014 NVO3 Architecture December 2016 + + +5.5. ARP and Neighbor Discovery + + Strictly speaking, for an L2 service, special processing of the + Address Resolution Protocol (ARP) [RFC826] and IPv6 Neighbor + Discovery (ND) [RFC4861] is not required. ARP requests are + broadcast, and an NVO3 can deliver ARP requests to all members of a + given L2 virtual network just as it does for any packet sent to an L2 + broadcast address. Similarly, ND requests are sent via IP multicast, + which NVO3 can support by delivering via L2 multicast. However, as a + performance optimization, an NVE can intercept ARP (or ND) requests + from its attached TSs and respond to them directly using information + in its mapping tables. Since an NVE will have mechanisms for + determining the NVE address associated with a given TS, the NVE can + leverage the same mechanisms to suppress sending ARP and ND requests + for a given TS to other members of the VN. The NVO3 architecture + supports such a capability. + +6. NVE-NVE Interaction + + Individual NVEs will interact with each other for the purposes of + tunneling and delivering traffic to remote TSs. At a minimum, a + control protocol may be needed for tunnel setup and maintenance. For + example, tunneled traffic may need to be encrypted or integrity + protected, in which case it will be necessary to set up appropriate + security associations between NVE peers. It may also be desirable to + perform tunnel maintenance (e.g., continuity checks) on a tunnel in + order to detect when a remote NVE becomes unreachable. Such generic + tunnel setup and maintenance functions are not generally + NVO3-specific. Hence, the NVO3 architecture expects to leverage + existing tunnel maintenance protocols rather than defining new ones. + + Some NVE-NVE interactions may be specific to NVO3 (in particular, be + related to information kept in mapping tables) and agnostic to the + specific tunnel type being used. For example, when tunneling traffic + for TS-X to a remote NVE, it is possible that TS-X is not presently + associated with the remote NVE. Normally, this should not happen, + but there could be race conditions where the information an NVE has + learned from the NVA is out of date relative to actual conditions. + In such cases, the remote NVE could return an error or warning + indication, allowing the sending NVE to attempt a recovery or + otherwise attempt to mitigate the situation. + + The NVE-NVE interaction could signal a range of indications, for + example: + + o "No such TS here", upon a receipt of a tunneled packet for an + unknown TS + + + + +Black, et al. Informational [Page 20] + +RFC 8014 NVO3 Architecture December 2016 + + + o "TS-X not here, try the following NVE instead" (i.e., a redirect) + + o "Delivered to correct NVE but could not deliver packet to TS-X" + + When an NVE receives information from a remote NVE that conflicts + with the information it has in its own mapping tables, it should + consult with the NVA to resolve those conflicts. In particular, it + should confirm that the information it has is up to date, and it + might indicate the error to the NVA so as to nudge the NVA into + following up (as appropriate). While it might make sense for an NVE + to update its mapping table temporarily in response to an error from + a remote NVE, any changes must be handled carefully as doing so can + raise security considerations if the received information cannot be + authenticated. That said, a sending NVE might still take steps to + mitigate a problem, such as applying rate limiting to data traffic + towards a particular NVE or TS. + +7. Network Virtualization Authority (NVA) + + Before sending traffic to and receiving traffic from a virtual + network, an NVE must obtain the information needed to build its + internal forwarding tables and state as listed in Section 4.3. An + NVE can obtain such information from a Network Virtualization + Authority (NVA). + + The NVA is the entity that is expected to provide address mapping and + other information to NVEs. NVEs can interact with an NVA to obtain + any required information they need in order to properly forward + traffic on behalf of tenants. The term "NVA" refers to the overall + system, without regard to its scope or how it is implemented. + +7.1. How an NVA Obtains Information + + There are two primary ways in which an NVA can obtain the address + dissemination information it manages: from the VM orchestration + system and/or directly from the NVEs themselves. + + On virtualized systems, the NVA may be able to obtain the address- + mapping information associated with VMs from the VM orchestration + system itself. If the VM orchestration system contains a master + database for all the virtualization information, having the NVA + obtain information directly from the orchestration system would be a + natural approach. Indeed, the NVA could effectively be co-located + with the VM orchestration system itself. In such systems, the VM + orchestration system communicates with the NVE indirectly through the + hypervisor. + + + + + +Black, et al. Informational [Page 21] + +RFC 8014 NVO3 Architecture December 2016 + + + However, as described in Section 4, not all NVEs are associated with + hypervisors. In such cases, NVAs cannot leverage VM orchestration + protocols to interact with an NVE and will instead need to peer + directly with them. By peering directly with an NVE, NVAs can obtain + information about the TSs connected to that NVE and can distribute + information to the NVE about the VNs those TSs are associated with. + For example, whenever a Tenant System attaches to an NVE, that NVE + would notify the NVA that the TS is now associated with that NVE. + Likewise, when a TS detaches from an NVE, that NVE would inform the + NVA. By communicating directly with NVEs, both the NVA and the NVE + are able to maintain up-to-date information about all active tenants + and the NVEs to which they are attached. + +7.2. Internal NVA Architecture + + For reliability and fault tolerance reasons, an NVA would be + implemented in a distributed or replicated manner without single + points of failure. How the NVA is implemented, however, is not + important to an NVE so long as the NVA provides a consistent and + well-defined interface to the NVE. For example, an NVA could be + implemented via database techniques whereby a server stores address- + mapping information in a traditional (possibly replicated) database. + Alternatively, an NVA could be implemented in a distributed fashion + using an existing (or modified) routing protocol to maintain and + distribute mappings. So long as there is a clear interface between + the NVE and NVA, how an NVA is architected and implemented is not + important to an NVE. + + A number of architectural approaches could be used to implement NVAs + themselves. NVAs manage address bindings and distribute them to + where they need to go. One approach would be to use the Border + Gateway Protocol (BGP) [RFC4364] (possibly with extensions) and route + reflectors. Another approach could use a transaction-based database + model with replicated servers. Because the implementation details + are local to an NVA, there is no need to pick exactly one solution + technology, so long as the external interfaces to the NVEs (and + remote NVAs) are sufficiently well defined to achieve + interoperability. + +7.3. NVA External Interface + + Conceptually, from the perspective of an NVE, an NVA is a single + entity. An NVE interacts with the NVA, and it is the NVA's + responsibility to ensure that interactions between the NVE and NVA + result in consistent behavior across the NVA and all other NVEs using + the same NVA. Because an NVA is built from multiple internal + components, an NVA will have to ensure that information flows to all + internal NVA components appropriately. + + + +Black, et al. Informational [Page 22] + +RFC 8014 NVO3 Architecture December 2016 + + + One architectural question is how the NVA presents itself to the NVE. + For example, an NVA could be required to provide access via a single + IP address. If NVEs only have one IP address to interact with, it + would be the responsibility of the NVA to handle NVA component + failures, e.g., by using a "floating IP address" that migrates among + NVA components to ensure that the NVA can always be reached via the + one address. Having all NVA accesses through a single IP address, + however, adds constraints to implementing robust failover, load + balancing, etc. + + In the NVO3 architecture, an NVA is accessed through one or more IP + addresses (or an IP address/port combination). If multiple IP + addresses are used, each IP address provides equivalent + functionality, meaning that an NVE can use any of the provided + addresses to interact with the NVA. Should one address stop working, + an NVE is expected to failover to another. While the different + addresses result in equivalent functionality, one address may respond + more quickly than another, e.g., due to network conditions, load on + the server, etc. + + To provide some control over load balancing, NVA addresses may have + an associated priority. Addresses are used in order of priority, + with no explicit preference among NVA addresses having the same + priority. To provide basic load balancing among NVAs of equal + priorities, NVEs could use some randomization input to select among + equal-priority NVAs. Such a priority scheme facilitates failover and + load balancing, for example, by allowing a network operator to + specify a set of primary and backup NVAs. + + It may be desirable to have individual NVA addresses responsible for + a subset of information about an NV Domain. In such a case, NVEs + would use different NVA addresses for obtaining or updating + information about particular VNs or TS bindings. Key questions with + such an approach are how information would be partitioned and how an + NVE could determine which address to use to get the information it + needs. + + Another possibility is to treat the information on which NVA + addresses to use as cached (soft-state) information at the NVEs, so + that any NVA address can be used to obtain any information, but NVEs + are informed of preferences for which addresses to use for particular + information on VNs or TS bindings. That preference information would + be cached for future use to improve behavior, e.g., if all requests + for a specific subset of VNs are forwarded to a specific NVA + component, the NVE can optimize future requests within that subset by + sending them directly to that NVA component via its address. + + + + + +Black, et al. Informational [Page 23] + +RFC 8014 NVO3 Architecture December 2016 + + +8. NVE-NVA Protocol + + As outlined in Section 4.3, an NVE needs certain information in order + to perform its functions. To obtain such information from an NVA, an + NVE-NVA protocol is needed. The NVE-NVA protocol provides two + functions. First, it allows an NVE to obtain information about the + location and status of other TSs with which it needs to communicate. + Second, the NVE-NVA protocol provides a way for NVEs to provide + updates to the NVA about the TSs attached to that NVE (e.g., when a + TS attaches or detaches from the NVE) or about communication errors + encountered when sending traffic to remote NVEs. For example, an NVE + could indicate that a destination it is trying to reach at a + destination NVE is unreachable for some reason. + + While having a direct NVE-NVA protocol might seem straightforward, + the existence of existing VM orchestration systems complicates the + choices an NVE has for interacting with the NVA. + +8.1. NVE-NVA Interaction Models + + An NVE interacts with an NVA in at least two (quite different) ways: + + o NVEs embedded within the same server as the hypervisor can obtain + necessary information entirely through the hypervisor-facing side + of the NVE. Such an approach is a natural extension to existing + VM orchestration systems supporting server virtualization because + an existing protocol between the hypervisor and VM orchestration + system already exists and can be leveraged to obtain any needed + information. Specifically, VM orchestration systems used to + create, terminate, and migrate VMs already use well-defined + (though typically proprietary) protocols to handle the + interactions between the hypervisor and VM orchestration system. + For such systems, it is a natural extension to leverage the + existing orchestration protocol as a sort of proxy protocol for + handling the interactions between an NVE and the NVA. Indeed, + existing implementations can already do this. + + o Alternatively, an NVE can obtain needed information by interacting + directly with an NVA via a protocol operating over the data-center + underlay network. Such an approach is needed to support NVEs that + are not associated with systems performing server virtualization + (e.g., as in the case of a standalone gateway) or where the NVE + needs to communicate directly with the NVA for other reasons. + + The NVO3 architecture will focus on support for the second model + above. Existing virtualization environments are already using the + first model, but they are not sufficient to cover the case of + + + + +Black, et al. Informational [Page 24] + +RFC 8014 NVO3 Architecture December 2016 + + + standalone gateways -- such gateways may not support virtualization + and do not interface with existing VM orchestration systems. + +8.2. Direct NVE-NVA Protocol + + An NVE can interact directly with an NVA via an NVE-NVA protocol. + Such a protocol can be either independent of the NVA internal + protocol or an extension of it. Using a purpose-specific protocol + would provide architectural separation and independence between the + NVE and NVA. The NVE and NVA interact in a well-defined way, and + changes in the NVA (or NVE) do not need to impact each other. Using + a dedicated protocol also ensures that both NVE and NVA + implementations can evolve independently and without dependencies on + each other. Such independence is important because the upgrade path + for NVEs and NVAs is quite different. Upgrading all the NVEs at a + site will likely be more difficult in practice than upgrading NVAs + because of their large number -- one on each end device. In + practice, it would be prudent to assume that once an NVE has been + implemented and deployed, it may be challenging to get subsequent NVE + extensions and changes implemented and deployed, whereas an NVA (and + its associated internal protocols) is more likely to evolve over time + as experience is gained from usage and upgrades will involve fewer + nodes. + + Requirements for a direct NVE-NVA protocol can be found in [NVE-NVA]. + +8.3. Propagating Information Between NVEs and NVAs + + Information flows between NVEs and NVAs in both directions. The NVA + maintains information about all VNs in the NV Domain so that NVEs do + not need to do so themselves. NVEs obtain information from the NVA + about where a given remote TS destination resides. NVAs, in turn, + obtain information from NVEs about the individual TSs attached to + those NVEs. + + While the NVA could push information relevant to every virtual + network to every NVE, such an approach scales poorly and is + unnecessary. In practice, a given NVE will only need and want to + know about VNs to which it is attached. Thus, an NVE should be able + to subscribe to updates only for the virtual networks it is + interested in receiving updates for. The NVO3 architecture supports + a model where an NVE is not required to have full mapping tables for + all virtual networks in an NV Domain. + + Before sending unicast traffic to a remote TS (or TSs for broadcast + or multicast traffic), an NVE must know where the remote TS(s) + currently reside. When a TS attaches to a virtual network, the NVE + obtains information about that VN from the NVA. The NVA can provide + + + +Black, et al. Informational [Page 25] + +RFC 8014 NVO3 Architecture December 2016 + + + that information to the NVE at the time the TS attaches to the VN, + either because the NVE requests the information when the attach + operation occurs or because the VM orchestration system has initiated + the attach operation and provides associated mapping information to + the NVE at the same time. + + There are scenarios where an NVE may wish to query the NVA about + individual mappings within a VN. For example, when sending traffic + to a remote TS on a remote NVE, that TS may become unavailable (e.g., + because it has migrated elsewhere or has been shut down, in which + case the remote NVE may return an error indication). In such + situations, the NVE may need to query the NVA to obtain updated + mapping information for a specific TS or to verify that the + information is still correct despite the error condition. Note that + such a query could also be used by the NVA as an indication that + there may be an inconsistency in the network and that it should take + steps to verify that the information it has about the current state + and location of a specific TS is still correct. + + For very large virtual networks, the amount of state an NVE needs to + maintain for a given virtual network could be significant. Moreover, + an NVE may only be communicating with a small subset of the TSs on + such a virtual network. In such cases, the NVE may find it desirable + to maintain state only for those destinations it is actively + communicating with. In such scenarios, an NVE may not want to + maintain full mapping information about all destinations on a VN. + However, if it needs to communicate with a destination for which it + does not have mapping information, it will need to be able to query + the NVA on demand for the missing information on a per-destination + basis. + + The NVO3 architecture will need to support a range of operations + between the NVE and NVA. Requirements for those operations can be + found in [NVE-NVA]. + +9. Federated NVAs + + An NVA provides service to the set of NVEs in its NV Domain. Each + NVA manages network virtualization information for the virtual + networks within its NV Domain. An NV Domain is administered by a + single entity. + + In some cases, it will be necessary to expand the scope of a specific + VN or even an entire NV Domain beyond a single NVA. For example, an + administrator managing multiple data centers may wish to operate all + of its data centers as a single NV Region. Such cases are handled by + having different NVAs peer with each other to exchange mapping + information about specific VNs. NVAs operate in a federated manner + + + +Black, et al. Informational [Page 26] + +RFC 8014 NVO3 Architecture December 2016 + + + with a set of NVAs operating as a loosely coupled federation of + individual NVAs. If a virtual network spans multiple NVAs (e.g., + located at different data centers), and an NVE needs to deliver + tenant traffic to an NVE that is part of a different NV Domain, it + still interacts only with its NVA, even when obtaining mappings for + NVEs associated with a different NV Domain. + + Figure 3 shows a scenario where two separate NV Domains (A and B) + share information about a VN. VM1 and VM2 both connect to the same + VN, even though the two VMs are in separate NV Domains. There are + two cases to consider. In the first case, NV Domain B does not allow + NVE-A to tunnel traffic directly to NVE-B. There could be a number + of reasons for this. For example, NV Domains A and B may not share a + common address space (i.e., traversal through a NAT device is + required), or for policy reasons, a domain might require that all + traffic between separate NV Domains be funneled through a particular + device (e.g., a firewall). In such cases, NVA-2 will advertise to + NVA-1 that VM1 on the VN is available and direct that traffic between + the two nodes be forwarded via IP-G (an IP Gateway). IP-G would then + decapsulate received traffic from one NV Domain, translate it + appropriately for the other domain, and re-encapsulate the packet for + delivery. + + xxxxxx xxxx +-----+ + +-----+ xxxxxx xxxxxx xxxxxx xxxxx | VM2 | + | VM1 | xx xx xxx xx |-----| + |-----| xx x xx x |NVE-B| + |NVE-A| x x +----+ x x +-----+ + +--+--+ x NV Domain A x |IP-G|--x x | + +-------x xx--+ | x xx | + x x +----+ x NV Domain B x | + +---x xx xx x---+ + | xxxx xx +->xx xx + | xxxxxxxx | xx xx + +---+-+ | xx xx + |NVA-1| +--+--+ xx xxx + +-----+ |NVA-2| xxxx xxxx + +-----+ xxxxx + + Figure 3: VM1 and VM2 in Different NV Domains + + NVAs at one site share information and interact with NVAs at other + sites, but only in a controlled manner. It is expected that policy + and access control will be applied at the boundaries between + different sites (and NVAs) so as to minimize dependencies on external + NVAs that could negatively impact the operation within a site. It is + an architectural principle that operations involving NVAs at one site + not be immediately impacted by failures or errors at another site. + + + +Black, et al. Informational [Page 27] + +RFC 8014 NVO3 Architecture December 2016 + + + (Of course, communication between NVEs in different NV Domains may be + impacted by such failures or errors.) It is a strong requirement + that an NVA continue to operate properly for local NVEs even if + external communication is interrupted (e.g., should communication + between a local and remote NVA fail). + + At a high level, a federation of interconnected NVAs has some + analogies to BGP and Autonomous Systems. Like an Autonomous System, + NVAs at one site are managed by a single administrative entity and do + not interact with external NVAs except as allowed by policy. + Likewise, the interface between NVAs at different sites is well + defined so that the internal details of operations at one site are + largely hidden to other sites. Finally, an NVA only peers with other + NVAs that it has a trusted relationship with, i.e., where a VN is + intended to span multiple NVAs. + + Reasons for using a federated model include: + + o Provide isolation among NVAs operating at different sites at + different geographic locations. + + o Control the quantity and rate of information updates that flow + (and must be processed) between different NVAs in different data + centers. + + o Control the set of external NVAs (and external sites) a site peers + with. A site will only peer with other sites that are cooperating + in providing an overlay service. + + o Allow policy to be applied between sites. A site will want to + carefully control what information it exports (and to whom) as + well as what information it is willing to import (and from whom). + + o Allow different protocols and architectures to be used for intra- + NVA vs. inter-NVA communication. For example, within a single + data center, a replicated transaction server using database + techniques might be an attractive implementation option for an + NVA, and protocols optimized for intra-NVA communication would + likely be different from protocols involving inter-NVA + communication between different sites. + + o Allow for optimized protocols rather than using a one-size-fits- + all approach. Within a data center, networks tend to have lower + latency, higher speed, and higher redundancy when compared with + WAN links interconnecting data centers. The design constraints + and trade-offs for a protocol operating within a data-center + network are different from those operating over WAN links. While + a single protocol could be used for both cases, there could be + + + +Black, et al. Informational [Page 28] + +RFC 8014 NVO3 Architecture December 2016 + + + advantages to using different and more specialized protocols for + the intra- and inter-NVA case. + +9.1. Inter-NVA Peering + + To support peering between different NVAs, an inter-NVA protocol is + needed. The inter-NVA protocol defines what information is exchanged + between NVAs. It is assumed that the protocol will be used to share + addressing information between data centers and must scale well over + WAN links. + +10. Control Protocol Work Areas + + The NVO3 architecture consists of two major distinct entities: NVEs + and NVAs. In order to provide isolation and independence between + these two entities, the NVO3 architecture calls for well-defined + protocols for interfacing between them. For an individual NVA, the + architecture calls for a logically centralized entity that could be + implemented in a distributed or replicated fashion. While the IETF + may choose to define one or more specific architectural approaches to + building individual NVAs, there is little need to pick exactly one + approach to the exclusion of others. An NVA for a single domain will + likely be deployed as a single vendor product; thus, there is little + benefit in standardizing the internal structure of an NVA. + + Individual NVAs peer with each other in a federated manner. The NVO3 + architecture calls for a well-defined interface between NVAs. + + Finally, a hypervisor-NVE protocol is needed to cover the split-NVE + scenario described in Section 4.2. + +11. NVO3 Data-Plane Encapsulation + + When tunneling tenant traffic, NVEs add an encapsulation header to + the original tenant packet. The exact encapsulation to use for NVO3 + does not seem to be critical. The main requirement is that the + encapsulation support a Context ID of sufficient size. A number of + encapsulations already exist that provide a VN Context of sufficient + size for NVO3. For example, Virtual eXtensible Local Area Network + (VXLAN) [RFC7348] has a 24-bit VXLAN Network Identifier (VNI). + Network Virtualization using Generic Routing Encapsulation (NVGRE) + [RFC7637] has a 24-bit Tenant Network ID (TNI). MPLS-over-GRE + provides a 20-bit label field. While there is widespread recognition + that a 12-bit VN Context would be too small (only 4096 distinct + values), it is generally agreed that 20 bits (1 million distinct + values) and 24 bits (16.8 million distinct values) are sufficient for + a wide variety of deployment scenarios. + + + + +Black, et al. Informational [Page 29] + +RFC 8014 NVO3 Architecture December 2016 + + +12. Operations, Administration, and Maintenance (OAM) + + The simplicity of operating and debugging overlay networks will be + critical for successful deployment. + + Overlay networks are based on tunnels between NVEs, so the + Operations, Administration, and Maintenance (OAM) [RFC6291] framework + for overlay networks can draw from prior IETF OAM work for tunnel- + based networks, specifically L2VPN OAM [RFC6136]. RFC 6136 focuses + on Fault Management and Performance Management as fundamental to + L2VPN service delivery, leaving the Configuration Management, + Accounting Management, and Security Management components of the Open + Systems Interconnection (OSI) Fault, Configuration, Accounting, + Performance, and Security (FCAPS) taxonomy [M.3400] for further + study. This section does likewise for NVO3 OAM, but those three + areas continue to be important parts of complete OAM functionality + for NVO3. + + The relationship between the overlay and underlay networks is a + consideration for fault and performance management -- a fault in the + underlay may manifest as fault and/or performance issues in the + overlay. Diagnosing and fixing such issues are complicated by NVO3 + abstracting the underlay network away from the overlay network (e.g., + intermediate nodes on the underlay network path between NVEs are + hidden from overlay VNs). + + NVO3-specific OAM techniques, protocol constructs, and tools are + needed to provide visibility beyond this abstraction to diagnose and + correct problems that appear in the overlay. Two examples are + underlay-aware traceroute [TRACEROUTE-VXLAN] and ping protocol + constructs for overlay networks [VXLAN-FAILURE] [NVO3-OVERLAY]. + + NVO3-specific tools and techniques are best viewed as complements to + (i.e., not as replacements for) single-network tools that apply to + the overlay and/or underlay networks. Coordination among the + individual network tools (for the overlay and underlay networks) and + NVO3-aware, dual-network tools is required to achieve effective + monitoring and fault diagnosis. For example, the defect detection + intervals and performance measurement intervals ought to be + coordinated among all tools involved in order to provide consistency + and comparability of results. + + For further discussion of NVO3 OAM requirements, see [NVO3-OAM]. + + + + + + + + +Black, et al. Informational [Page 30] + +RFC 8014 NVO3 Architecture December 2016 + + +13. Summary + + This document presents the overall architecture for NVO3. The + architecture calls for three main areas of protocol work: + + 1. A hypervisor-NVE protocol to support split-NVEs as discussed in + Section 4.2 + + 2. An NVE-NVA protocol for disseminating VN information (e.g., inner + to outer address mappings) + + 3. An NVA-NVA protocol for exchange of information about specific + virtual networks between federated NVAs + + It should be noted that existing protocols or extensions of existing + protocols are applicable. + +14. Security Considerations + + The data plane and control plane described in this architecture will + need to address potential security threats. + + For the data plane, tunneled application traffic may need protection + against being misdelivered, being modified, or having its content + exposed to an inappropriate third party. In all cases, encryption + between authenticated tunnel endpoints (e.g., via use of IPsec + [RFC4301]) and enforcing policies that control which endpoints and + VNs are permitted to exchange traffic can be used to mitigate risks. + + For the control plane, a combination of authentication and encryption + can be used between NVAs, between the NVA and NVE, as well as between + different components of the split-NVE approach. All entities will + need to properly authenticate with each other and enable encryption + for their interactions as appropriate to protect sensitive + information. + + Leakage of sensitive information about users or other entities + associated with VMs whose traffic is virtualized can also be covered + by using encryption for the control-plane protocols and enforcing + policies that control which NVO3 components are permitted to exchange + control-plane traffic. + + Control-plane elements such as NVEs and NVAs need to collect + performance and other data in order to carry out their functions. + This data can sometimes be unexpectedly sensitive, for example, + allowing non-obvious inferences of activity within a VM. This + provides a reason to minimize the data collected in some environments + in order to limit potential exposure of sensitive information. As + + + +Black, et al. Informational [Page 31] + +RFC 8014 NVO3 Architecture December 2016 + + + noted briefly in RFC 6973 [RFC6973] and RFC 7258 [RFC7258], there is + an inevitable tension between being privacy sensitive and taking into + account network operations in NVO3 protocol development. + + See the NVO3 framework security considerations in RFC 7365 [RFC7365] + for further discussion. + +15. Informative References + + [FRAMEWORK-MCAST] + Ghanwani, A., Dunbar, L., McBride, M., Bannai, V., and R. + Krishnan, "A Framework for Multicast in Network + Virtualization Overlays", Work in Progress, + draft-ietf-nvo3-mcast-framework-05, May 2016. + + [IEEE.802.1Q] + IEEE, "IEEE Standard for Local and metropolitan area + networks--Bridges and Bridged Networks", IEEE 802.1Q-2014, + DOI 10.1109/ieeestd.2014.6991462, + <http://ieeexplore.ieee.org/servlet/ + opac?punumber=6991460>. + + [M.3400] ITU-T, "TMN management functions", ITU-T + Recommendation M.3400, February 2000, + <https://www.itu.int/rec/T-REC-M.3400-200002-I/>. + + [NVE-NVA] Kreeger, L., Dutt, D., Narten, T., and D. Black, "Network + Virtualization NVE to NVA Control Protocol Requirements", + Work in Progress, draft-ietf-nvo3-nve-nva-cp-req-05, March + 2016. + + [NVO3-OAM] Chen, H., Ed., Ashwood-Smith, P., Xia, L., Iyengar, R., + Tsou, T., Sajassi, A., Boucadair, M., Jacquenet, C., + Daikoku, M., Ghanwani, A., and R. Krishnan, "NVO3 + Operations, Administration, and Maintenance Requirements", + Work in Progress, draft-ashwood-nvo3-oam-requirements-04, + October 2015. + + [NVO3-OVERLAY] + Kumar, N., Pignataro, C., Rao, D., and S. Aldrin, + "Detecting NVO3 Overlay Data Plane failures", Work in + Progress, draft-kumar-nvo3-overlay-ping-01, January 2014. + + [RFC826] Plummer, D., "Ethernet Address Resolution Protocol: Or + Converting Network Protocol Addresses to 48.bit Ethernet + Address for Transmission on Ethernet Hardware", STD 37, + RFC 826, DOI 10.17487/RFC0826, November 1982, + <http://www.rfc-editor.org/info/rfc826>. + + + +Black, et al. Informational [Page 32] + +RFC 8014 NVO3 Architecture December 2016 + + + [RFC4301] Kent, S. and K. Seo, "Security Architecture for the + Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, + December 2005, <http://www.rfc-editor.org/info/rfc4301>. + + [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private + Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February + 2006, <http://www.rfc-editor.org/info/rfc4364>. + + [RFC4861] Narten, T., Nordmark, E., Simpson, W., and H. Soliman, + "Neighbor Discovery for IP version 6 (IPv6)", RFC 4861, + DOI 10.17487/RFC4861, September 2007, + <http://www.rfc-editor.org/info/rfc4861>. + + [RFC6136] Sajassi, A., Ed. and D. Mohan, Ed., "Layer 2 Virtual + Private Network (L2VPN) Operations, Administration, and + Maintenance (OAM) Requirements and Framework", RFC 6136, + DOI 10.17487/RFC6136, March 2011, + <http://www.rfc-editor.org/info/rfc6136>. + + [RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu, + D., and S. Mansfield, "Guidelines for the Use of the "OAM" + Acronym in the IETF", BCP 161, RFC 6291, + DOI 10.17487/RFC6291, June 2011, + <http://www.rfc-editor.org/info/rfc6291>. + + [RFC6973] Cooper, A., Tschofenig, H., Aboba, B., Peterson, J., + Morris, J., Hansen, M., and R. Smith, "Privacy + Considerations for Internet Protocols", RFC 6973, + DOI 10.17487/RFC6973, July 2013, + <http://www.rfc-editor.org/info/rfc6973>. + + [RFC7258] Farrell, S. and H. Tschofenig, "Pervasive Monitoring Is an + Attack", BCP 188, RFC 7258, DOI 10.17487/RFC7258, May + 2014, <http://www.rfc-editor.org/info/rfc7258>. + + [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, + L., Sridhar, T., Bursell, M., and C. Wright, "Virtual + eXtensible Local Area Network (VXLAN): A Framework for + Overlaying Virtualized Layer 2 Networks over Layer 3 + Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, + <http://www.rfc-editor.org/info/rfc7348>. + + [RFC7364] Narten, T., Ed., Gray, E., Ed., Black, D., Fang, L., + Kreeger, L., and M. Napierala, "Problem Statement: + Overlays for Network Virtualization", RFC 7364, + DOI 10.17487/RFC7364, October 2014, + <http://www.rfc-editor.org/info/rfc7364>. + + + + +Black, et al. Informational [Page 33] + +RFC 8014 NVO3 Architecture December 2016 + + + [RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y. + Rekhter, "Framework for Data Center (DC) Network + Virtualization", RFC 7365, DOI 10.17487/RFC7365, October + 2014, <http://www.rfc-editor.org/info/rfc7365>. + + [RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network + Virtualization Using Generic Routing Encapsulation", + RFC 7637, DOI 10.17487/RFC7637, September 2015, + <http://www.rfc-editor.org/info/rfc7637>. + + [TRACEROUTE-VXLAN] + Nordmark, E., Appanna, C., Lo, A., Boutros, S., and A. + Dubey, "Layer-Transcending Traceroute for Overlay Networks + like VXLAN", Work in Progress, draft-nordmark-nvo3- + transcending-traceroute-03, July 2016. + + [USECASES] + Yong, L., Dunbar, L., Toy, M., Isaac, A., and V. Manral, + "Use Cases for Data Center Network Virtualization Overlay + Networks", Work in Progress, draft-ietf-nvo3-use-case-15, + December 2016. + + [VXLAN-FAILURE] + Jain, P., Singh, K., Balus, F., Henderickx, W., and V. + Bannai, "Detecting VXLAN Segment Failure", Work in + Progress, draft-jain-nvo3-vxlan-ping-00, June 2013. + +Acknowledgements + + Helpful comments and improvements to this document have come from + Alia Atlas, Abdussalam Baryun, Spencer Dawkins, Linda Dunbar, Stephen + Farrell, Anton Ivanov, Lizhong Jin, Suresh Krishnan, Mirja Kuehlwind, + Greg Mirsky, Carlos Pignataro, Dennis (Xiaohong) Qin, Erik Smith, + Takeshi Takahashi, Ziye Yang, and Lucy Yong. + + + + + + + + + + + + + + + + + +Black, et al. Informational [Page 34] + +RFC 8014 NVO3 Architecture December 2016 + + +Authors' Addresses + + David Black + Dell EMC + + Email: david.black@dell.com + + Jon Hudson + Independent + + Email: jon.hudson@gmail.com + + + Lawrence Kreeger + Independent + + Email: lkreeger@gmail.com + + + Marc Lasserre + Independent + + Email: mmlasserre@gmail.com + + + Thomas Narten + IBM + + Email: narten@us.ibm.com + + + + + + + + + + + + + + + + + + + + + + +Black, et al. Informational [Page 35] + |