From 4bfd864f10b68b71482b35c818559068ef8d5797 Mon Sep 17 00:00:00 2001 From: Thomas Voss Date: Wed, 27 Nov 2024 20:54:24 +0100 Subject: doc: Add RFC documents --- doc/rfc/rfc8204.txt | 1347 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1347 insertions(+) create mode 100644 doc/rfc/rfc8204.txt (limited to 'doc/rfc/rfc8204.txt') diff --git a/doc/rfc/rfc8204.txt b/doc/rfc/rfc8204.txt new file mode 100644 index 0000000..9e93f1b --- /dev/null +++ b/doc/rfc/rfc8204.txt @@ -0,0 +1,1347 @@ + + + + + + +Internet Engineering Task Force (IETF) M. Tahhan +Request for Comments: 8204 B. O'Mahony +Category: Informational Intel +ISSN: 2070-1721 A. Morton + AT&T Labs + September 2017 + + + Benchmarking Virtual Switches in the Open Platform for NFV (OPNFV) + +Abstract + + This memo describes the contributions of the Open Platform for NFV + (OPNFV) project on Virtual Switch Performance (VSPERF), particularly + in the areas of test setups and configuration parameters for the + system under test. This project has extended the current and + completed work of the Benchmarking Methodology Working Group in the + IETF and references existing literature. The Benchmarking + Methodology Working Group has traditionally conducted laboratory + characterization of dedicated physical implementations of + internetworking functions. Therefore, this memo describes the + additional considerations when virtual switches are implemented on + general-purpose hardware. The expanded tests and benchmarks are also + influenced by the OPNFV mission to support virtualization of the + "telco" infrastructure. + +Status of This Memo + + This document is not an Internet Standards Track specification; it is + published for informational purposes. + + This document is a product of the Internet Engineering Task Force + (IETF). It represents the consensus of the IETF community. It has + received public review and has been approved for publication by the + Internet Engineering Steering Group (IESG). Not all documents + approved by the IESG are a candidate for any level of Internet + Standard; see Section 2 of RFC 7841. + + Information about the current status of this document, any errata, + and how to provide feedback on it may be obtained at + https://www.rfc-editor.org/info/rfc8204. + + + + + + + + + + +Tahhan, et al. Informational [Page 1] + +RFC 8204 Benchmarking vSwitches September 2017 + + +Copyright Notice + + Copyright (c) 2017 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (https://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the Simplified BSD License. + +Table of Contents + + 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 + 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4 + 1.2. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 4 + 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 + 3. Benchmarking Considerations . . . . . . . . . . . . . . . . . 5 + 3.1. Comparison with Physical Network Functions . . . . . . . 5 + 3.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 6 + 3.3. New Configuration Parameters . . . . . . . . . . . . . . 6 + 3.4. Flow Classification . . . . . . . . . . . . . . . . . . . 8 + 3.5. Benchmarks Using Baselines with Resource Isolation . . . 9 + 4. VSPERF Specification Summary . . . . . . . . . . . . . . . . 11 + 5. 3x3 Matrix Coverage . . . . . . . . . . . . . . . . . . . . . 18 + 5.1. Speed of Activation . . . . . . . . . . . . . . . . . . . 19 + 5.2. Accuracy of Activation . . . . . . . . . . . . . . . . . 19 + 5.3. Reliability of Activation . . . . . . . . . . . . . . . . 19 + 5.4. Scale of Activation . . . . . . . . . . . . . . . . . . . 19 + 5.5. Speed of Operation . . . . . . . . . . . . . . . . . . . 19 + 5.6. Accuracy of Operation . . . . . . . . . . . . . . . . . . 19 + 5.7. Reliability of Operation . . . . . . . . . . . . . . . . 20 + 5.8. Scalability of Operation . . . . . . . . . . . . . . . . 20 + 5.9. Summary . . . . . . . . . . . . . . . . . . . . . . . . . 20 + 6. Security Considerations . . . . . . . . . . . . . . . . . . . 21 + 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 + 7.1. Normative References . . . . . . . . . . . . . . . . . . 21 + 7.2. Informative References . . . . . . . . . . . . . . . . . 22 + Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 23 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 + + + + + + + +Tahhan, et al. Informational [Page 2] + +RFC 8204 Benchmarking vSwitches September 2017 + + +1. Introduction + + The Benchmarking Methodology Working Group (BMWG) has traditionally + conducted laboratory characterization of dedicated physical + implementations of internetworking functions. The black-box + benchmarks of throughput, latency, forwarding rates, and others have + served our industry for many years. Now, Network Function + Virtualization (NFV) has the goal of transforming how internetwork + functions are implemented and therefore has garnered much attention. + + A virtual switch (vSwitch) is an important aspect of the NFV + infrastructure; it provides connectivity between and among physical + network functions and virtual network functions. As a result, there + are many vSwitch benchmarking efforts but few specifications to guide + the many new test design choices. This is a complex problem and an + industry-wide work in progress. In the future, several of BMWG's + fundamental specifications will likely be updated as more testing + experience helps to form consensus around new methodologies, and BMWG + should continue to collaborate with all organizations that share the + same goal. + + This memo describes the contributions of the Open Platform for NFV + (OPNFV) project on Virtual Switch Performance (VSPERF) + characterization through the Danube 3.0 (fourth) release [DanubeRel] + to the chartered work of the BMWG (with stable references to their + test descriptions). This project has extended the current and + completed work of the BMWG IETF and references existing literature. + For example, the most often referenced RFC is [RFC2544] (which + depends on [RFC1242]), so the foundation of the benchmarking work in + OPNFV is common and strong. The recommended extensions are + specifically in the areas of test setups and configuration parameters + for the system under test. + + See [VSPERFhome] for more background and the OPNFV website for + general information [OPNFV]. + + The authors note that OPNFV distinguishes itself from other open + source compute and networking projects through its emphasis on + existing "telco" services as opposed to cloud computing. There are + many ways in which telco requirements have different emphasis on + performance dimensions when compared to cloud computing: support for + and transfer of isochronous media streams is one example. + + + + + + + + + +Tahhan, et al. Informational [Page 3] + +RFC 8204 Benchmarking vSwitches September 2017 + + +1.1. Requirements Language + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and + "OPTIONAL" in this document are to be interpreted as described in + BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all + capitals, as shown here. + +1.2. Abbreviations + + For the purposes of this document, the following abbreviations apply: + + ACK Acknowledge + ACPI Advanced Configuration and Power Interface + BIOS Basic Input Output System + BMWG Benchmarking Methodology Working Group + CPDP Control Plane Data Plane + CPU Central Processing Unit + DIMM Dual In-line Memory Module + DPDK Data Plane Development Kit + DUT Device Under Test + GRUB Grand Unified Bootloader + ID Identification + IMIX Internet Mix + IP Internet Protocol + IPPM IP Performance Metrics + LAN Local Area Network + LTD Level Test Design + NFV Network Functions Virtualization + NIC Network Interface Card + NUMA Non-uniform Memory Access + OPNFV Open Platform for NFV + OS Operating System + PCI Peripheral Component Interconnect + PDV Packet Delay Variation + SR/IOV Single Root / Input Output Virtualization + SUT System Under Test + TCP Transmission Control Protocol + TSO TCP Segment Offload + UDP User Datagram Protocol + VM Virtual Machine + VNF Virtualised Network Function + VSPERF OPNFV vSwitch Performance Project + + + + + + + + +Tahhan, et al. Informational [Page 4] + +RFC 8204 Benchmarking vSwitches September 2017 + + +2. Scope + + The primary purpose and scope of the memo is to describe key aspects + of vSwitch benchmarking, particularly in the areas of test setups and + configuration parameters for the system under test, and extend the + body of extensive BMWG literature and experience. Initial feedback + indicates that many of these extensions may be applicable beyond this + memo's current scope (to hardware switches in the NFV infrastructure + and to virtual routers, for example). Additionally, this memo serves + as a vehicle to include more detail and relevant commentary from BMWG + and other open source communities under BMWG's chartered work to + characterize the NFV infrastructure. + + The benchmarking covered in this memo should be applicable to many + types of vSwitches and remain vSwitch agnostic to a great degree. + There has been no attempt to track and test all features of any + specific vSwitch implementation. + +3. Benchmarking Considerations + + This section highlights some specific considerations (from [RFC8172]) + related to benchmarks for virtual switches. The OPNFV project is + sharing its present view on these areas as they develop their + specifications in the Level Test Design (LTD) document as defined by + [IEEE829]. + +3.1. Comparison with Physical Network Functions + + To compare the performance of virtual designs and implementations + with their physical counterparts, identical benchmarks are needed. + BMWG has developed specifications for many physical network + functions. The BMWG has recommended reusing existing benchmarks and + methods in [RFC8172], and the OPNFV LTD expands on them as described + here. A key configuration aspect for vSwitches is the number of + parallel CPU cores required to achieve comparable performance with a + given physical device or whether some limit of scale will be reached + before the vSwitch can achieve the comparable performance level. + + It's unlikely that the virtual switch will be the only application + running on the SUT, so CPU utilization, cache utilization, and memory + footprint should also be recorded for the virtual implementations of + internetworking functions. However, internally measured metrics such + as these are not benchmarks; they may be useful for the audience + (e.g., operations) to know and may also be useful if there is a + problem encountered during testing. + + + + + + +Tahhan, et al. Informational [Page 5] + +RFC 8204 Benchmarking vSwitches September 2017 + + + Benchmark comparability between virtual and physical/hardware + implementations of equivalent functions will likely place more + detailed and exact requirements on the "testing systems" (in terms of + stream generation, algorithms to search for maximum values, and their + configurations). This is another area for standards development to + appreciate; however, this is a topic for a future document. + +3.2. Continued Emphasis on Black-Box Benchmarks + + External observations remain essential as the basis for benchmarks. + Internal observations with a fixed specification and interpretation + will be provided in parallel to assist the development of operations + procedures when the technology is deployed. + +3.3. New Configuration Parameters + + A key consideration when conducting any sort of benchmark is trying + to ensure the consistency and repeatability of test results. When + benchmarking the performance of a vSwitch, there are many factors + that can affect the consistency of results; one key factor is + matching the various hardware and software details of the SUT. This + section lists some of the many new parameters that this project + believes are critical to report in order to achieve repeatability. + + It has been the goal of the project to produce repeatable results, + and a large set of the parameters believed to be critical is provided + so that the benchmarking community can better appreciate the increase + in configuration complexity inherent in this work. The parameter set + below is assumed sufficient for the infrastructure in use by the + VSPERF project to obtain repeatable results from test to test. + + Hardware details (platform, processor, memory, and network) + including: + + o BIOS version, release date, and any configurations that were + modified + + o Power management at all levels (ACPI sleep states, processor + package, OS, etc.) + + o CPU microcode level + + o Number of enabled cores + + o Number of cores used for the test + + o Memory information (type and size) + + + + +Tahhan, et al. Informational [Page 6] + +RFC 8204 Benchmarking vSwitches September 2017 + + + o Memory DIMM configurations (quad rank performance may not be the + same as dual rank) in size, frequency, and slot locations + + o Number of physical NICs and their details (manufacturer, versions, + type, and the PCI slot they are plugged into) + + o NIC interrupt configuration (and any special features in use) + + o PCI configuration parameters (payload size, early ACK option, + etc.) + + Software details including: + + o OS RunLevel + + o OS version (for host and VNF) + + o Kernel version (for host and VNF) + + o GRUB boot parameters (for host and VNF) + + o Hypervisor details (type and version) + + o Selected vSwitch, version number, or commit ID used + + o vSwitch launch command line if it has been parameterized + + o Memory allocation to the vSwitch + + o Which NUMA node it is using and how many memory channels + + o DPDK or any other software dependency version number or commit ID + used + + o Memory allocation to a VM - if it's from Hugepages/elsewhere + + o VM storage type - snapshot, independent persistent, independent + non-persistent + + o Number of VMs + + o Number of virtual NICs (vNICs) - versions, type, and driver + + o Number of virtual CPUs and their core affinity on the host + + o Number of vNICs and their interrupt configurations + + + + + +Tahhan, et al. Informational [Page 7] + +RFC 8204 Benchmarking vSwitches September 2017 + + + o Thread affinitization for the applications (including the vSwitch + itself) on the host + + o Details of resource isolation, such as CPUs designated for Host/ + Kernel (isolcpu) and CPUs designated for specific processes + (taskset). + + Test traffic information: + + o Test duration + + o Number of flows + + o Traffic type - UDP, TCP, and others + + o Frame Sizes - fixed or IMIX [RFC6985] (note that with + [IEEE802.1ac], frames may be longer than 1500 bytes and up to 2000 + bytes) + + o Deployment Scenario - defines the communications path in the SUT + +3.4. Flow Classification + + Virtual switches group packets into flows by processing and matching + particular packet or frame header information, or by matching packets + based on the input ports. Thus, a flow can be thought of as a + sequence of packets that have the same set of header field values or + have arrived on the same physical or logical port. Performance + results can vary based on the parameters the vSwitch uses to match + for a flow. The recommended flow classification parameters for any + vSwitch performance tests are: the input port (physical or logical), + the source MAC address, the destination MAC address, the source IP + address, the destination IP address, and the Ethernet protocol type + field (although classification may take place on other fields, such + as source and destination transport port numbers). It is essential + to increase the flow timeout time on a vSwitch before conducting any + performance tests that do not intend to measure the flow setup time + (see Section 3 of [RFC2889]). Normally, the first packet of a + particular stream will install the flow in the virtual switch, which + introduces additional latency; subsequent packets of the same flow + are not subject to this latency if the flow is already installed on + the vSwitch. + + + + + + + + + +Tahhan, et al. Informational [Page 8] + +RFC 8204 Benchmarking vSwitches September 2017 + + +3.5. Benchmarks Using Baselines with Resource Isolation + + This outline describes the measurement of baselines with isolated + resources at a high level, which is the intended approach at this + time. + + 1. Baselines: + + * Optional: Benchmark platform forwarding capability without a + vSwitch or VNF for at least 72 hours (serves as a means of + platform validation and a means to obtain the base performance + for the platform in terms of its maximum forwarding rate and + latency). + + __ + +--------------------------------------------------+ | + | +------------------------------------------+ | | + | | | | | + | | Simple Forwarding App | | Host + | | | | | + | +------------------------------------------+ | | + | | NIC | | | + +---+------------------------------------------+---+ __| + ^ : + | | + : v + +--------------------------------------------------+ + | | + | Traffic Generator | + | | + +--------------------------------------------------+ + + Figure 1: Benchmark Platform Forwarding Capability + + + + + + + + + + + + + + + + + + +Tahhan, et al. Informational [Page 9] + +RFC 8204 Benchmarking vSwitches September 2017 + + + * Benchmark VNF forwarding capability with direct connectivity + (vSwitch bypass, e.g., SR/IOV) for at least 72 hours (serves + as a means of VNF validation and a means to obtain the base + performance for the VNF in terms of its maximum forwarding + rate and latency). The metrics gathered from this test will + serve as a key comparison point for vSwitch bypass + technologies performance and vSwitch performance. + + __ + +--------------------------------------------------+ __ | + | +------------------------------------------+ | | | + | | | | Host/ | + | | VNF | | Guest | + | | | | | | + | +------------------------------------------+ | __| | + | | Passthrough/SR-IOV | | Host + | +------------------------------------------+ | | + | | NIC | | | + +---+------------------------------------------+---+ __| + ^ : + | | + : v + +--------------------------------------------------+ + | | + | Traffic Generator | + | | + +--------------------------------------------------+ + + Figure 2: Benchmark VNF Forwarding Capability + + * Benchmarking with isolated resources alone and with other + resources (both hardware and software) disabled; for example, + vSwitch and VM are SUT. + + * Benchmarking with isolated resources alone, thus leaving some + resources unused. + + * Benchmarking with isolated resources and all resources + occupied. + + 2. Next Steps: + + * Limited sharing + + * Production scenarios + + * Stressful scenarios + + + + +Tahhan, et al. Informational [Page 10] + +RFC 8204 Benchmarking vSwitches September 2017 + + +4. VSPERF Specification Summary + + The overall specification in preparation is referred to as a Level + Test Design (LTD) document, which will contain a suite of performance + tests. The base performance tests in the LTD are based on the + pre-existing specifications developed by the BMWG to test the + performance of physical switches. These specifications include: + + o Benchmarking Methodology for Network Interconnect Devices + [RFC2544] + + o Benchmarking Methodology for LAN Switching [RFC2889] + + o Device Reset Characterization [RFC6201] + + o Packet Delay Variation Applicability Statement [RFC5481] + + The two most recent RFCs above ([RFC6201] and [RFC5481]) are being + applied in benchmarking for the first time and represent a + development challenge for test equipment developers. Fortunately, + many members of the testing system community have engaged on the + VSPERF project, including an open source test system. + + In addition to this, the LTD also reuses the terminology defined by: + + o Benchmarking Terminology for LAN Switching Devices [RFC2285] + + It is recommended that these references be included in future + benchmarking specifications: + + o Methodology for IP Multicast Benchmarking [RFC3918] + + o Packet Reordering Metrics [RFC4737] + + As one might expect, the most fundamental internetworking + characteristics of throughput and latency remain important when the + switch is virtualized, and these benchmarks figure prominently in the + specification. + + When considering characteristics important to "telco" network + functions, additional performance metrics are needed. In this case, + the project specifications have referenced metrics from the IETF IP + Performance Metrics (IPPM) literature. This means that the latency + test described in [RFC2544] is replaced by measurement of a metric + derived from IPPM's [RFC7679], where a set of statistical summaries + will be provided (mean, max, min, and percentiles). Further metrics + planned to be benchmarked include packet delay variation as defined + by [RFC5481], reordering, burst behaviour, DUT availability, DUT + + + +Tahhan, et al. Informational [Page 11] + +RFC 8204 Benchmarking vSwitches September 2017 + + + capacity, and packet loss in long-term testing at the throughput + level, where some low level of background loss may be present and + characterized. + + Tests have been designed to collect the metrics below: + + o Throughput tests are designed to measure the maximum forwarding + rate (in frames per second, fps) and bit rate (in Mbps) for a + constant load (as defined by [RFC1242]) without traffic loss. + + o Packet and frame-delay distribution tests are designed to measure + the average minimum and maximum packet (and/or frame) delay for + constant loads. + + o Packet delay tests are designed to understand latency distribution + for different packet sizes and to uncover outliers over an + extended test run. + + o Scalability tests are designed to understand how the virtual + switch performs with an increasing number of flows, number of + active ports, configuration complexity of the forwarding logic, + etc. + + o Stream performance tests (with TCP or UDP) are designed to measure + bulk data transfer performance, i.e., how fast systems can send + and receive data through the switch. + + o Control-path and data-path coupling tests are designed to + understand how closely the data path and the control path are + coupled, as well as the effect of this coupling on the performance + of the DUT (for example, delay of the initial packet of a flow). + + o CPU and memory consumption tests are designed to understand the + virtual switch's footprint on the system and are conducted as + auxiliary measurements with the benchmarks above. They include + CPU utilization, cache utilization, and memory footprint. + + o The so-called "soak" tests, where the selected test is conducted + over a long period of time (with an ideal duration of 24 hours but + only long enough to determine that stability issues exist when + found; there is no requirement to continue a test when a DUT + exhibits instability over time). The key performance + characteristics and benchmarks for a DUT are determined (using + short duration tests) prior to conducting soak tests. The purpose + of soak tests is to capture transient changes in performance, + which may occur due to infrequent processes, memory leaks, or the + low-probability coincidence of two or more processes. The + stability of the DUT is the paramount consideration, so + + + +Tahhan, et al. Informational [Page 12] + +RFC 8204 Benchmarking vSwitches September 2017 + + + performance must be evaluated periodically during continuous + testing, and this results in use of frame rate metrics [RFC2889] + instead of throughput [RFC2544] (which requires stopping traffic + to allow time for all traffic to exit internal queues), for + example. + + Additional test specification development should include: + + o Request/response performance tests (with TCP or UDP), which + measure the transaction rate through the switch. + + o Noisy neighbor tests, in order to understand the effects of + resource sharing on the performance of a virtual switch. + + o Tests derived from examination of ETSI NFV Draft GS IFA003 + requirements [IFA003] on characterization of acceleration + technologies applied to vSwitches. + + The flexibility of deployment of a virtual switch within a network + means that it is necessary to characterize the performance of a + vSwitch in various deployment scenarios. The deployment scenarios + under consideration are shown in the following figures: + + __ + +--------------------------------------------------+ | + | +--------------------+ | | + | | | | | + | | v | | Host + | +--------------+ +--------------+ | | + | | PHY Port | vSwitch | PHY Port | | | + +---+--------------+------------+--------------+---+ __| + ^ : + | | + : v + +--------------------------------------------------+ + | | + | Traffic Generator | + | | + +--------------------------------------------------+ + + Figure 3: Physical Port to Virtual Switch to Physical Port + + + + + + + + + + +Tahhan, et al. Informational [Page 13] + +RFC 8204 Benchmarking vSwitches September 2017 + + + __ + +---------------------------------------------------+ | + | | | + | +-------------------------------------------+ | | + | | Application | | | + | +-------------------------------------------+ | | + | ^ : | | + | | | | | Guest + | : v | | + | +---------------+ +---------------+ | | + | | Logical Port 0| | Logical Port 1| | | + +---+---------------+-----------+---------------+---+ __| + ^ : + | | + : v __ + +---+---------------+----------+---------------+---+ | + | | Logical Port 0| | Logical Port 1| | | + | +---------------+ +---------------+ | | + | ^ : | | + | | | | | Host + | : v | | + | +--------------+ +--------------+ | | + | | PHY Port | vSwitch | PHY Port | | | + +---+--------------+------------+--------------+---+ __| + ^ : + | | + : v + +--------------------------------------------------+ + | | + | Traffic Generator | + | | + +--------------------------------------------------+ + + Figure 4: Physical Port to Virtual Switch to VNF to Virtual Switch to + Physical Port + + + + + + + + + + + + + + + + +Tahhan, et al. Informational [Page 14] + +RFC 8204 Benchmarking vSwitches September 2017 + + + __ + +----------------------+ +----------------------+ | + | Guest 1 | | Guest 2 | | + | +---------------+ | | +---------------+ | | + | | Application | | | | Application | | | + | +---------------+ | | +---------------+ | | + | ^ | | | ^ | | | + | | v | | | v | | Guests + | +---------------+ | | +---------------+ | | + | | Logical Ports | | | | Logical Ports | | | + | | 0 1 | | | | 0 1 | | | + +---+---------------+--+ +---+---------------+--+__| + ^ : ^ : + | | | | + : v : v _ + +---+---------------+---------+---------------+--+ | + | | 0 1 | | 3 4 | | | + | | Logical Ports | | Logical Ports | | | + | +---------------+ +---------------+ | | + | ^ | ^ | | | Host + | | \-----------------/ v | | + | +--------------+ +--------------+ | | + | | PHY Ports | vSwitch | PHY Ports | | | + +---+--------------+----------+--------------+---+_| + ^ : + | | + : v + +--------------------------------------------------+ + | | + | Traffic Generator | + | | + +--------------------------------------------------+ + + Figure 5: Physical Port to Virtual Switch to VNF to Virtual Switch to + VNF to Virtual Switch to Physical Port + + + + + + + + + + + + + + + + +Tahhan, et al. Informational [Page 15] + +RFC 8204 Benchmarking vSwitches September 2017 + + + __ + +---------------------------------------------------+ | + | | | + | +-------------------------------------------+ | | + | | Application | | | + | +-------------------------------------------+ | | + | ^ | | + | | | | Guest + | : | | + | +---------------+ | | + | | Logical Port 0| | | + +---+---------------+-------------------------------+ __| + ^ + | + : __ + +---+---------------+------------------------------+ | + | | Logical Port 0| | | + | +---------------+ | | + | ^ | | + | | | | Host + | : | | + | +--------------+ | | + | | PHY Port | vSwitch | | + +---+--------------+------------ -------------- ---+ __| + ^ + | + : + +--------------------------------------------------+ + | | + | Traffic Generator | + | | + +--------------------------------------------------+ + + Figure 6: Physical Port to Virtual Switch to VNF + + + + + + + + + + + + + + + + + +Tahhan, et al. Informational [Page 16] + +RFC 8204 Benchmarking vSwitches September 2017 + + + __ + +---------------------------------------------------+ | + | | | + | +-------------------------------------------+ | | + | | Application | | | + | +-------------------------------------------+ | | + | : | | + | | | | Guest + | v | | + | +---------------+ | | + | | Logical Port | | | + +-------------------------------+---------------+---+ __| + : + | + v __ + +------------------------------+---------------+---+ | + | | Logical Port | | | + | +---------------+ | | + | : | | + | | | | Host + | v | | + | +--------------+ | | + | vSwitch | PHY Port | | | + +-------------------------------+--------------+---+ __| + : + | + v + +--------------------------------------------------+ + | | + | Traffic Generator | + | | + +--------------------------------------------------+ + + Figure 7: VNF to Virtual Switch to Physical Port + + + + + + + + + + + + + + + + + +Tahhan, et al. Informational [Page 17] + +RFC 8204 Benchmarking vSwitches September 2017 + + + __ + +----------------------+ +----------------------+ | + | Guest 1 | | Guest 2 | | + | +---------------+ | | +---------------+ | | + | | Application | | | | Application | | | + | +---------------+ | | +---------------+ | | + | | | | ^ | | + | v | | | | | Guests + | +---------------+ | | +---------------+ | | + | | Logical Ports | | | | Logical Ports | | | + | | 0 | | | | 0 | | | + +---+---------------+--+ +---+---------------+--+__| + : ^ + | | + v : _ + +---+---------------+---------+---------------+--+ | + | | 1 | | 1 | | | + | | Logical Ports | | Logical Ports | | | + | +---------------+ +---------------+ | | + | | ^ | | Host + | \-----------------/ | | + | | | + | vSwitch | | + +------------------------------------------------+_| + + Figure 8: VNF to Virtual Switch to VNF + + A set of deployment scenario figures is available on the VSPERF "Test + Methodology" wiki page [TestTopo]. + +5. 3x3 Matrix Coverage + + This section organizes the many existing test specifications into the + "3x3" matrix (introduced in [RFC8172]). Because the LTD + specification ID names are quite long, this section is organized into + lists for each occupied cell of the matrix (not all are occupied; + also, the matrix has grown to 3x4 to accommodate scale metrics when + displaying the coverage of many metrics/benchmarks). The current + version of the LTD specification is available; see [LTD]. + + The tests listed below assess the activation of paths in the data + plane rather than the control plane. + + A complete list of tests with short summaries is available on the + VSPERF "LTD Test Spec Overview" wiki page [LTDoverV]. + + + + + + +Tahhan, et al. Informational [Page 18] + +RFC 8204 Benchmarking vSwitches September 2017 + + +5.1. Speed of Activation + + o Activation.RFC2889.AddressLearningRate + + o PacketLatency.InitialPacketProcessingLatency + +5.2. Accuracy of Activation + + o CPDP.Coupling.Flow.Addition + +5.3. Reliability of Activation + + o Throughput.RFC2544.SystemRecoveryTime + + o Throughput.RFC2544.ResetTime + +5.4. Scale of Activation + + o Activation.RFC2889.AddressCachingCapacity + +5.5. Speed of Operation + + o Throughput.RFC2544.PacketLossRate + + o Stress.RFC2544.0PacketLoss + + o Throughput.RFC2544.PacketLossRateFrameModification + + o Throughput.RFC2544.BackToBackFrames + + o Throughput.RFC2889.MaxForwardingRate + + o Throughput.RFC2889.ForwardPressure + + o Throughput.RFC2889.BroadcastFrameForwarding + + o Throughput.RFC2544.WorstN-BestN + + o Throughput.Overlay.Network..RFC2544.PacketLossRatio + +5.6. Accuracy of Operation + + o Throughput.RFC2889.ErrorFramesFiltering + + o Throughput.RFC2544.Profile + + + + + + +Tahhan, et al. Informational [Page 19] + +RFC 8204 Benchmarking vSwitches September 2017 + + +5.7. Reliability of Operation + + o Throughput.RFC2889.Soak + + o Throughput.RFC2889.SoakFrameModification + + o PacketDelayVariation.RFC3393.Soak + +5.8. Scalability of Operation + + o Scalability.RFC2544.0PacketLoss + + o MemoryBandwidth.RFC2544.0PacketLoss.Scalability + + o Scalability.VNF.RFC2544.PacketLossProfile + + o Scalability.VNF.RFC2544.PacketLossRatio + +5.9. Summary + + |---------------------------------------------------------------------| + | | | | | | + | | SPEED | ACCURACY | RELIABILITY | SCALE | + | | | | | | + |---------------------------------------------------------------------| + | | | | | | + | Activation | X | X | X | X | + | | | | | | + |---------------------------------------------------------------------| + | | | | | | + | Operation | X | X | X | X | + | | | | | | + |---------------------------------------------------------------------| + | | | | | | + | De-activation| | | | | + | | | | | | + |---------------------------------------------------------------------| + + + + + + + + + + + + + + +Tahhan, et al. Informational [Page 20] + +RFC 8204 Benchmarking vSwitches September 2017 + + +6. Security Considerations + + Benchmarking activities as described in this memo are limited to + technology characterization of a Device Under Test/System Under Test + (DUT/SUT) using controlled stimuli in a laboratory environment with + dedicated address space and the constraints specified in the sections + above. + + The benchmarking network topology will be an independent test setup + and MUST NOT be connected to devices that may forward the test + traffic into a production network or misroute traffic to the test + management network. + + Further, benchmarking is performed on a "black-box" basis and relies + solely on measurements observable external to the DUT/SUT. + + Special capabilities SHOULD NOT exist in the DUT/SUT specifically for + benchmarking purposes. Any implications for network security arising + from the DUT/SUT SHOULD be identical in the lab and in production + networks. + +7. References + +7.1. Normative References + + [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate + Requirement Levels", BCP 14, RFC 2119, + DOI 10.17487/RFC2119, March 1997, + . + + [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN + Switching Devices", RFC 2285, DOI 10.17487/RFC2285, + February 1998, . + + [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for + Network Interconnect Devices", RFC 2544, + DOI 10.17487/RFC2544, March 1999, + . + + [RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology + for LAN Switching Devices", RFC 2889, + DOI 10.17487/RFC2889, August 2000, + . + + [RFC3918] Stopp, D. and B. Hickman, "Methodology for IP Multicast + Benchmarking", RFC 3918, DOI 10.17487/RFC3918, October + 2004, . + + + + +Tahhan, et al. Informational [Page 21] + +RFC 8204 Benchmarking vSwitches September 2017 + + + [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, + S., and J. Perser, "Packet Reordering Metrics", RFC 4737, + DOI 10.17487/RFC4737, November 2006, + . + + [RFC6201] Asati, R., Pignataro, C., Calabria, F., and C. Olvera, + "Device Reset Characterization", RFC 6201, + DOI 10.17487/RFC6201, March 2011, + . + + [RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet + Sizes for Additional Testing", RFC 6985, + DOI 10.17487/RFC6985, July 2013, + . + + [RFC7679] Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, + Ed., "A One-Way Delay Metric for IP Performance Metrics + (IPPM)", STD 81, RFC 7679, DOI 10.17487/RFC7679, January + 2016, . + + [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC + 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, + May 2017, . + +7.2. Informative References + + [BENCHMARK-METHOD] + Huang, L., Ed., Rong, G., Ed., Mandeville, B., and B. + Hickman, "Benchmarking Methodology for Virtualization + Network Performance", Work in Progress, draft-huang-bmwg- + virtual-network-performance-03, July 2017. + + [DanubeRel] + OPNFV, "Danube", + . + + [IEEE802.1ac] + IEEE, "IEEE Standard for Local and metropolitan area + networks -- Media Access Control (MAC) Service + Definition", IEEE 802.1AC-2016, + DOI 10.1109/IEEESTD.2017.7875381, 2016, + . + + [IEEE829] IEEE, "IEEE Standard for Software and System Test + Documentation", IEEE 829-2008, + DOI 10.1109/IEEESTD.2008.4578383, + . + + + +Tahhan, et al. Informational [Page 22] + +RFC 8204 Benchmarking vSwitches September 2017 + + + [IFA003] ETSI, "Network Functions Virtualisation (NFV); + Acceleration Technologies; vSwitch Benchmarking and + Acceleration Specification", ETSI GS NFV-IFA 003 V2.1.1, + April 2016, . + + [LTD] Tahhan, M., "VSPERF Level Test Design (LTD)", + . + + [LTDoverV] Morton, A., "LTD Test Spec Overview", + . + + [OPNFV] OPNFV, "OPNFV", . + + [RFC1242] Bradner, S., "Benchmarking Terminology for Network + Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, + July 1991, . + + [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation + Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, + March 2009, . + + [RFC8172] Morton, A., "Considerations for Benchmarking Virtual + Network Functions and Their Infrastructure", RFC 8172, + DOI 10.17487/RFC8172, July 2017, + . + + [TestTopo] Snyder, E., "Test Methodology", + . + + [VSPERFhome] + Tahhan, M., "VSPERF Home", + . + +Acknowledgements + + The authors appreciate and acknowledge comments from Scott Bradner, + Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik, + Christian Trautman, Benoit Claise, and others for their reviews. + + We also acknowledge the early work in [BENCHMARK-METHOD] and useful + discussion with the authors. + + + + + +Tahhan, et al. Informational [Page 23] + +RFC 8204 Benchmarking vSwitches September 2017 + + +Authors' Addresses + + Maryam Tahhan + Intel + + Email: maryam.tahhan@intel.com + + + Billy O'Mahony + Intel + + Email: billy.o.mahony@intel.com + + + Al Morton + AT&T Labs + 200 Laurel Avenue South + Middletown, NJ 07748 + United States of America + + Phone: +1 732 420 1571 + Fax: +1 732 368 1192 + Email: acmorton@att.com + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Tahhan, et al. Informational [Page 24] + -- cgit v1.2.3