summaryrefslogtreecommitdiff
path: root/doc/rfc/rfc8204.txt
diff options
context:
space:
mode:
Diffstat (limited to 'doc/rfc/rfc8204.txt')
-rw-r--r--doc/rfc/rfc8204.txt1347
1 files changed, 1347 insertions, 0 deletions
diff --git a/doc/rfc/rfc8204.txt b/doc/rfc/rfc8204.txt
new file mode 100644
index 0000000..9e93f1b
--- /dev/null
+++ b/doc/rfc/rfc8204.txt
@@ -0,0 +1,1347 @@
+
+
+
+
+
+
+Internet Engineering Task Force (IETF) M. Tahhan
+Request for Comments: 8204 B. O'Mahony
+Category: Informational Intel
+ISSN: 2070-1721 A. Morton
+ AT&T Labs
+ September 2017
+
+
+ Benchmarking Virtual Switches in the Open Platform for NFV (OPNFV)
+
+Abstract
+
+ This memo describes the contributions of the Open Platform for NFV
+ (OPNFV) project on Virtual Switch Performance (VSPERF), particularly
+ in the areas of test setups and configuration parameters for the
+ system under test. This project has extended the current and
+ completed work of the Benchmarking Methodology Working Group in the
+ IETF and references existing literature. The Benchmarking
+ Methodology Working Group has traditionally conducted laboratory
+ characterization of dedicated physical implementations of
+ internetworking functions. Therefore, this memo describes the
+ additional considerations when virtual switches are implemented on
+ general-purpose hardware. The expanded tests and benchmarks are also
+ influenced by the OPNFV mission to support virtualization of the
+ "telco" infrastructure.
+
+Status of This Memo
+
+ This document is not an Internet Standards Track specification; it is
+ published for informational purposes.
+
+ This document is a product of the Internet Engineering Task Force
+ (IETF). It represents the consensus of the IETF community. It has
+ received public review and has been approved for publication by the
+ Internet Engineering Steering Group (IESG). Not all documents
+ approved by the IESG are a candidate for any level of Internet
+ Standard; see Section 2 of RFC 7841.
+
+ Information about the current status of this document, any errata,
+ and how to provide feedback on it may be obtained at
+ https://www.rfc-editor.org/info/rfc8204.
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 1]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+Copyright Notice
+
+ Copyright (c) 2017 IETF Trust and the persons identified as the
+ document authors. All rights reserved.
+
+ This document is subject to BCP 78 and the IETF Trust's Legal
+ Provisions Relating to IETF Documents
+ (https://trustee.ietf.org/license-info) in effect on the date of
+ publication of this document. Please review these documents
+ carefully, as they describe your rights and restrictions with respect
+ to this document. Code Components extracted from this document must
+ include Simplified BSD License text as described in Section 4.e of
+ the Trust Legal Provisions and are provided without warranty as
+ described in the Simplified BSD License.
+
+Table of Contents
+
+ 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
+ 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4
+ 1.2. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 4
+ 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
+ 3. Benchmarking Considerations . . . . . . . . . . . . . . . . . 5
+ 3.1. Comparison with Physical Network Functions . . . . . . . 5
+ 3.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 6
+ 3.3. New Configuration Parameters . . . . . . . . . . . . . . 6
+ 3.4. Flow Classification . . . . . . . . . . . . . . . . . . . 8
+ 3.5. Benchmarks Using Baselines with Resource Isolation . . . 9
+ 4. VSPERF Specification Summary . . . . . . . . . . . . . . . . 11
+ 5. 3x3 Matrix Coverage . . . . . . . . . . . . . . . . . . . . . 18
+ 5.1. Speed of Activation . . . . . . . . . . . . . . . . . . . 19
+ 5.2. Accuracy of Activation . . . . . . . . . . . . . . . . . 19
+ 5.3. Reliability of Activation . . . . . . . . . . . . . . . . 19
+ 5.4. Scale of Activation . . . . . . . . . . . . . . . . . . . 19
+ 5.5. Speed of Operation . . . . . . . . . . . . . . . . . . . 19
+ 5.6. Accuracy of Operation . . . . . . . . . . . . . . . . . . 19
+ 5.7. Reliability of Operation . . . . . . . . . . . . . . . . 20
+ 5.8. Scalability of Operation . . . . . . . . . . . . . . . . 20
+ 5.9. Summary . . . . . . . . . . . . . . . . . . . . . . . . . 20
+ 6. Security Considerations . . . . . . . . . . . . . . . . . . . 21
+ 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 21
+ 7.1. Normative References . . . . . . . . . . . . . . . . . . 21
+ 7.2. Informative References . . . . . . . . . . . . . . . . . 22
+ Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 23
+ Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24
+
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 2]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+1. Introduction
+
+ The Benchmarking Methodology Working Group (BMWG) has traditionally
+ conducted laboratory characterization of dedicated physical
+ implementations of internetworking functions. The black-box
+ benchmarks of throughput, latency, forwarding rates, and others have
+ served our industry for many years. Now, Network Function
+ Virtualization (NFV) has the goal of transforming how internetwork
+ functions are implemented and therefore has garnered much attention.
+
+ A virtual switch (vSwitch) is an important aspect of the NFV
+ infrastructure; it provides connectivity between and among physical
+ network functions and virtual network functions. As a result, there
+ are many vSwitch benchmarking efforts but few specifications to guide
+ the many new test design choices. This is a complex problem and an
+ industry-wide work in progress. In the future, several of BMWG's
+ fundamental specifications will likely be updated as more testing
+ experience helps to form consensus around new methodologies, and BMWG
+ should continue to collaborate with all organizations that share the
+ same goal.
+
+ This memo describes the contributions of the Open Platform for NFV
+ (OPNFV) project on Virtual Switch Performance (VSPERF)
+ characterization through the Danube 3.0 (fourth) release [DanubeRel]
+ to the chartered work of the BMWG (with stable references to their
+ test descriptions). This project has extended the current and
+ completed work of the BMWG IETF and references existing literature.
+ For example, the most often referenced RFC is [RFC2544] (which
+ depends on [RFC1242]), so the foundation of the benchmarking work in
+ OPNFV is common and strong. The recommended extensions are
+ specifically in the areas of test setups and configuration parameters
+ for the system under test.
+
+ See [VSPERFhome] for more background and the OPNFV website for
+ general information [OPNFV].
+
+ The authors note that OPNFV distinguishes itself from other open
+ source compute and networking projects through its emphasis on
+ existing "telco" services as opposed to cloud computing. There are
+ many ways in which telco requirements have different emphasis on
+ performance dimensions when compared to cloud computing: support for
+ and transfer of isochronous media streams is one example.
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 3]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+1.1. Requirements Language
+
+ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+ "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
+ "OPTIONAL" in this document are to be interpreted as described in
+ BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
+ capitals, as shown here.
+
+1.2. Abbreviations
+
+ For the purposes of this document, the following abbreviations apply:
+
+ ACK Acknowledge
+ ACPI Advanced Configuration and Power Interface
+ BIOS Basic Input Output System
+ BMWG Benchmarking Methodology Working Group
+ CPDP Control Plane Data Plane
+ CPU Central Processing Unit
+ DIMM Dual In-line Memory Module
+ DPDK Data Plane Development Kit
+ DUT Device Under Test
+ GRUB Grand Unified Bootloader
+ ID Identification
+ IMIX Internet Mix
+ IP Internet Protocol
+ IPPM IP Performance Metrics
+ LAN Local Area Network
+ LTD Level Test Design
+ NFV Network Functions Virtualization
+ NIC Network Interface Card
+ NUMA Non-uniform Memory Access
+ OPNFV Open Platform for NFV
+ OS Operating System
+ PCI Peripheral Component Interconnect
+ PDV Packet Delay Variation
+ SR/IOV Single Root / Input Output Virtualization
+ SUT System Under Test
+ TCP Transmission Control Protocol
+ TSO TCP Segment Offload
+ UDP User Datagram Protocol
+ VM Virtual Machine
+ VNF Virtualised Network Function
+ VSPERF OPNFV vSwitch Performance Project
+
+
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 4]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+2. Scope
+
+ The primary purpose and scope of the memo is to describe key aspects
+ of vSwitch benchmarking, particularly in the areas of test setups and
+ configuration parameters for the system under test, and extend the
+ body of extensive BMWG literature and experience. Initial feedback
+ indicates that many of these extensions may be applicable beyond this
+ memo's current scope (to hardware switches in the NFV infrastructure
+ and to virtual routers, for example). Additionally, this memo serves
+ as a vehicle to include more detail and relevant commentary from BMWG
+ and other open source communities under BMWG's chartered work to
+ characterize the NFV infrastructure.
+
+ The benchmarking covered in this memo should be applicable to many
+ types of vSwitches and remain vSwitch agnostic to a great degree.
+ There has been no attempt to track and test all features of any
+ specific vSwitch implementation.
+
+3. Benchmarking Considerations
+
+ This section highlights some specific considerations (from [RFC8172])
+ related to benchmarks for virtual switches. The OPNFV project is
+ sharing its present view on these areas as they develop their
+ specifications in the Level Test Design (LTD) document as defined by
+ [IEEE829].
+
+3.1. Comparison with Physical Network Functions
+
+ To compare the performance of virtual designs and implementations
+ with their physical counterparts, identical benchmarks are needed.
+ BMWG has developed specifications for many physical network
+ functions. The BMWG has recommended reusing existing benchmarks and
+ methods in [RFC8172], and the OPNFV LTD expands on them as described
+ here. A key configuration aspect for vSwitches is the number of
+ parallel CPU cores required to achieve comparable performance with a
+ given physical device or whether some limit of scale will be reached
+ before the vSwitch can achieve the comparable performance level.
+
+ It's unlikely that the virtual switch will be the only application
+ running on the SUT, so CPU utilization, cache utilization, and memory
+ footprint should also be recorded for the virtual implementations of
+ internetworking functions. However, internally measured metrics such
+ as these are not benchmarks; they may be useful for the audience
+ (e.g., operations) to know and may also be useful if there is a
+ problem encountered during testing.
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 5]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+ Benchmark comparability between virtual and physical/hardware
+ implementations of equivalent functions will likely place more
+ detailed and exact requirements on the "testing systems" (in terms of
+ stream generation, algorithms to search for maximum values, and their
+ configurations). This is another area for standards development to
+ appreciate; however, this is a topic for a future document.
+
+3.2. Continued Emphasis on Black-Box Benchmarks
+
+ External observations remain essential as the basis for benchmarks.
+ Internal observations with a fixed specification and interpretation
+ will be provided in parallel to assist the development of operations
+ procedures when the technology is deployed.
+
+3.3. New Configuration Parameters
+
+ A key consideration when conducting any sort of benchmark is trying
+ to ensure the consistency and repeatability of test results. When
+ benchmarking the performance of a vSwitch, there are many factors
+ that can affect the consistency of results; one key factor is
+ matching the various hardware and software details of the SUT. This
+ section lists some of the many new parameters that this project
+ believes are critical to report in order to achieve repeatability.
+
+ It has been the goal of the project to produce repeatable results,
+ and a large set of the parameters believed to be critical is provided
+ so that the benchmarking community can better appreciate the increase
+ in configuration complexity inherent in this work. The parameter set
+ below is assumed sufficient for the infrastructure in use by the
+ VSPERF project to obtain repeatable results from test to test.
+
+ Hardware details (platform, processor, memory, and network)
+ including:
+
+ o BIOS version, release date, and any configurations that were
+ modified
+
+ o Power management at all levels (ACPI sleep states, processor
+ package, OS, etc.)
+
+ o CPU microcode level
+
+ o Number of enabled cores
+
+ o Number of cores used for the test
+
+ o Memory information (type and size)
+
+
+
+
+Tahhan, et al. Informational [Page 6]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+ o Memory DIMM configurations (quad rank performance may not be the
+ same as dual rank) in size, frequency, and slot locations
+
+ o Number of physical NICs and their details (manufacturer, versions,
+ type, and the PCI slot they are plugged into)
+
+ o NIC interrupt configuration (and any special features in use)
+
+ o PCI configuration parameters (payload size, early ACK option,
+ etc.)
+
+ Software details including:
+
+ o OS RunLevel
+
+ o OS version (for host and VNF)
+
+ o Kernel version (for host and VNF)
+
+ o GRUB boot parameters (for host and VNF)
+
+ o Hypervisor details (type and version)
+
+ o Selected vSwitch, version number, or commit ID used
+
+ o vSwitch launch command line if it has been parameterized
+
+ o Memory allocation to the vSwitch
+
+ o Which NUMA node it is using and how many memory channels
+
+ o DPDK or any other software dependency version number or commit ID
+ used
+
+ o Memory allocation to a VM - if it's from Hugepages/elsewhere
+
+ o VM storage type - snapshot, independent persistent, independent
+ non-persistent
+
+ o Number of VMs
+
+ o Number of virtual NICs (vNICs) - versions, type, and driver
+
+ o Number of virtual CPUs and their core affinity on the host
+
+ o Number of vNICs and their interrupt configurations
+
+
+
+
+
+Tahhan, et al. Informational [Page 7]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+ o Thread affinitization for the applications (including the vSwitch
+ itself) on the host
+
+ o Details of resource isolation, such as CPUs designated for Host/
+ Kernel (isolcpu) and CPUs designated for specific processes
+ (taskset).
+
+ Test traffic information:
+
+ o Test duration
+
+ o Number of flows
+
+ o Traffic type - UDP, TCP, and others
+
+ o Frame Sizes - fixed or IMIX [RFC6985] (note that with
+ [IEEE802.1ac], frames may be longer than 1500 bytes and up to 2000
+ bytes)
+
+ o Deployment Scenario - defines the communications path in the SUT
+
+3.4. Flow Classification
+
+ Virtual switches group packets into flows by processing and matching
+ particular packet or frame header information, or by matching packets
+ based on the input ports. Thus, a flow can be thought of as a
+ sequence of packets that have the same set of header field values or
+ have arrived on the same physical or logical port. Performance
+ results can vary based on the parameters the vSwitch uses to match
+ for a flow. The recommended flow classification parameters for any
+ vSwitch performance tests are: the input port (physical or logical),
+ the source MAC address, the destination MAC address, the source IP
+ address, the destination IP address, and the Ethernet protocol type
+ field (although classification may take place on other fields, such
+ as source and destination transport port numbers). It is essential
+ to increase the flow timeout time on a vSwitch before conducting any
+ performance tests that do not intend to measure the flow setup time
+ (see Section 3 of [RFC2889]). Normally, the first packet of a
+ particular stream will install the flow in the virtual switch, which
+ introduces additional latency; subsequent packets of the same flow
+ are not subject to this latency if the flow is already installed on
+ the vSwitch.
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 8]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+3.5. Benchmarks Using Baselines with Resource Isolation
+
+ This outline describes the measurement of baselines with isolated
+ resources at a high level, which is the intended approach at this
+ time.
+
+ 1. Baselines:
+
+ * Optional: Benchmark platform forwarding capability without a
+ vSwitch or VNF for at least 72 hours (serves as a means of
+ platform validation and a means to obtain the base performance
+ for the platform in terms of its maximum forwarding rate and
+ latency).
+
+ __
+ +--------------------------------------------------+ |
+ | +------------------------------------------+ | |
+ | | | | |
+ | | Simple Forwarding App | | Host
+ | | | | |
+ | +------------------------------------------+ | |
+ | | NIC | | |
+ +---+------------------------------------------+---+ __|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | Traffic Generator |
+ | |
+ +--------------------------------------------------+
+
+ Figure 1: Benchmark Platform Forwarding Capability
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 9]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+ * Benchmark VNF forwarding capability with direct connectivity
+ (vSwitch bypass, e.g., SR/IOV) for at least 72 hours (serves
+ as a means of VNF validation and a means to obtain the base
+ performance for the VNF in terms of its maximum forwarding
+ rate and latency). The metrics gathered from this test will
+ serve as a key comparison point for vSwitch bypass
+ technologies performance and vSwitch performance.
+
+ __
+ +--------------------------------------------------+ __ |
+ | +------------------------------------------+ | | |
+ | | | | Host/ |
+ | | VNF | | Guest |
+ | | | | | |
+ | +------------------------------------------+ | __| |
+ | | Passthrough/SR-IOV | | Host
+ | +------------------------------------------+ | |
+ | | NIC | | |
+ +---+------------------------------------------+---+ __|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | Traffic Generator |
+ | |
+ +--------------------------------------------------+
+
+ Figure 2: Benchmark VNF Forwarding Capability
+
+ * Benchmarking with isolated resources alone and with other
+ resources (both hardware and software) disabled; for example,
+ vSwitch and VM are SUT.
+
+ * Benchmarking with isolated resources alone, thus leaving some
+ resources unused.
+
+ * Benchmarking with isolated resources and all resources
+ occupied.
+
+ 2. Next Steps:
+
+ * Limited sharing
+
+ * Production scenarios
+
+ * Stressful scenarios
+
+
+
+
+Tahhan, et al. Informational [Page 10]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+4. VSPERF Specification Summary
+
+ The overall specification in preparation is referred to as a Level
+ Test Design (LTD) document, which will contain a suite of performance
+ tests. The base performance tests in the LTD are based on the
+ pre-existing specifications developed by the BMWG to test the
+ performance of physical switches. These specifications include:
+
+ o Benchmarking Methodology for Network Interconnect Devices
+ [RFC2544]
+
+ o Benchmarking Methodology for LAN Switching [RFC2889]
+
+ o Device Reset Characterization [RFC6201]
+
+ o Packet Delay Variation Applicability Statement [RFC5481]
+
+ The two most recent RFCs above ([RFC6201] and [RFC5481]) are being
+ applied in benchmarking for the first time and represent a
+ development challenge for test equipment developers. Fortunately,
+ many members of the testing system community have engaged on the
+ VSPERF project, including an open source test system.
+
+ In addition to this, the LTD also reuses the terminology defined by:
+
+ o Benchmarking Terminology for LAN Switching Devices [RFC2285]
+
+ It is recommended that these references be included in future
+ benchmarking specifications:
+
+ o Methodology for IP Multicast Benchmarking [RFC3918]
+
+ o Packet Reordering Metrics [RFC4737]
+
+ As one might expect, the most fundamental internetworking
+ characteristics of throughput and latency remain important when the
+ switch is virtualized, and these benchmarks figure prominently in the
+ specification.
+
+ When considering characteristics important to "telco" network
+ functions, additional performance metrics are needed. In this case,
+ the project specifications have referenced metrics from the IETF IP
+ Performance Metrics (IPPM) literature. This means that the latency
+ test described in [RFC2544] is replaced by measurement of a metric
+ derived from IPPM's [RFC7679], where a set of statistical summaries
+ will be provided (mean, max, min, and percentiles). Further metrics
+ planned to be benchmarked include packet delay variation as defined
+ by [RFC5481], reordering, burst behaviour, DUT availability, DUT
+
+
+
+Tahhan, et al. Informational [Page 11]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+ capacity, and packet loss in long-term testing at the throughput
+ level, where some low level of background loss may be present and
+ characterized.
+
+ Tests have been designed to collect the metrics below:
+
+ o Throughput tests are designed to measure the maximum forwarding
+ rate (in frames per second, fps) and bit rate (in Mbps) for a
+ constant load (as defined by [RFC1242]) without traffic loss.
+
+ o Packet and frame-delay distribution tests are designed to measure
+ the average minimum and maximum packet (and/or frame) delay for
+ constant loads.
+
+ o Packet delay tests are designed to understand latency distribution
+ for different packet sizes and to uncover outliers over an
+ extended test run.
+
+ o Scalability tests are designed to understand how the virtual
+ switch performs with an increasing number of flows, number of
+ active ports, configuration complexity of the forwarding logic,
+ etc.
+
+ o Stream performance tests (with TCP or UDP) are designed to measure
+ bulk data transfer performance, i.e., how fast systems can send
+ and receive data through the switch.
+
+ o Control-path and data-path coupling tests are designed to
+ understand how closely the data path and the control path are
+ coupled, as well as the effect of this coupling on the performance
+ of the DUT (for example, delay of the initial packet of a flow).
+
+ o CPU and memory consumption tests are designed to understand the
+ virtual switch's footprint on the system and are conducted as
+ auxiliary measurements with the benchmarks above. They include
+ CPU utilization, cache utilization, and memory footprint.
+
+ o The so-called "soak" tests, where the selected test is conducted
+ over a long period of time (with an ideal duration of 24 hours but
+ only long enough to determine that stability issues exist when
+ found; there is no requirement to continue a test when a DUT
+ exhibits instability over time). The key performance
+ characteristics and benchmarks for a DUT are determined (using
+ short duration tests) prior to conducting soak tests. The purpose
+ of soak tests is to capture transient changes in performance,
+ which may occur due to infrequent processes, memory leaks, or the
+ low-probability coincidence of two or more processes. The
+ stability of the DUT is the paramount consideration, so
+
+
+
+Tahhan, et al. Informational [Page 12]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+ performance must be evaluated periodically during continuous
+ testing, and this results in use of frame rate metrics [RFC2889]
+ instead of throughput [RFC2544] (which requires stopping traffic
+ to allow time for all traffic to exit internal queues), for
+ example.
+
+ Additional test specification development should include:
+
+ o Request/response performance tests (with TCP or UDP), which
+ measure the transaction rate through the switch.
+
+ o Noisy neighbor tests, in order to understand the effects of
+ resource sharing on the performance of a virtual switch.
+
+ o Tests derived from examination of ETSI NFV Draft GS IFA003
+ requirements [IFA003] on characterization of acceleration
+ technologies applied to vSwitches.
+
+ The flexibility of deployment of a virtual switch within a network
+ means that it is necessary to characterize the performance of a
+ vSwitch in various deployment scenarios. The deployment scenarios
+ under consideration are shown in the following figures:
+
+ __
+ +--------------------------------------------------+ |
+ | +--------------------+ | |
+ | | | | |
+ | | v | | Host
+ | +--------------+ +--------------+ | |
+ | | PHY Port | vSwitch | PHY Port | | |
+ +---+--------------+------------+--------------+---+ __|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | Traffic Generator |
+ | |
+ +--------------------------------------------------+
+
+ Figure 3: Physical Port to Virtual Switch to Physical Port
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 13]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+ __
+ +---------------------------------------------------+ |
+ | | |
+ | +-------------------------------------------+ | |
+ | | Application | | |
+ | +-------------------------------------------+ | |
+ | ^ : | |
+ | | | | | Guest
+ | : v | |
+ | +---------------+ +---------------+ | |
+ | | Logical Port 0| | Logical Port 1| | |
+ +---+---------------+-----------+---------------+---+ __|
+ ^ :
+ | |
+ : v __
+ +---+---------------+----------+---------------+---+ |
+ | | Logical Port 0| | Logical Port 1| | |
+ | +---------------+ +---------------+ | |
+ | ^ : | |
+ | | | | | Host
+ | : v | |
+ | +--------------+ +--------------+ | |
+ | | PHY Port | vSwitch | PHY Port | | |
+ +---+--------------+------------+--------------+---+ __|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | Traffic Generator |
+ | |
+ +--------------------------------------------------+
+
+ Figure 4: Physical Port to Virtual Switch to VNF to Virtual Switch to
+ Physical Port
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 14]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+ __
+ +----------------------+ +----------------------+ |
+ | Guest 1 | | Guest 2 | |
+ | +---------------+ | | +---------------+ | |
+ | | Application | | | | Application | | |
+ | +---------------+ | | +---------------+ | |
+ | ^ | | | ^ | | |
+ | | v | | | v | | Guests
+ | +---------------+ | | +---------------+ | |
+ | | Logical Ports | | | | Logical Ports | | |
+ | | 0 1 | | | | 0 1 | | |
+ +---+---------------+--+ +---+---------------+--+__|
+ ^ : ^ :
+ | | | |
+ : v : v _
+ +---+---------------+---------+---------------+--+ |
+ | | 0 1 | | 3 4 | | |
+ | | Logical Ports | | Logical Ports | | |
+ | +---------------+ +---------------+ | |
+ | ^ | ^ | | | Host
+ | | \-----------------/ v | |
+ | +--------------+ +--------------+ | |
+ | | PHY Ports | vSwitch | PHY Ports | | |
+ +---+--------------+----------+--------------+---+_|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | Traffic Generator |
+ | |
+ +--------------------------------------------------+
+
+ Figure 5: Physical Port to Virtual Switch to VNF to Virtual Switch to
+ VNF to Virtual Switch to Physical Port
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 15]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+ __
+ +---------------------------------------------------+ |
+ | | |
+ | +-------------------------------------------+ | |
+ | | Application | | |
+ | +-------------------------------------------+ | |
+ | ^ | |
+ | | | | Guest
+ | : | |
+ | +---------------+ | |
+ | | Logical Port 0| | |
+ +---+---------------+-------------------------------+ __|
+ ^
+ |
+ : __
+ +---+---------------+------------------------------+ |
+ | | Logical Port 0| | |
+ | +---------------+ | |
+ | ^ | |
+ | | | | Host
+ | : | |
+ | +--------------+ | |
+ | | PHY Port | vSwitch | |
+ +---+--------------+------------ -------------- ---+ __|
+ ^
+ |
+ :
+ +--------------------------------------------------+
+ | |
+ | Traffic Generator |
+ | |
+ +--------------------------------------------------+
+
+ Figure 6: Physical Port to Virtual Switch to VNF
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 16]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+ __
+ +---------------------------------------------------+ |
+ | | |
+ | +-------------------------------------------+ | |
+ | | Application | | |
+ | +-------------------------------------------+ | |
+ | : | |
+ | | | | Guest
+ | v | |
+ | +---------------+ | |
+ | | Logical Port | | |
+ +-------------------------------+---------------+---+ __|
+ :
+ |
+ v __
+ +------------------------------+---------------+---+ |
+ | | Logical Port | | |
+ | +---------------+ | |
+ | : | |
+ | | | | Host
+ | v | |
+ | +--------------+ | |
+ | vSwitch | PHY Port | | |
+ +-------------------------------+--------------+---+ __|
+ :
+ |
+ v
+ +--------------------------------------------------+
+ | |
+ | Traffic Generator |
+ | |
+ +--------------------------------------------------+
+
+ Figure 7: VNF to Virtual Switch to Physical Port
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 17]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+ __
+ +----------------------+ +----------------------+ |
+ | Guest 1 | | Guest 2 | |
+ | +---------------+ | | +---------------+ | |
+ | | Application | | | | Application | | |
+ | +---------------+ | | +---------------+ | |
+ | | | | ^ | |
+ | v | | | | | Guests
+ | +---------------+ | | +---------------+ | |
+ | | Logical Ports | | | | Logical Ports | | |
+ | | 0 | | | | 0 | | |
+ +---+---------------+--+ +---+---------------+--+__|
+ : ^
+ | |
+ v : _
+ +---+---------------+---------+---------------+--+ |
+ | | 1 | | 1 | | |
+ | | Logical Ports | | Logical Ports | | |
+ | +---------------+ +---------------+ | |
+ | | ^ | | Host
+ | \-----------------/ | |
+ | | |
+ | vSwitch | |
+ +------------------------------------------------+_|
+
+ Figure 8: VNF to Virtual Switch to VNF
+
+ A set of deployment scenario figures is available on the VSPERF "Test
+ Methodology" wiki page [TestTopo].
+
+5. 3x3 Matrix Coverage
+
+ This section organizes the many existing test specifications into the
+ "3x3" matrix (introduced in [RFC8172]). Because the LTD
+ specification ID names are quite long, this section is organized into
+ lists for each occupied cell of the matrix (not all are occupied;
+ also, the matrix has grown to 3x4 to accommodate scale metrics when
+ displaying the coverage of many metrics/benchmarks). The current
+ version of the LTD specification is available; see [LTD].
+
+ The tests listed below assess the activation of paths in the data
+ plane rather than the control plane.
+
+ A complete list of tests with short summaries is available on the
+ VSPERF "LTD Test Spec Overview" wiki page [LTDoverV].
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 18]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+5.1. Speed of Activation
+
+ o Activation.RFC2889.AddressLearningRate
+
+ o PacketLatency.InitialPacketProcessingLatency
+
+5.2. Accuracy of Activation
+
+ o CPDP.Coupling.Flow.Addition
+
+5.3. Reliability of Activation
+
+ o Throughput.RFC2544.SystemRecoveryTime
+
+ o Throughput.RFC2544.ResetTime
+
+5.4. Scale of Activation
+
+ o Activation.RFC2889.AddressCachingCapacity
+
+5.5. Speed of Operation
+
+ o Throughput.RFC2544.PacketLossRate
+
+ o Stress.RFC2544.0PacketLoss
+
+ o Throughput.RFC2544.PacketLossRateFrameModification
+
+ o Throughput.RFC2544.BackToBackFrames
+
+ o Throughput.RFC2889.MaxForwardingRate
+
+ o Throughput.RFC2889.ForwardPressure
+
+ o Throughput.RFC2889.BroadcastFrameForwarding
+
+ o Throughput.RFC2544.WorstN-BestN
+
+ o Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
+
+5.6. Accuracy of Operation
+
+ o Throughput.RFC2889.ErrorFramesFiltering
+
+ o Throughput.RFC2544.Profile
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 19]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+5.7. Reliability of Operation
+
+ o Throughput.RFC2889.Soak
+
+ o Throughput.RFC2889.SoakFrameModification
+
+ o PacketDelayVariation.RFC3393.Soak
+
+5.8. Scalability of Operation
+
+ o Scalability.RFC2544.0PacketLoss
+
+ o MemoryBandwidth.RFC2544.0PacketLoss.Scalability
+
+ o Scalability.VNF.RFC2544.PacketLossProfile
+
+ o Scalability.VNF.RFC2544.PacketLossRatio
+
+5.9. Summary
+
+ |---------------------------------------------------------------------|
+ | | | | | |
+ | | SPEED | ACCURACY | RELIABILITY | SCALE |
+ | | | | | |
+ |---------------------------------------------------------------------|
+ | | | | | |
+ | Activation | X | X | X | X |
+ | | | | | |
+ |---------------------------------------------------------------------|
+ | | | | | |
+ | Operation | X | X | X | X |
+ | | | | | |
+ |---------------------------------------------------------------------|
+ | | | | | |
+ | De-activation| | | | |
+ | | | | | |
+ |---------------------------------------------------------------------|
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 20]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+6. Security Considerations
+
+ Benchmarking activities as described in this memo are limited to
+ technology characterization of a Device Under Test/System Under Test
+ (DUT/SUT) using controlled stimuli in a laboratory environment with
+ dedicated address space and the constraints specified in the sections
+ above.
+
+ The benchmarking network topology will be an independent test setup
+ and MUST NOT be connected to devices that may forward the test
+ traffic into a production network or misroute traffic to the test
+ management network.
+
+ Further, benchmarking is performed on a "black-box" basis and relies
+ solely on measurements observable external to the DUT/SUT.
+
+ Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
+ benchmarking purposes. Any implications for network security arising
+ from the DUT/SUT SHOULD be identical in the lab and in production
+ networks.
+
+7. References
+
+7.1. Normative References
+
+ [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
+ Requirement Levels", BCP 14, RFC 2119,
+ DOI 10.17487/RFC2119, March 1997,
+ <https://www.rfc-editor.org/info/rfc2119>.
+
+ [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN
+ Switching Devices", RFC 2285, DOI 10.17487/RFC2285,
+ February 1998, <https://www.rfc-editor.org/info/rfc2285>.
+
+ [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
+ Network Interconnect Devices", RFC 2544,
+ DOI 10.17487/RFC2544, March 1999,
+ <https://www.rfc-editor.org/info/rfc2544>.
+
+ [RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology
+ for LAN Switching Devices", RFC 2889,
+ DOI 10.17487/RFC2889, August 2000,
+ <https://www.rfc-editor.org/info/rfc2889>.
+
+ [RFC3918] Stopp, D. and B. Hickman, "Methodology for IP Multicast
+ Benchmarking", RFC 3918, DOI 10.17487/RFC3918, October
+ 2004, <https://www.rfc-editor.org/info/rfc3918>.
+
+
+
+
+Tahhan, et al. Informational [Page 21]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+ [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov,
+ S., and J. Perser, "Packet Reordering Metrics", RFC 4737,
+ DOI 10.17487/RFC4737, November 2006,
+ <https://www.rfc-editor.org/info/rfc4737>.
+
+ [RFC6201] Asati, R., Pignataro, C., Calabria, F., and C. Olvera,
+ "Device Reset Characterization", RFC 6201,
+ DOI 10.17487/RFC6201, March 2011,
+ <https://www.rfc-editor.org/info/rfc6201>.
+
+ [RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet
+ Sizes for Additional Testing", RFC 6985,
+ DOI 10.17487/RFC6985, July 2013,
+ <https://www.rfc-editor.org/info/rfc6985>.
+
+ [RFC7679] Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton,
+ Ed., "A One-Way Delay Metric for IP Performance Metrics
+ (IPPM)", STD 81, RFC 7679, DOI 10.17487/RFC7679, January
+ 2016, <https://www.rfc-editor.org/info/rfc7679>.
+
+ [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
+ 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
+ May 2017, <https://www.rfc-editor.org/info/rfc8174>.
+
+7.2. Informative References
+
+ [BENCHMARK-METHOD]
+ Huang, L., Ed., Rong, G., Ed., Mandeville, B., and B.
+ Hickman, "Benchmarking Methodology for Virtualization
+ Network Performance", Work in Progress, draft-huang-bmwg-
+ virtual-network-performance-03, July 2017.
+
+ [DanubeRel]
+ OPNFV, "Danube",
+ <https://wiki.opnfv.org/display/SWREL/Danube>.
+
+ [IEEE802.1ac]
+ IEEE, "IEEE Standard for Local and metropolitan area
+ networks -- Media Access Control (MAC) Service
+ Definition", IEEE 802.1AC-2016,
+ DOI 10.1109/IEEESTD.2017.7875381, 2016,
+ <https://standards.ieee.org/findstds/
+ standard/802.1AC-2016.html>.
+
+ [IEEE829] IEEE, "IEEE Standard for Software and System Test
+ Documentation", IEEE 829-2008,
+ DOI 10.1109/IEEESTD.2008.4578383,
+ <http://ieeexplore.ieee.org/document/4578383/>.
+
+
+
+Tahhan, et al. Informational [Page 22]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+ [IFA003] ETSI, "Network Functions Virtualisation (NFV);
+ Acceleration Technologies; vSwitch Benchmarking and
+ Acceleration Specification", ETSI GS NFV-IFA 003 V2.1.1,
+ April 2016, <http://www.etsi.org/deliver/etsi_gs/
+ NFV-IFA/001_099/003/02.01.01_60/
+ gs_NFV-IFA003v020101p.pdf>.
+
+ [LTD] Tahhan, M., "VSPERF Level Test Design (LTD)",
+ <http://docs.opnfv.org/en/stable-danube/
+ submodules/vswitchperf/docs/testing/developer/
+ requirements/vswitchperf_ltd.html#>.
+
+ [LTDoverV] Morton, A., "LTD Test Spec Overview",
+ <https://wiki.opnfv.org/display/vsperf/
+ LTD+Test+Spec+Overview>.
+
+ [OPNFV] OPNFV, "OPNFV", <https://www.opnfv.org/>.
+
+ [RFC1242] Bradner, S., "Benchmarking Terminology for Network
+ Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242,
+ July 1991, <https://www.rfc-editor.org/info/rfc1242>.
+
+ [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation
+ Applicability Statement", RFC 5481, DOI 10.17487/RFC5481,
+ March 2009, <https://www.rfc-editor.org/info/rfc5481>.
+
+ [RFC8172] Morton, A., "Considerations for Benchmarking Virtual
+ Network Functions and Their Infrastructure", RFC 8172,
+ DOI 10.17487/RFC8172, July 2017,
+ <https://www.rfc-editor.org/info/rfc8172>.
+
+ [TestTopo] Snyder, E., "Test Methodology",
+ <https://wiki.opnfv.org/display/vsperf/Test+Methodology>.
+
+ [VSPERFhome]
+ Tahhan, M., "VSPERF Home",
+ <https://wiki.opnfv.org/display/vsperf/VSperf+Home>.
+
+Acknowledgements
+
+ The authors appreciate and acknowledge comments from Scott Bradner,
+ Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik,
+ Christian Trautman, Benoit Claise, and others for their reviews.
+
+ We also acknowledge the early work in [BENCHMARK-METHOD] and useful
+ discussion with the authors.
+
+
+
+
+
+Tahhan, et al. Informational [Page 23]
+
+RFC 8204 Benchmarking vSwitches September 2017
+
+
+Authors' Addresses
+
+ Maryam Tahhan
+ Intel
+
+ Email: maryam.tahhan@intel.com
+
+
+ Billy O'Mahony
+ Intel
+
+ Email: billy.o.mahony@intel.com
+
+
+ Al Morton
+ AT&T Labs
+ 200 Laurel Avenue South
+ Middletown, NJ 07748
+ United States of America
+
+ Phone: +1 732 420 1571
+ Fax: +1 732 368 1192
+ Email: acmorton@att.com
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Informational [Page 24]
+