diff options
Diffstat (limited to 'doc/rfc/rfc8239.txt')
| -rw-r--r-- | doc/rfc/rfc8239.txt | 1067 |
1 files changed, 1067 insertions, 0 deletions
diff --git a/doc/rfc/rfc8239.txt b/doc/rfc/rfc8239.txt new file mode 100644 index 0000000..7072467 --- /dev/null +++ b/doc/rfc/rfc8239.txt @@ -0,0 +1,1067 @@ + + + + + + +Internet Engineering Task Force (IETF) L. Avramov +Request for Comments: 8239 Google +Category: Informational J. Rapp +ISSN: 2070-1721 VMware + August 2017 + + + Data Center Benchmarking Methodology + +Abstract + + The purpose of this informational document is to establish test and + evaluation methodology and measurement techniques for physical + network equipment in the data center. RFC 8238 is a prerequisite for + this document, as it contains terminology that is considered + normative. Many of these terms and methods may be applicable beyond + the scope of this document as the technologies originally applied in + the data center are deployed elsewhere. + +Status of This Memo + + This document is not an Internet Standards Track specification; it is + published for informational purposes. + + This document is a product of the Internet Engineering Task Force + (IETF). It represents the consensus of the IETF community. It has + received public review and has been approved for publication by the + Internet Engineering Steering Group (IESG). Not all documents + approved by the IESG are a candidate for any level of Internet + Standard; see Section 2 of RFC 7841. + + Information about the current status of this document, any errata, + and how to provide feedback on it may be obtained at + http://www.rfc-editor.org/info/rfc8239. + + + + + + + + + + + + + + + + + +Avramov & Rapp Informational [Page 1] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + +Copyright Notice + + Copyright (c) 2017 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (http://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the Simplified BSD License. + +Table of Contents + + 1. Introduction ....................................................3 + 1.1. Requirements Language ......................................4 + 1.2. Methodology Format and Repeatability Recommendation ........4 + 2. Line-Rate Testing ...............................................4 + 2.1. Objective ..................................................4 + 2.2. Methodology ................................................4 + 2.3. Reporting Format ...........................................5 + 3. Buffering Testing ...............................................6 + 3.1. Objective ..................................................6 + 3.2. Methodology ................................................7 + 3.3. Reporting Format ...........................................9 + 4. Microburst Testing .............................................10 + 4.1. Objective .................................................10 + 4.2. Methodology ...............................................10 + 4.3. Reporting Format ..........................................11 + 5. Head-of-Line Blocking ..........................................12 + 5.1. Objective .................................................12 + 5.2. Methodology ...............................................12 + 5.3. Reporting Format ..........................................14 + 6. Incast Stateful and Stateless Traffic ..........................15 + 6.1. Objective .................................................15 + 6.2. Methodology ...............................................15 + 6.3. Reporting Format ..........................................17 + 7. Security Considerations ........................................17 + 8. IANA Considerations ............................................17 + 9. References .....................................................18 + 9.1. Normative References ......................................18 + 9.2. Informative References ....................................18 + Acknowledgments ...................................................19 + Authors' Addresses ................................................19 + + + + +Avramov & Rapp Informational [Page 2] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + +1. Introduction + + Traffic patterns in the data center are not uniform and are + constantly changing. They are dictated by the nature and variety of + applications utilized in the data center. They can be largely + east-west traffic flows (server to server inside the data center) in + one data center and north-south (from the outside of the data center + to the server) in another, while others may combine both. Traffic + patterns can be bursty in nature and contain many-to-one, + many-to-many, or one-to-many flows. Each flow may also be small and + latency sensitive or large and throughput sensitive while containing + a mix of UDP and TCP traffic. All of these can coexist in a single + cluster and flow through a single network device simultaneously. + Benchmarking tests for network devices have long used [RFC1242], + [RFC2432], [RFC2544], [RFC2889], and [RFC3918], which have largely + been focused around various latency attributes and throughput + [RFC2889] of the Device Under Test (DUT) being benchmarked. These + standards are good at measuring theoretical throughput, forwarding + rates, and latency under testing conditions; however, they do not + represent real traffic patterns that may affect these networking + devices. + + Currently, typical data center networking devices are + characterized by: + + - High port density (48 ports or more). + + - High speed (currently, up to 100 GB/s per port). + + - High throughput (line rate on all ports for Layer 2 and/or + Layer 3). + + - Low latency (in the microsecond or nanosecond range). + + - Low amount of buffer (in the MB range per networking device). + + - Layer 2 and Layer 3 forwarding capability (Layer 3 not mandatory). + + This document provides a methodology for benchmarking data center + physical network equipment DUTs, including congestion scenarios, + switch buffer analysis, microburst, and head-of-line blocking, while + also using a wide mix of traffic conditions. [RFC8238] is a + prerequisite for this document, as it contains terminology that is + considered normative. + + + + + + + +Avramov & Rapp Informational [Page 3] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + +1.1. Requirements Language + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and + "OPTIONAL" in this document are to be interpreted as described in + BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all + capitals, as shown here. + +1.2. Methodology Format and Repeatability Recommendation + + The following format is used in Sections 2 through 6 of this + document: + + - Objective + + - Methodology + + - Reporting Format + + For each test methodology described in this document, it is critical + that repeatability of the results be obtained. The recommendation is + to perform enough iterations of the given test and to make sure that + the result is consistent. This is especially important in the + context of the tests described in Section 3, as the buffering testing + has historically been the least reliable. The number of iterations + SHOULD be explicitly reported. The relative standard deviation + SHOULD be below 10%. + +2. Line-Rate Testing + +2.1. Objective + + The objective of this test is to provide a "maximum rate" test for + the performance values for throughput, latency, and jitter. It is + meant to provide (1) the tests to perform and (2) methodology for + verifying that a DUT is capable of forwarding packets at line rate + under non-congested conditions. + +2.2. Methodology + + A traffic generator SHOULD be connected to all ports on the DUT. Two + tests MUST be conducted: (1) a port-pair test [RFC2544] [RFC3918] and + (2) a test using a full-mesh DUT [RFC2889] [RFC3918]. + + For all tests, the traffic generator's sending rate MUST be less than + or equal to 99.98% of the nominal value of the line rate (with no + further Parts Per Million (PPM) adjustment to account for interface + clock tolerances), to ensure stressing of the DUT in reasonable + + + +Avramov & Rapp Informational [Page 4] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + + worst-case conditions (see [RFC8238], Section 5 for more details). + Test results at a lower rate MAY be provided for better understanding + of performance increase in terms of latency and jitter when the rate + is lower than 99.98%. The receiving rate of the traffic SHOULD be + captured during this test as a percentage of line rate. + + The test MUST provide the statistics of minimum, average, and + maximum of the latency distribution, for the exact same iteration of + the test. + + The test MUST provide the statistics of minimum, average, and maximum + of the jitter distribution, for the exact same iteration of the test. + + Alternatively, when a traffic generator cannot be connected to all + ports on the DUT, a snake test MUST be used for line-rate testing, + excluding latency and jitter, as those would become irrelevant. The + snake test is performed as follows: + + - Connect the first and last port of the DUT to a traffic generator. + + - Connect, back to back and sequentially, all the ports in between: + port 2 to port 3, port 4 to port 5, etc., to port N-2 to port N-1, + where N is the total number of ports of the DUT. + + - Configure port 1 and port 2 in the same VLAN X, port 3 and port 4 + in the same VLAN Y, etc., and port N-1 and port N in the same + VLAN Z. + + This snake test provides the capability to test line rate for Layer 2 + and Layer 3 [RFC2544] [RFC3918] in instances where a traffic + generator with only two ports is available. Latency and jitter are + not to be considered for this test. + +2.3. Reporting Format + + The report MUST include the following: + + - Physical-layer calibration information, as defined in [RFC8238], + Section 4. + + - Number of ports used. + + - Reading for "throughput received as a percentage of bandwidth", + while sending 99.98% of the nominal value of the line rate on each + port, for each packet size from 64 bytes to 9216 bytes. As + guidance, with a packet-size increment of 64 bytes between each + iteration being ideal, 256-byte and 512-byte packets are also + + + + +Avramov & Rapp Informational [Page 5] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + + often used. The most common packet-size ordering for the report + is 64 bytes, 128 bytes, 256 bytes, 512 bytes, 1024 bytes, + 1518 bytes, 4096 bytes, 8000 bytes, and 9216 bytes. + + The pattern for testing can be expressed using [RFC6985]. + + - Throughput needs to be expressed as a percentage of total + transmitted frames. + + - Packet drops MUST be expressed as a count of packets and SHOULD be + expressed as a percentage of line rate. + + - For latency and jitter, values are expressed in units of time + (usually microseconds or nanoseconds), reading across packet sizes + from 64 bytes to 9216 bytes. + + - For latency and jitter, provide minimum, average, and maximum + values. If different iterations are done to gather the minimum, + average, and maximum values, this SHOULD be specified in the + report, along with a justification for why the information could + not have been gathered in the same test iteration. + + - For jitter, a histogram describing the population of packets + measured per latency or latency buckets is RECOMMENDED. + + - The tests for throughput, latency, and jitter MAY be conducted as + individual independent trials, with proper documentation provided + in the report, but SHOULD be conducted at the same time. + + - The methodology assumes that the DUT has at least nine ports, as + certain methodologies require nine or more ports. + +3. Buffering Testing + +3.1. Objective + + The objective of this test is to measure the size of the buffer of a + DUT under typical/many/multiple conditions. Buffer architectures + between multiple DUTs can differ and include egress buffering, shared + egress buffering SoC (Switch-on-Chip), ingress buffering, or a + combination thereof. The test methodology covers the buffer + measurement, regardless of buffer architecture used in the DUT. + + + + + + + + + +Avramov & Rapp Informational [Page 6] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + +3.2. Methodology + + A traffic generator MUST be connected to all ports on the DUT. The + methodology for measuring buffering for a data center switch is based + on using known congestion of known fixed packet size, along with + maximum latency value measurements. The maximum latency will + increase until the first packet drop occurs. At this point, the + maximum latency value will remain constant. This is the point of + inflection of this maximum latency change to a constant value. There + MUST be multiple ingress ports receiving a known amount of frames at + a known fixed size, destined for the same egress port in order to + create a known congestion condition. The total amount of packets + sent from the oversubscribed port minus one, multiplied by the packet + size, represents the maximum port buffer size at the measured + inflection point. + + Note that the tests described in procedures 1), 2), 3), and 4) in + this section have iterations called "first iteration", "second + iteration", and "last iteration". The idea is to show the first + two iterations so the reader understands the logic of how to keep + incrementing the iterations. The last iteration shows the end state + of the variables. + + 1) Measure the highest buffer efficiency. + + o First iteration: Ingress port 1 sending 64-byte packets at line + rate to egress port 2, while port 3 is sending a known low + amount of oversubscription traffic (1% recommended) with the + same packet size of 64 bytes to egress port 2. Measure the + buffer size value of the number of frames sent from the port + sending the oversubscribed traffic up to the inflection point + multiplied by the frame size. + + o Second iteration: Ingress port 1 sending 65-byte packets at + line rate to egress port 2, while port 3 is sending a known low + amount of oversubscription traffic (1% recommended) with the + same packet size of 65 bytes to egress port 2. Measure the + buffer size value of the number of frames sent from the port + sending the oversubscribed traffic up to the inflection point + multiplied by the frame size. + + o Last iteration: Ingress port 1 sending packets of size B bytes + at line rate to egress port 2, while port 3 is sending a known + low amount of oversubscription traffic (1% recommended) with + the same packet size of B bytes to egress port 2. Measure the + buffer size value of the number of frames sent from the port + sending the oversubscribed traffic up to the inflection point + multiplied by the frame size. + + + +Avramov & Rapp Informational [Page 7] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + + When the B value is found to provide the largest buffer size, then + size B allows the highest buffer efficiency. + + 2) Measure maximum port buffer size. + + At fixed packet size B as determined in procedure 1), for a fixed + default Differentiated Services Code Point (DSCP) / Class of + Service (CoS) value of 0 and for unicast traffic, proceed with the + following: + + o First iteration: Ingress port 1 sending line rate to egress + port 2, while port 3 is sending a known low amount of + oversubscription traffic (1% recommended) with the same packet + size to egress port 2. Measure the buffer size value by + multiplying the number of extra frames sent by the frame size. + + o Second iteration: Ingress port 2 sending line rate to egress + port 3, while port 4 is sending a known low amount of + oversubscription traffic (1% recommended) with the same packet + size to egress port 3. Measure the buffer size value by + multiplying the number of extra frames sent by the frame size. + + o Last iteration: Ingress port N-2 sending line rate to egress + port N-1, while port N is sending a known low amount of + oversubscription traffic (1% recommended) with the same packet + size to egress port N. Measure the buffer size value by + multiplying the number of extra frames sent by the frame size. + + This test series MAY be repeated using all different DSCP/CoS + values of traffic, and then using multicast traffic, in order to + find out if there is any DSCP/CoS impact on the buffer size. + + 3) Measure maximum port pair buffer sizes. + + o First iteration: Ingress port 1 sending line rate to egress + port 2, ingress port 3 sending line rate to egress port 4, etc. + Ingress port N-1 and port N will oversubscribe, at 1% of line + rate, egress port 2 and port 3, respectively. Measure the + buffer size value by multiplying the number of extra frames + sent by the frame size for each egress port. + + o Second iteration: Ingress port 1 sending line rate to egress + port 2, ingress port 3 sending line rate to egress port 4, etc. + Ingress port N-1 and port N will oversubscribe, at 1% of line + rate, egress port 4 and port 5, respectively. Measure the + buffer size value by multiplying the number of extra frames + sent by the frame size for each egress port. + + + + +Avramov & Rapp Informational [Page 8] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + + o Last iteration: Ingress port 1 sending line rate to egress + port 2, ingress port 3 sending line rate to egress port 4, etc. + Ingress port N-1 and port N will oversubscribe, at 1% of line + rate, egress port N-3 and port N-2, respectively. Measure the + buffer size value by multiplying the number of extra frames + sent by the frame size for each egress port. + + This test series MAY be repeated using all different DSCP/CoS + values of traffic and then using multicast traffic. + + 4) Measure maximum DUT buffer size with many-to-one ports. + + o First iteration: Ingress ports 1,2,... N-1 each sending + [(1/[N-1])*99.98]+[1/[N-1]] % of line rate per port to egress + port N. + + o Second iteration: Ingress ports 2,... N each sending + [(1/[N-1])*99.98]+[1/[N-1]] % of line rate per port to egress + port 1. + + o Last iteration: Ingress ports N,1,2...N-2 each sending + [(1/[N-1])*99.98]+[1/[N-1]] % of line rate per port to egress + port N-1. + + This test series MAY be repeated using all different CoS values of + traffic and then using multicast traffic. + + Unicast traffic, and then multicast traffic, SHOULD be used in order + to determine the proportion of buffer for the documented selection of + tests. Also, the CoS value for the packets SHOULD be provided for + each test iteration, as the buffer allocation size MAY differ per CoS + value. It is RECOMMENDED that the ingress and egress ports be varied + in a random but documented fashion in multiple tests in order to + measure the buffer size for each port of the DUT. + +3.3. Reporting Format + + The report MUST include the following: + + - The packet size used for the most efficient buffer used, + along with the DSCP/CoS value. + + - The maximum port buffer size for each port. + + - The maximum DUT buffer size. + + - The packet size used in the test. + + + + +Avramov & Rapp Informational [Page 9] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + + - The amount of oversubscription, if different than 1%. + + - The number of ingress and egress ports, along with their location + on the DUT. + + - The repeatability of the test needs to be indicated: the number of + iterations of the same test and the percentage of variation + between results for each of the tests (min, max, avg). + + The percentage of variation is a metric providing a sense of how big + the difference is between the measured value and the previous values. + + For example, for a latency test where the minimum latency is + measured, the percentage of variation (PV) of the minimum latency + will indicate by how much this value has varied between the current + test executed and the previous one. + + PV = ((x2-x1)/x1)*100, where x2 is the minimum latency value in the + current test and x1 is the minimum latency value obtained in the + previous test. + + The same formula is used for maximum and average variations measured. + +4. Microburst Testing + +4.1. Objective + + The objective of this test is to find the maximum amount of packet + bursts that a DUT can sustain under various configurations. + + This test provides additional methodology that supplements the tests + described in [RFC1242], [RFC2432], [RFC2544], [RFC2889], and + [RFC3918]. + + - All bursts should be sent with 100% intensity. Note: "Intensity" + is defined in [RFC8238], Section 6.1.1. + + - All ports of the DUT must be used for this test. + + - It is recommended that all ports be tested simultaneously. + +4.2. Methodology + + A traffic generator MUST be connected to all ports on the DUT. In + order to cause congestion, two or more ingress ports MUST send bursts + of packets destined for the same egress port. The simplest of the + setups would be two ingress ports and one egress port (2 to 1). + + + + +Avramov & Rapp Informational [Page 10] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + + The burst MUST be sent with an intensity (as defined in [RFC8238], + Section 6.1.1) of 100%, meaning that the burst of packets will be + sent with a minimum interpacket gap. The amount of packets contained + in the burst will be trial variable and increase until there is a + non-zero packet loss measured. The aggregate amount of packets from + all the senders will be used to calculate the maximum microburst + amount that the DUT can sustain. + + It is RECOMMENDED that the ingress and egress ports be varied in + multiple tests in order to measure the maximum microburst capacity. + + The intensity of a microburst (see [RFC8238], Section 6.1.1) MAY be + varied in order to obtain the microburst capacity at various + ingress rates. + + It is RECOMMENDED that all ports on the DUT be tested simultaneously, + and in various configurations, in order to understand all the + combinations of ingress ports, egress ports, and intensities. + + An example would be: + + o First iteration: N-1 ingress ports sending to one egress port. + + o Second iteration: N-2 ingress ports sending to two egress ports. + + o Last iteration: Two ingress ports sending to N-2 egress ports. + +4.3. Reporting Format + + The report MUST include the following: + + - The maximum number of packets received per ingress port with the + maximum burst size obtained with zero packet loss. + + - The packet size used in the test. + + - The number of ingress and egress ports, along with their location + on the DUT. + + - The repeatability of the test needs to be indicated: the number of + iterations of the same test and the percentage of variation + between results (min, max, avg). + + + + + + + + + +Avramov & Rapp Informational [Page 11] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + +5. Head-of-Line Blocking + +5.1. Objective + + Head-of-line blocking (HOLB) is a performance-limiting phenomenon + that occurs when packets are held up by the first packet ahead + waiting to be transmitted to a different output port. This is + defined in RFC 2889, Section 5.5 ("Congestion Control"). This + section expands on RFC 2889 in the context of data center + benchmarking. + + The objective of this test is to understand the DUT's behavior in the + HOLB scenario and measure the packet loss. + + The differences between this HOLB test and RFC 2889 are as follows: + + - This HOLB test starts with eight ports in two groups of four ports + each, instead of four ports (as compared with Section 5.5 of + RFC 2889). + + - This HOLB test shifts all the port numbers by one in a second + iteration of the test; this is new, as compared to the HOLB test + described in RFC 2889. The shifting port numbers continue until + all ports are the first in the group; the purpose of this is to + make sure that all permutations are tested in order to cover + differences in behavior in the SoC of the DUT. + + - Another test within this HOLB test expands the group of ports, + such that traffic is divided among four ports instead of two + (25% instead of 50% per port). + + - Section 5.3 lists requirements that supplement the requirements + listed in RFC 2889, Section 5.5. + +5.2. Methodology + + In order to cause congestion in the form of HOLB, groups of + four ports are used. A group has two ingress ports and two + egress ports. The first ingress port MUST have two flows configured, + each going to a different egress port. The second ingress port will + congest the second egress port by sending line rate. The goal is to + measure if there is loss on the flow for the first egress port, which + is not oversubscribed. + + A traffic generator MUST be connected to at least eight ports on the + DUT and SHOULD be connected using all the DUT ports. + + + + + +Avramov & Rapp Informational [Page 12] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + + Note that the tests described in procedures 1) and 2) in this section + have iterations called "first iteration", "second iteration", and + "last iteration". The idea is to show the first two iterations so + the reader understands the logic of how to keep incrementing the + iterations. The last iteration shows the end state of the variables. + + 1) Measure two groups with eight DUT ports. + + o First iteration: Measure the packet loss for two groups with + consecutive ports. + + The composition of the first group is as follows: + + Ingress port 1 sending 50% of traffic to egress port 3 + and ingress port 1 sending 50% of traffic to egress port 4. + Ingress port 2 sending line rate to egress port 4. + Measure the amount of traffic loss for the traffic from ingress + port 1 to egress port 3. + + The composition of the second group is as follows: + + Ingress port 5 sending 50% of traffic to egress port 7 + and ingress port 5 sending 50% of traffic to egress port 8. + Ingress port 6 sending line rate to egress port 8. + Measure the amount of traffic loss for the traffic from ingress + port 5 to egress port 7. + + o Second iteration: Repeat the first iteration by shifting all + the ports from N to N+1. + + The composition of the first group is as follows: + + Ingress port 2 sending 50% of traffic to egress port 4 + and ingress port 2 sending 50% of traffic to egress port 5. + Ingress port 3 sending line rate to egress port 5. + Measure the amount of traffic loss for the traffic from ingress + port 2 to egress port 4. + + The composition of the second group is as follows: + + Ingress port 6 sending 50% of traffic to egress port 8 + and ingress port 6 sending 50% of traffic to egress port 9. + Ingress port 7 sending line rate to egress port 9. + Measure the amount of traffic loss for the traffic from ingress + port 6 to egress port 8. + + + + + + +Avramov & Rapp Informational [Page 13] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + + o Last iteration: When the first port of the first group is + connected to the last DUT port and the last port of the second + group is connected to the seventh port of the DUT. + + Measure the amount of traffic loss for the traffic from ingress + port N to egress port 2 and from ingress port 4 to egress port 6. + + 2) Measure with N/4 groups with N DUT ports. + + The traffic from the ingress port is split across four egress + ports (100/4 = 25%). + + o First iteration: Expand to fully utilize all the DUT ports in + increments of four. Repeat the methodology of procedure 1) + with all the groups of ports possible to achieve on the device, + and measure the amount of traffic loss for each port group. + + o Second iteration: Shift by +1 the start of each consecutive + port of the port groups. + + o Last iteration: Shift by N-1 the start of each consecutive port + of the port groups, and measure the amount of traffic loss for + each port group. + +5.3. Reporting Format + + For each test, the report MUST include the following: + + - The port configuration, including the number and location of + ingress and egress ports located on the DUT. + + - If HOLB was observed in accordance with the HOLB test described in + Section 5. + + - Percent of traffic loss. + + - The repeatability of the test needs to be indicated: the number of + iterations of the same test and the percentage of variation + between results (min, max, avg). + + + + + + + + + + + + +Avramov & Rapp Informational [Page 14] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + +6. Incast Stateful and Stateless Traffic + +6.1. Objective + + The objective of this test is to measure the values for TCP Goodput + [TCP-INCAST] and latency with a mix of large and small flows. The + test is designed to simulate a mixed environment of stateful flows + that require high rates of goodput and stateless flows that require + low latency. Stateful flows are created by generating TCP traffic, + and stateless flows are created using UDP traffic. + +6.2. Methodology + + In order to simulate the effects of stateless and stateful traffic on + the DUT, there MUST be multiple ingress ports receiving traffic + destined for the same egress port. There also MAY be a mix of + stateful and stateless traffic arriving on a single ingress port. + The simplest setup would be two ingress ports receiving traffic + destined to the same egress port. + + One ingress port MUST maintain a TCP connection through the ingress + port to a receiver connected to an egress port. Traffic in the TCP + stream MUST be sent at the maximum rate allowed by the traffic + generator. At the same time, the TCP traffic is flowing through the + DUT, and the stateless traffic is sent destined to a receiver on the + same egress port. The stateless traffic MUST be a microburst of + 100% intensity. + + It is RECOMMENDED that the ingress and egress ports be varied in + multiple tests in order to measure the maximum microburst capacity. + + The intensity of a microburst MAY be varied in order to obtain the + microburst capacity at various ingress rates. + + It is RECOMMENDED that all ports on the DUT be used in the test. + + The tests described below have iterations called "first iteration", + "second iteration", and "last iteration". The idea is to show the + first two iterations so the reader understands the logic of how to + keep incrementing the iterations. The last iteration shows the + end state of the variables. + + + + + + + + + + +Avramov & Rapp Informational [Page 15] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + + For example: + + Stateful traffic port variation (TCP traffic): + + TCP traffic needs to be generated for this test. During the + iterations, the number of egress ports MAY vary as well. + + o First iteration: One ingress port receiving stateful TCP traffic + and one ingress port receiving stateless traffic destined to + one egress port. + + o Second iteration: Two ingress ports receiving stateful TCP traffic + and one ingress port receiving stateless traffic destined to + one egress port. + + o Last iteration: N-2 ingress ports receiving stateful TCP traffic + and one ingress port receiving stateless traffic destined to + one egress port. + + Stateless traffic port variation (UDP traffic): + + UDP traffic needs to be generated for this test. During the + iterations, the number of egress ports MAY vary as well. + + o First iteration: One ingress port receiving stateful TCP traffic + and one ingress port receiving stateless traffic destined to + one egress port. + + o Second iteration: One ingress port receiving stateful TCP traffic + and two ingress ports receiving stateless traffic destined to + one egress port. + + o Last iteration: One ingress port receiving stateful TCP traffic + and N-2 ingress ports receiving stateless traffic destined to + one egress port. + + + + + + + + + + + + + + + + +Avramov & Rapp Informational [Page 16] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + +6.3. Reporting Format + + The report MUST include the following: + + - Number of ingress and egress ports, along with designation of + stateful or stateless flow assignment. + + - Stateful flow goodput. + + - Stateless flow latency. + + - The repeatability of the test needs to be indicated: the number of + iterations of the same test and the percentage of variation + between results (min, max, avg). + +7. Security Considerations + + Benchmarking activities as described in this memo are limited to + technology characterization using controlled stimuli in a laboratory + environment, with dedicated address space and the constraints + specified in the sections above. + + The benchmarking network topology will be an independent test setup + and MUST NOT be connected to devices that may forward the test + traffic into a production network or misroute traffic to the test + management network. + + Further, benchmarking is performed on a "black-box" basis, relying + solely on measurements observable external to the DUT. + + Special capabilities SHOULD NOT exist in the DUT specifically for + benchmarking purposes. Any implications for network security arising + from the DUT SHOULD be identical in the lab and in production + networks. + +8. IANA Considerations + + This document does not require any IANA actions. + + + + + + + + + + + + + +Avramov & Rapp Informational [Page 17] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + +9. References + +9.1. Normative References + + [RFC1242] Bradner, S., "Benchmarking Terminology for Network + Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, + July 1991, <https://www.rfc-editor.org/info/rfc1242>. + + [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate + Requirement Levels", BCP 14, RFC 2119, + DOI 10.17487/RFC2119, March 1997, + <https://www.rfc-editor.org/info/rfc2119>. + + [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for + Network Interconnect Devices", RFC 2544, + DOI 10.17487/RFC2544, March 1999, + <https://www.rfc-editor.org/info/rfc2544>. + + [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in + RFC 2119 Key Words", BCP 14, RFC 8174, + DOI 10.17487/RFC8174, May 2017, + <https://www.rfc-editor.org/info/rfc8174>. + + [RFC8238] Avramov, L. and J. Rapp, "Data Center Benchmarking + Terminology", RFC 8238, DOI 10.17487/RFC8238, August 2017, + <https://www.rfc-editor.org/info/rfc8238>. + +9.2. Informative References + + [RFC2432] Dubray, K., "Terminology for IP Multicast Benchmarking", + RFC 2432, DOI 10.17487/RFC2432, October 1998, + <https://www.rfc-editor.org/info/rfc2432>. + + [RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology + for LAN Switching Devices", RFC 2889, + DOI 10.17487/RFC2889, August 2000, + <https://www.rfc-editor.org/info/rfc2889>. + + [RFC3918] Stopp, D. and B. Hickman, "Methodology for IP Multicast + Benchmarking", RFC 3918, DOI 10.17487/RFC3918, + October 2004, <https://www.rfc-editor.org/info/rfc3918>. + + + + + + + + + + +Avramov & Rapp Informational [Page 18] + +RFC 8239 Data Center Benchmarking Methodology August 2017 + + + [RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet + Sizes for Additional Testing", RFC 6985, + DOI 10.17487/RFC6985, July 2013, + <https://www.rfc-editor.org/info/rfc6985>. + + [TCP-INCAST] + Chen, Y., Griffith, R., Zats, D., Joseph, A., and R. Katz, + "Understanding TCP Incast and Its Implications for Big + Data Workloads", April 2012, <http://yanpeichen.com/ + professional/usenixLoginIncastReady.pdf>. + +Acknowledgments + + The authors would like to thank Al Morton and Scott Bradner for their + reviews and feedback. + +Authors' Addresses + + Lucien Avramov + Google + 1600 Amphitheatre Parkway + Mountain View, CA 94043 + United States of America + + Email: lucien.avramov@gmail.com + + + Jacob Rapp + VMware + 3401 Hillview Ave. + Palo Alto, CA 94304 + United States of America + + Email: jhrapp@gmail.com + + + + + + + + + + + + + + + + + +Avramov & Rapp Informational [Page 19] + |