From 4bfd864f10b68b71482b35c818559068ef8d5797 Mon Sep 17 00:00:00 2001 From: Thomas Voss Date: Wed, 27 Nov 2024 20:54:24 +0100 Subject: doc: Add RFC documents --- doc/rfc/rfc6815.txt | 619 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 619 insertions(+) create mode 100644 doc/rfc/rfc6815.txt (limited to 'doc/rfc/rfc6815.txt') diff --git a/doc/rfc/rfc6815.txt b/doc/rfc/rfc6815.txt new file mode 100644 index 0000000..b36623f --- /dev/null +++ b/doc/rfc/rfc6815.txt @@ -0,0 +1,619 @@ + + + + + + +Internet Engineering Task Force (IETF) S. Bradner +Request for Comments: 6815 Harvard University +Updates: 2544 K. Dubray +Category: Informational Juniper Networks +ISSN: 2070-1721 J. McQuaid + Turnip Video + A. Morton + AT&T Labs + November 2012 + + + Applicability Statement for RFC 2544: + Use on Production Networks Considered Harmful + +Abstract + + The Benchmarking Methodology Working Group (BMWG) has been developing + key performance metrics and laboratory test methods since 1990, and + continues this work at present. The methods described in RFC 2544 + are intended to generate traffic that overloads network device + resources in order to assess their capacity. Overload of shared + resources would likely be harmful to user traffic performance on a + production network, and there are further negative consequences + identified with production application of the methods. This memo + clarifies the scope of RFC 2544 and other IETF BMWG benchmarking work + for isolated test environments only, and it encourages new standards + activity for measurement methods applicable outside that scope. + +Status of This Memo + + This document is not an Internet Standards Track specification; it is + published for informational purposes. + + This document is a product of the Internet Engineering Task Force + (IETF). It represents the consensus of the IETF community. It has + received public review and has been approved for publication by the + Internet Engineering Steering Group (IESG). Not all documents + approved by the IESG are a candidate for any level of Internet + Standard; see Section 2 of RFC 5741. + + Information about the current status of this document, any errata, + and how to provide feedback on it may be obtained at + http://www.rfc-editor.org/info/rfc6815. + + + + + + + + +Bradner, et al. Informational [Page 1] + +RFC 6815 RFC 2544 Applicability November 2012 + + +Copyright Notice + + Copyright (c) 2012 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (http://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the Simplified BSD License. + +Table of Contents + + 1. Introduction ....................................................3 + 1.1. Requirements Language ......................................4 + 2. Scope and Goals .................................................4 + 3. The Concept of an Isolated Test Environment .....................4 + 4. Why the Methods of RFC 2544 Are Intended Only for ITE ...........4 + 4.1. Experimental Control and Accuracy ..........................4 + 4.2. Containing Damage ..........................................5 + 5. Advisory on RFC 2544 Methods in Production Networks .............5 + 6. Considering Performance Testing in Production Networks ..........6 + 7. Security Considerations .........................................7 + 8. Acknowledgements ................................................7 + 9. References ......................................................8 + 9.1. Normative References .......................................8 + 9.2. Informative References .....................................8 + Appendix A. Example of RFC 2544 Method Failure in Production + Network Measurement ....................................9 + + + + + + + + + + + + + + + + + + +Bradner, et al. Informational [Page 2] + +RFC 6815 RFC 2544 Applicability November 2012 + + +1. Introduction + + This memo clarifies the scope and use of IETF Benchmarking + Methodology Working Group (BMWG) tests including [RFC2544], which + discusses and defines several tests that may be used to characterize + the performance of a network interconnecting device. All readers of + this memo must read and fully understand [RFC2544]. + + Benchmarking methodologies (beginning with [RFC2544]) have always + relied on test conditions that can only be produced and replicated + reliably in the laboratory. These methodologies are not appropriate + for inclusion in wider specifications such as: + + 1. Validation of telecommunication service configuration, such as + the Committed Information Rate (CIR). + + 2. Validation of performance metrics in a telecommunication Service + Level Agreement (SLA), such as frame loss and latency. + + 3. Telecommunication service activation testing, where traffic that + shares network resources with the test might be adversely + affected. + + Above, we distinguish "telecommunication service" (where a network + service provider contracts with a customer to transfer information + between specified interfaces at different geographic locations) from + the generic term "service". Below, we use the adjective "production" + to refer to networks carrying live user traffic. [RFC2544] used the + term "real-world" to refer to production networks and to + differentiate them from test networks. + + Although [RFC2544] has been held up as the standard reference for the + testing listed above, we believe that the actual methods used vary + from [RFC2544] in significant ways. Since the only citation is to + [RFC2544], the modifications are opaque to the standards community + and to users in general. + + Since applying the test traffic and methods described in [RFC2544] on + a production network risks causing overload in shared resources, + there is direct risk of harming user traffic if the methods are + misused in this way. Therefore, the IETF BMWG developed this + Applicability Statement for [RFC2544] to directly address the + situation. + + + + + + + + +Bradner, et al. Informational [Page 3] + +RFC 6815 RFC 2544 Applicability November 2012 + + +1.1. Requirements Language + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in RFC 2119 [RFC2119]. + +2. Scope and Goals + + This memo clarifies the scope of [RFC2544] with the goal of providing + guidance to the industry on its applicability, which is limited to + laboratory testing. + +3. The Concept of an Isolated Test Environment + + An Isolated Test Environment (ITE) used with the methods of [RFC2544] + (as illustrated in Figures 1 through 3 of [RFC2544]) has the ability + to: + + o contain the test streams to paths within the desired setup + + o prevent non-test traffic from traversing the test setup + + These features allow unfettered experimentation, while at the same + time protecting lab equipment management/control LANs and other + production networks from the unwanted effects of the test traffic. + +4. Why the Methods of RFC 2544 Are Intended Only for ITE + + The following sections discuss some of the reasons why [RFC2544] + methods are applicable only for isolated laboratory use, and the + consequences of applying these methods outside the lab environment. + +4.1. Experimental Control and Accuracy + + All of the tests described in RFC 2544 require that the tester and + device under test are the only devices on the networks that are + transmitting data. The presence of other traffic (unwanted on the + ITE network) would mean that the specified test conditions have not + been achieved and flawed results are a likely consequence. + + If any other traffic appears and the amount varies over time, the + repeatability of any test result will likely depend to some degree on + the amount and variation of the other traffic. + + The presence of other traffic makes accurate, repeatable, and + consistent measurements of the performance of the device under test + very unlikely, since the complete details of test conditions will not + be reported. + + + +Bradner, et al. Informational [Page 4] + +RFC 6815 RFC 2544 Applicability November 2012 + + + For example, the RFC 2544 Throughput Test attempts to characterize a + maximum reliable load; thus, there will be testing above the maximum + that causes packet/frame loss. Any other sources of traffic on the + network will cause packet loss to occur at a tester data rate lower + than the rate that would be achieved without the extra traffic. + +4.2. Containing Damage + + [RFC2544] methods, specifically to determine Throughput as defined in + [RFC1242] and other benchmarks, may overload the resources of the + device under test, and they may cause failure modes in the device + under test. Since failures can become the root cause of more + widespread failure, it is clearly desirable to contain all test + traffic within the ITE. + + In addition, such testing can have a negative effect on any traffic + that shares resources with the test stream(s) since, in most cases, + the traffic load will be close to the capacity of the network links. + + Appendix C.2.2 of [RFC2544] (as adjusted by errata) gives the private + IPv4 address range for testing: + + "...The network addresses 198.18.0.0 through 198.19.255.255 have been + assigned to the BMWG by the IANA for this purpose. This assignment + was made to minimize the chance of conflict in case a testing device + were to be accidentally connected to part of the Internet. The + specific use of the addresses is detailed below." + + In other words, devices operating on the Internet may be configured + to discard any traffic they observe in this address range, as it is + intended for laboratory ITE use only. Thus, if testers using the + assigned testing address ranges are connected to the Internet and + test packets are forwarded across the Internet, it is likely that the + packets will be discarded and the test will not work. + + We note that a range of IPv6 addresses has been assigned to BMWG for + laboratory test purposes, in [RFC5180] (as amended by errata). + + See the Security Considerations section below for further + considerations on containing damage. + +5. Advisory on RFC 2544 Methods in Production Networks + + The tests in [RFC2544] were designed to measure the performance of + network devices, not of networks, and certainly not production + networks carrying user traffic on shared resources. There will be + undesirable consequences when applying these methods outside the + isolated test environment. + + + +Bradner, et al. Informational [Page 5] + +RFC 6815 RFC 2544 Applicability November 2012 + + + One negative consequence stems from reliance on frame loss as an + indicator of resource exhaustion in [RFC2544] methods. In practice, + link-layer and physical-layer errors prevent production networks from + operating loss-free. The [RFC2544] methods will not correctly assess + Throughput when loss from uncontrolled sources is present. Frame + loss occurring at the SLA levels of some networks could affect every + iteration of Throughput testing (when each step includes sufficient + packets to experience facility-related loss). Flawed results waste + the time and resources of the testing service user and of the service + provider when called to dispute the measurement. These are + additional examples of harm that compliance with this advisory should + help to avoid. See Appendix A for an example. + + The methods described in [RFC2544] are intended to generate traffic + that overloads network device resources in order to assess their + capacity. Overload of shared resources would likely be harmful to + user traffic performance on a production network. These tests MUST + NOT be used on production networks and as discussed above. The tests + will not produce a reliable or accurate benchmarking result on a + production network. + + [RFC2544] methods have never been validated on a network path, even + when that path is not part of a production network and carrying no + other traffic. It is unknown whether the tests can be used to + measure valid and reliable performance of a multi-device, multi- + network path. It is possible that some of the tests may prove valid + in some path scenarios, but that work has not been done or has not + been shared with the IETF community. Thus, such testing is + contraindicated by the BMWG. + +6. Considering Performance Testing in Production Networks + + The IETF has addressed the problem of production network performance + measurement by chartering a different working group: IP Performance + Metrics (IPPM). This working group has developed a set of standard + metrics to assess the quality, performance, and reliability of + Internet packet transfer services. These metrics can be measured by + network operators, end users, or independent testing groups. We note + that some IPPM metrics differ from RFC 2544 metrics with similar + names, and there is likely to be confusion if the details are + ignored. + + IPPM has not yet standardized methods for raw capacity measurement of + Internet paths. Such testing needs to adequately consider the strong + possibility for degradation to any other traffic that may be present + due to congestion. There are no specific methods proposed for + activation of a packet transfer service in IPPM at this time. Thus, + individuals who need to conduct capacity tests on production networks + + + +Bradner, et al. Informational [Page 6] + +RFC 6815 RFC 2544 Applicability November 2012 + + + should actively participate in standards development to ensure their + methods receive appropriate industry review and agreement, in the + IETF or in alternate standards development organizations. + + Other standards may help to fill gaps in telecommunication service + testing. For example, the IETF has many standards intended to assist + with network Operations, Administration, and Maintenance (OAM). + ITU-T Study Group 12 has a Recommendation on service activation test + methodology [Y.1564]. + + The world will not spin off axis while waiting for appropriate and + standardized methods to emerge from the consensus process. + +7. Security Considerations + + This Applicability Statement intends to help preserve the security of + the Internet by clarifying that the scope of [RFC2544] and other BMWG + memos are all limited to testing in a laboratory ITE, thus avoiding + accidental Denial-of-Service attacks or congestion due to high + traffic volume test streams. + + All benchmarking activities are limited to technology + characterization using controlled stimuli in a laboratory + environment, with dedicated address space and the other constraints + [RFC2544]. + + The benchmarking network topology will be an independent test setup + and MUST NOT be connected to devices that may forward the test + traffic into a production network or misroute traffic to the test + management network. + + Further, benchmarking is performed on a "black-box" basis, relying + solely on measurements observable external to the device under test/ + system under test (DUT/SUT). + + Special capabilities SHOULD NOT exist in the DUT/SUT specifically for + benchmarking purposes. Any implications for network security arising + from the DUT/SUT SHOULD be identical in the lab and in production + networks. + +8. Acknowledgements + + Thanks to Matt Zekauskas, Bill Cerveny, Barry Constantine, Curtis + Villamizar, David Newman, and Adrian Farrel for suggesting + improvements to this memo. + + + + + + +Bradner, et al. Informational [Page 7] + +RFC 6815 RFC 2544 Applicability November 2012 + + + Specifically, Al Morton would like to thank his coauthors, who + constitute the complete set of Chairmen-Emeritus of the BMWG, for + returning from other pursuits to develop this statement and see it + through to approval. This has been a rare privilege; one that likely + will not be matched in the IETF again: + + Scott Bradner served as Chairman from 1990 to 1993 + Jim McQuaid served as Chairman from 1993 to 1995 + Kevin Dubray served as Chairman from 1995 to 2006 + + It's all about the band. + +9. References + +9.1. Normative References + + [RFC1242] Bradner, S., "Benchmarking terminology for network + interconnection devices", RFC 1242, July 1991. + + [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate + Requirement Levels", BCP 14, RFC 2119, March 1997. + + [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for + Network Interconnect Devices", RFC 2544, March 1999. + + [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. + Dugatkin, "IPv6 Benchmarking Methodology for Network + Interconnect Devices", RFC 5180, May 2008. + +9.2. Informative References + + [Bryant] Bonica, R. and S. Bryant, "RFC2544 Testing in Production + Network", Work in Progress, October 2012. + + [Y.1564] ITU-T Recommendation Y.1564, "Ethernet Service Activation + Test Methodology", March 2011. + + + + + + + + + + + + + + + +Bradner, et al. Informational [Page 8] + +RFC 6815 RFC 2544 Applicability November 2012 + + +Appendix A. Example of RFC 2544 Method Failure in Production Network + Measurement + + This Appendix provides an example illustrating how [RFC2544] methods + applied on production networks can easily produce a form of harm from + flawed and misleading results. + + The [RFC2544] Throughput benchmarking method usually includes the + following steps: + + a. Set the offered traffic level, less than max of the ingress + link(s). + + b. Send the test traffic through the device under test (DUT) and + count all frames successfully transferred. + + c. If all frames are received, increment traffic level and repeat + step b. + + d. If one or more frames are lost, the level is in the DUT-overload + region (step b may be repeated at a reduced traffic level to more + exactly determine the maximum rate at which none of the frames + are dropped by the DUT, defined as the Throughput [RFC1242]). + + e. Report the Throughput values, the x-y of graph of frame size and + Throughput, and other information in accordance with [RFC2544]. + + In this method, frame loss is the sole indicator of overload and + therefore the determining factor in the measurement of Throughput + using the [RFC2544] methodology (even though the results may not + report frame loss per se). + + Frame loss is subject to many factors in addition to operating above + the Throughput traffic level. These factors include optical + interference (which may be due to dirty interfaces, crossover from + other signals, fiber bend and temperature, etc.) and electrical + interference (caused by local sources of radio signals, electrical + spikes, solar particles, etc.). In the laboratory environment many + of these issues can be carefully controlled through cleaning and + isolation. Since [RFC2544] methodologies are primarily intended to + test devices and not paths, the total length of path, the number of + interfaces, and compound risk of random frame loss can be kept to a + minimum. + + In a production network, however, there will be many interfaces and + many kilometers of path under test. This considerably increases the + risk of random frame loss. + + + + +Bradner, et al. Informational [Page 9] + +RFC 6815 RFC 2544 Applicability November 2012 + + + The risk of frame loss caused by outside effects is significantly + higher in production networks, and significantly higher with long + paths (both those with long physical path lengths, and those with + large numbers of interfaces in the path). Thus, the risk of falsely + low reported Throughput using an [RFC2544] methodology test is + considerably increased in a production network. + + Therefore, to successfully conduct tests with similar objectives to + those in [RFC2544] in a production network, it will be necessary to + develop modifications to the methodologies defined in [RFC2544] and + standards to describe them. See [Bryant] for an in-progress effort + and [Y.1564] for an approved method adapted to production service + activation. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Bradner, et al. Informational [Page 10] + +RFC 6815 RFC 2544 Applicability November 2012 + + +Authors' Addresses + + Scott Bradner + Harvard University + 1350 Mass. Ave., Room 760 + Cambridge, MA 02138 + USA + + Phone: +1 617 495 3864 + EMail: sob@harvard.edu + URI: http://www.sobco.com + + + Kevin Dubray + Juniper Networks + + + Jim McQuaid + Turnip Video + 6 Cobbleridge Court + Durham, North Carolina 27713 + USA + + Phone: +1 919-619-3220 + EMail: jim@turnipvideo.com + URI: www.turnipvideo.com + + + Al Morton + AT&T Labs + 200 Laurel Avenue South + Middletown,, NJ 07748 + USA + + Phone: +1 732 420 1571 + Fax: +1 732 368 1192 + EMail: acmorton@att.com + URI: http://home.comcast.net/~acmacm/ + + + + + + + + + + + + + +Bradner, et al. Informational [Page 11] + -- cgit v1.2.3