summaryrefslogtreecommitdiff
path: root/doc/rfc/rfc6390.txt
diff options
context:
space:
mode:
Diffstat (limited to 'doc/rfc/rfc6390.txt')
-rw-r--r--doc/rfc/rfc6390.txt1291
1 files changed, 1291 insertions, 0 deletions
diff --git a/doc/rfc/rfc6390.txt b/doc/rfc/rfc6390.txt
new file mode 100644
index 0000000..70e3284
--- /dev/null
+++ b/doc/rfc/rfc6390.txt
@@ -0,0 +1,1291 @@
+
+
+
+
+
+
+Internet Engineering Task Force (IETF) A. Clark
+Request for Comments: 6390 Telchemy Incorporated
+BCP: 170 B. Claise
+Category: Best Current Practice Cisco Systems, Inc.
+ISSN: 2070-1721 October 2011
+
+
+ Guidelines for Considering New Performance Metric Development
+
+Abstract
+
+ This document describes a framework and a process for developing
+ Performance Metrics of protocols and applications transported over
+ IETF-specified protocols. These metrics can be used to characterize
+ traffic on live networks and services.
+
+Status of This Memo
+
+ This memo documents an Internet Best Current Practice.
+
+ This document is a product of the Internet Engineering Task Force
+ (IETF). It represents the consensus of the IETF community. It has
+ received public review and has been approved for publication by the
+ Internet Engineering Steering Group (IESG). Further information on
+ BCPs is available in Section 2 of RFC 5741.
+
+ Information about the current status of this document, any errata,
+ and how to provide feedback on it may be obtained at
+ http://www.rfc-editor.org/info/rfc6390.
+
+Copyright Notice
+
+ Copyright (c) 2011 IETF Trust and the persons identified as the
+ document authors. All rights reserved.
+
+ This document is subject to BCP 78 and the IETF Trust's Legal
+ Provisions Relating to IETF Documents
+ (http://trustee.ietf.org/license-info) in effect on the date of
+ publication of this document. Please review these documents
+ carefully, as they describe your rights and restrictions with respect
+ to this document. Code Components extracted from this document must
+ include Simplified BSD License text as described in Section 4.e of
+ the Trust Legal Provisions and are provided without warranty as
+ described in the Simplified BSD License.
+
+
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 1]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+ This document may contain material from IETF Documents or IETF
+ Contributions published or made publicly available before November
+ 10, 2008. The person(s) controlling the copyright in some of this
+ material may not have granted the IETF Trust the right to allow
+ modifications of such material outside the IETF Standards Process.
+ Without obtaining an adequate license from the person(s) controlling
+ the copyright in such materials, this document may not be modified
+ outside the IETF Standards Process, and derivative works of it may
+ not be created outside the IETF Standards Process, except to format
+ it for publication as an RFC or to translate it into languages other
+ than English.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 2]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+Table of Contents
+
+ 1. Introduction ....................................................4
+ 1.1. Background and Motivation ..................................4
+ 1.2. Organization of This Document ..............................5
+ 2. Terminology .....................................................5
+ 2.1. Requirements Language ......................................5
+ 2.2. Performance Metrics Directorate ............................5
+ 2.3. Quality of Service .........................................5
+ 2.4. Quality of Experience ......................................6
+ 2.5. Performance Metric .........................................6
+ 3. Purpose and Scope ...............................................6
+ 4. Relationship between QoS, QoE, and Application-Specific
+ Performance Metrics .............................................7
+ 5. Performance Metrics Development .................................7
+ 5.1. Identifying and Categorizing the Audience ..................7
+ 5.2. Definitions of a Performance Metric ........................8
+ 5.3. Computed Performance Metrics ...............................9
+ 5.3.1. Composed Performance Metrics ........................9
+ 5.3.2. Index ..............................................10
+ 5.4. Performance Metric Specification ..........................10
+ 5.4.1. Outline ............................................10
+ 5.4.2. Normative Parts of Performance Metric Definition ...11
+ 5.4.3. Informative Parts of Performance Metric
+ Definition .........................................13
+ 5.4.4. Performance Metric Definition Template .............14
+ 5.4.5. Example: Loss Rate .................................15
+ 5.5. Dependencies ..............................................16
+ 5.5.1. Timing Accuracy ....................................16
+ 5.5.2. Dependencies of Performance Metric Definitions on
+ Related Events or Metrics ..........................16
+ 5.5.3. Relationship between Performance Metric and
+ Lower-Layer Performance Metrics ....................17
+ 5.5.4. Middlebox Presence .................................17
+ 5.6. Organization of Results ...................................17
+ 5.7. Parameters: The Variables of a Performance Metric .........18
+ 6. Performance Metric Development Process .........................18
+ 6.1. New Proposals for Performance Metrics .....................18
+ 6.2. Reviewing Metrics .........................................19
+ 6.3. Performance Metrics Directorate Interaction with
+ Other WGs .................................................19
+ 6.4. Standards Track Performance Metrics .......................20
+ 7. Security Considerations ........................................20
+ 8. Acknowledgements ...............................................20
+ 9. References .....................................................21
+ 9.1. Normative References ......................................21
+ 9.2. Informative References ....................................21
+
+
+
+
+Clark & Claise Best Current Practice [Page 3]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+1. Introduction
+
+ Many networking technologies, applications, or services are
+ distributed in nature, and their performance may be impacted by IP
+ impairments, server capacity, congestion, and other factors. It is
+ important to measure the performance of applications and services to
+ ensure that quality objectives are being met and to support problem
+ diagnosis. Standardized metrics help ensure that performance
+ measurement is implemented consistently, and they facilitate
+ interpretation and comparison.
+
+ There are at least three phases in the development of performance
+ standards. They are as follows:
+
+ 1. Definition of a Performance Metric and its units of measure
+
+ 2. Specification of a method of measurement
+
+ 3. Specification of the reporting format
+
+ During the development of metrics, it is often useful to define
+ performance objectives and expected value ranges. This additional
+ information is typically not part of the formal specification of the
+ metric but does provide useful background for implementers and users
+ of the metric.
+
+ The intended audience for this document includes, but is not limited
+ to, IETF participants who write Performance Metrics documents in the
+ IETF, reviewers of such documents, and members of the Performance
+ Metrics Directorate.
+
+1.1. Background and Motivation
+
+ Previous IETF work related to the reporting of application
+ Performance Metrics includes "Real-time Application Quality-of-
+ Service Monitoring (RAQMON) Framework" [RFC4710]. This framework
+ extends the remote network monitoring (RMON) family of specifications
+ to allow real-time quality-of-service (QoS) monitoring of various
+ applications that run on devices such as IP phones, pagers, Instant
+ Messaging clients, mobile phones, and various other handheld
+ computing devices. Furthermore, "RTP Control Protocol Extended
+ Reports (RTCP XR)" [RFC3611] and "Session Initiation Protocol Event
+ Package for Voice Quality Reporting" [RFC6035] define protocols that
+ support real-time Quality of Experience (QoE) reporting for Voice
+ over IP (VoIP) and other applications running on devices such as IP
+ phones and mobile handsets.
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 4]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+ The IETF is also actively involved in the development of reliable
+ transport protocols, such as TCP [RFC0793] or the Stream Control
+ Transmission Protocol (SCTP) [RFC4960], which would affect the
+ relationship between IP performance and application performance.
+
+ Thus, there is a gap in the currently chartered coverage of IETF
+ Working Groups (WGs): development of Performance Metrics for
+ protocols above and below the IP layer that can be used to
+ characterize performance on live networks.
+
+ Similar to "Guidelines for Considering Operations and Management of
+ New Protocols and Protocol Extensions" [RFC5706], which is the
+ reference document for the IETF Operations Directorate, this document
+ should be consulted as part of the new Performance Metric review by
+ the members of the Performance Metrics Directorate.
+
+1.2. Organization of This Document
+
+ This document is divided into two major sections beyond the "Purpose
+ and Scope" section. The first is a definition and description of a
+ Performance Metric and its key aspects. The second defines a process
+ to develop these metrics that is applicable to the IETF environment.
+
+2. Terminology
+
+2.1. Requirements Language
+
+ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+ "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
+ document are to be interpreted as described in RFC 2119 [RFC2119].
+
+2.2. Performance Metrics Directorate
+
+ The Performance Metrics Directorate is a directorate that provides
+ guidance for Performance Metrics development in the IETF.
+
+ The Performance Metrics Directorate should be composed of experts in
+ the performance community, potentially selected from the IP
+ Performance Metrics (IPPM), Benchmarking Methodology (BMWG), and
+ Performance Metrics for Other Layers (PMOL) WGs.
+
+2.3. Quality of Service
+
+ Quality of Service (QoS) is defined in a way similar to the ITU
+ "Quality of Service (QoS)" section of [E.800], i.e., "Totality of
+ characteristics of a telecommunications service that bear on its
+ ability to satisfy stated and implied needs of the user of the
+ service".
+
+
+
+Clark & Claise Best Current Practice [Page 5]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+2.4. Quality of Experience
+
+ Quality of Experience (QoE) is defined in a way similar to the ITU
+ "QoS experienced/perceived by customer/user (QoSE)" section of
+ [E.800], i.e., "a statement expressing the level of quality that
+ customers/users believe they have experienced".
+
+ NOTE 1 - The level of QoS experienced and/or perceived by the
+ customer/user may be expressed by an opinion rating.
+
+ NOTE 2 - QoE has two main components: quantitative and
+ qualitative. The quantitative component can be influenced by the
+ complete end-to-end system effects (including user devices and
+ network infrastructure).
+
+ NOTE 3 - The qualitative component can be influenced by user
+ expectations, ambient conditions, psychological factors,
+ application context, etc.
+
+ NOTE 4 - QoE may also be considered as QoS delivered, received,
+ and interpreted by a user with the pertinent qualitative factors
+ influencing his/her perception of the service.
+
+2.5. Performance Metric
+
+ A Performance Metric is a quantitative measure of performance,
+ specific to an IETF-specified protocol or specific to an application
+ transported over an IETF-specified protocol. Examples of Performance
+ Metrics are the FTP response time for a complete file download, the
+ DNS response time to resolve the IP address, a database logging time,
+ etc.
+
+3. Purpose and Scope
+
+ The purpose of this document is to define a framework and a process
+ for developing Performance Metrics for protocols above and below the
+ IP layer (such as IP-based applications that operate over reliable or
+ datagram transport protocols). These metrics can be used to
+ characterize traffic on live networks and services. As such, this
+ document does not define any Performance Metrics.
+
+ The scope of this document covers guidelines for the Performance
+ Metrics Directorate members for considering new Performance Metrics
+ and suggests how the Performance Metrics Directorate will interact
+ with the rest of the IETF. However, this document is not intended to
+ supersede existing working methods within WGs that have existing
+ chartered work in this area.
+
+
+
+
+Clark & Claise Best Current Practice [Page 6]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+ This process is not intended to govern Performance Metric development
+ in existing IETF WGs that are focused on metrics development, such as
+ the IPPM and BMWG WGs. However, this guidelines document may be
+ useful in these activities and MAY be applied where appropriate. A
+ typical example is the development of Performance Metrics to be
+ exported with the IP Flow Information eXport (IPFIX) protocol
+ [RFC5101], with specific IPFIX information elements [RFC5102], which
+ would benefit from the framework in this document.
+
+ The framework in this document applies to Performance Metrics derived
+ from both active and passive measurements.
+
+4. Relationship between QoS, QoE, and Application-Specific Performance
+ Metrics
+
+ Network QoS deals with network and network protocol performance,
+ while QoE deals with the assessment of a user's experience in the
+ context of a task or a service. The topic of application-specific
+ Performance Metrics includes the measurement of performance at layers
+ between IP and the user. For example, network QoS metrics (packet
+ loss, delay, and delay variation [RFC5481]) can be used to estimate
+ application-specific Performance Metrics (de-jitter buffer size and
+ RTP-layer packet loss), and then combined with other known aspects of
+ a VoIP application (such as codec type) using an algorithm compliant
+ with ITU-T P.564 [P.564] to estimate a Mean Opinion Score (MOS)
+ [P.800]. However, the QoE for a particular VoIP user depends on the
+ specific context, such as a casual conversation, a business
+ conference call, or an emergency call. Finally, QoS and application-
+ specific Performance Metrics are quantitative, while QoE is
+ qualitative. Also, network QoS and application-specific Performance
+ Metrics can be directly or indirectly evident to the user, while the
+ QoE is directly evident.
+
+5. Performance Metrics Development
+
+ This section provides key definitions and qualifications of
+ Performance Metrics.
+
+5.1. Identifying and Categorizing the Audience
+
+ Many of the aspects of metric definition and reporting, even the
+ selection or determination of the essential metrics, depend on who
+ will use the results, and for what purpose. For example, the metric
+ description SHOULD include use cases and example reports that
+ illustrate service quality monitoring and maintenance or
+ identification and quantification of problems.
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 7]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+ All documents defining Performance Metrics SHOULD identify the
+ primary audience and its associated requirements. The audience can
+ influence both the definition of metrics and the methods of
+ measurement.
+
+ The key areas of variation between different metric users include:
+
+ o Suitability of passive measurements of live traffic or active
+ measurements using dedicated traffic
+
+ o Measurement in laboratory environment or on a network of deployed
+ devices
+
+ o Accuracy of the results
+
+ o Access to measurement points and configuration information
+
+ o Measurement topology (point-to-point, point-to-multipoint)
+
+ o Scale of the measurement system
+
+ o Measurements conducted on-demand or continuously
+
+ o Required reporting formats and periods
+
+ o Sampling criteria [RFC5474], such as systematic or probabilistic
+
+ o Period (and duration) of measurement, as the live traffic can have
+ patterns
+
+5.2. Definitions of a Performance Metric
+
+ A Performance Metric is a measure of an observable behavior of a
+ networking technology, an application, or a service. Most of the
+ time, the Performance Metric can be directly measured; however,
+ sometimes, the Performance Metric value is computed. The process for
+ determining the value of a metric may assume an implicit or explicit
+ underlying statistical process; in this case, the Performance Metric
+ is an estimate of a parameter of this process, assuming that the
+ statistical process closely models the behavior of the system.
+
+ A Performance Metric should serve some defined purposes. This may
+ include the measurement of capacity, quantifying how bad some
+ problems are, measurement of service level, problem diagnosis or
+ location, and other such uses. A Performance Metric may also be an
+ input to some other processes, for example, the computation of a
+ composite Performance Metric or a model or simulation of a system.
+ Tests of the "usefulness" of a Performance Metric include:
+
+
+
+Clark & Claise Best Current Practice [Page 8]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+ (i) the degree to which its absence would cause significant loss
+ of information on the behavior or performance of the application
+ or system being measured
+
+ (ii) the correlation between the Performance Metric, the QoS, and
+ the QoE delivered to the user (person or other application)
+
+ (iii) the degree to which the Performance Metric is able to
+ support the identification and location of problems affecting
+ service quality
+
+ (iv) the requirement to develop policies (Service Level Agreement,
+ and potentially Service Level Contract) based on the Performance
+ Metric
+
+ For example, consider a distributed application operating over a
+ network connection that is subject to packet loss. A Packet Loss
+ Rate (PLR) Performance Metric is defined as the mean packet loss
+ ratio over some time period. If the application performs poorly over
+ network connections with a high packet loss ratio and always performs
+ well when the packet loss ratio is zero, then the PLR Performance
+ Metric is useful to some degree. Some applications are sensitive to
+ short periods of high loss (bursty loss) and are relatively
+ insensitive to isolated packet loss events; for this type of
+ application, there would be very weak correlation between PLR and
+ application performance. A "better" Performance Metric would
+ consider both the packet loss ratio and the distribution of loss
+ events. If application performance is degraded when the PLR exceeds
+ some rate, then a useful Performance Metric may be a measure of the
+ duration and frequency of periods during which the PLR exceeds that
+ rate (as, for example, in RFC 3611).
+
+5.3. Computed Performance Metrics
+
+5.3.1. Composed Performance Metrics
+
+ Some Performance Metrics may not be measured directly, but can be
+ composed from base metrics that have been measured. A composed
+ Performance Metric is derived from other metrics by applying a
+ deterministic process or function (e.g., a composition function).
+ The process may use metrics that are identical to the metric being
+ composed, or metrics that are dissimilar, or some combination of both
+ types. Usually, the base metrics have a limited scope in time or
+ space, and they can be combined to estimate the performance of some
+ larger entities.
+
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 9]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+ Some examples of composed Performance Metrics and composed
+ Performance Metric definitions are as follows:
+
+ Spatial composition is defined as the composition of metrics of
+ the same type with differing spatial domains [RFC5835] [RFC6049].
+ Ideally, for spatially composed metrics to be meaningful, the
+ spatial domains should be non-overlapping and contiguous, and the
+ composition operation should be mathematically appropriate for the
+ type of metric.
+
+ Temporal composition is defined as the composition of sets of
+ metrics of the same type with differing time spans [RFC5835]. For
+ temporally composed metrics to be meaningful, the time spans
+ should be non-overlapping and contiguous, and the composition
+ operation should be mathematically appropriate for the type of
+ metric.
+
+ Temporal aggregation is a summarization of metrics into a smaller
+ number of metrics that relate to the total time span covered by
+ the original metrics. An example would be to compute the minimum,
+ maximum, and average values of a series of time-sampled values of
+ a metric.
+
+ In the context of flow records in IP Flow Information eXport (IPFIX),
+ the IPFIX Mediation Framework [RFC6183], based on "IP Flow
+ Information Export (IPFIX) Mediation: Problem Statement" [RFC5982],
+ also discusses some aspects of the temporal and spatial composition.
+
+5.3.2. Index
+
+ An index is a metric for which the output value range has been
+ selected for convenience or clarity, and the behavior of which is
+ selected to support ease of understanding, for example, the R Factor
+ [G.107]. The deterministic function for an index is often developed
+ after the index range and behavior have been determined.
+
+5.4. Performance Metric Specification
+
+5.4.1. Outline
+
+ A Performance Metric definition MUST have a normative part that
+ defines what the metric is and how it is measured or computed, and it
+ SHOULD have an informative part that describes the Performance Metric
+ and its application.
+
+
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 10]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+5.4.2. Normative Parts of Performance Metric Definition
+
+ The normative part of a Performance Metric definition MUST define at
+ least the following:
+
+ (i) Metric Name
+
+ Performance Metric names are RECOMMENDED to be unique within the
+ set of metrics being defined for the protocol layer and context.
+ While strict uniqueness may not be attainable (see the IPPM
+ registry [RFC6248] for an example of an IANA metric registry
+ failing to provide sufficient specificity), broad review must be
+ sought to avoid naming overlap. Note that the Performance Metrics
+ Directorate can help with suggestions for IANA metric registration
+ for unique naming. The Performance Metric name MAY be
+ descriptive.
+
+ (ii) Metric Description
+
+ The Performance Metric description MUST explain what the metric
+ is, what is being measured, and how this relates to the
+ performance of the system being measured.
+
+ (iii) Method of Measurement or Calculation
+
+ The method of measurement or calculation MUST define what is being
+ measured or computed and the specific algorithm to be used. Does
+ the measurement involve active or only passive measurements?
+ Terms such as "average" should be qualified (e.g., running average
+ or average over some interval). Exception cases SHOULD also be
+ defined with the appropriate handling method. For example, there
+ are a number of commonly used metrics related to packet loss;
+ these often don't define the criteria by which a packet is
+ determined to be lost (versus very delayed) or how duplicate
+ packets are handled. For example, if the average PLR during a
+ time interval is reported, and a packet's arrival is delayed from
+ one interval to the next, then was it "lost" during the interval
+ during which it should have arrived or should it be counted as
+ received?
+
+ Some methods of calculation might require discarding some data
+ collected (due to outliers) so as to make the measurement
+ parameters meaningful. One example is burstable billing that
+ sorts the 5-min samples and discards the top 5 percentile.
+
+
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 11]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+ Some parameters linked to the method MAY also be reported, in
+ order to fully interpret the Performance Metric, for example, the
+ time interval, the load, the minimum packet loss, the potential
+ measurement errors and their sources, the attainable accuracy of
+ the metric (e.g., +/- 0.1), the method of calculation, etc.
+
+ (iv) Units of Measurement
+
+ The units of measurement MUST be clearly stated.
+
+ (v) Measurement Point(s) with Potential Measurement Domain
+
+ If the measurement is specific to a measurement point, this SHOULD
+ be defined. The measurement domain MAY also be defined.
+ Specifically, if measurement points are spread across domains, the
+ measurement domain (intra-, inter-) is another factor to consider.
+
+ The Performance Metric definition should discuss how the
+ Performance Metric value might vary, depending on which
+ measurement point is chosen. For example, the time between a SIP
+ request [RFC3261] and the final response can be significantly
+ different at the User Agent Client (UAC) or User Agent Server
+ (UAS).
+
+ In some cases, the measurement requires multiple measurement
+ points: all measurement points SHOULD be defined, including the
+ measurement domain(s).
+
+ (vi) Measurement Timing
+
+ The acceptable range of timing intervals or sampling intervals for
+ a measurement, and the timing accuracy required for such
+ intervals, MUST be specified. Short sampling intervals or
+ frequent samples provide a rich source of information that can
+ help assess application performance but may lead to excessive
+ measurement data. Long measurement or sampling intervals reduce
+ the amount of reported and collected data such that it may be
+ insufficient to understand application performance or service
+ quality, insofar as the measured quantity may vary significantly
+ with time.
+
+ In the case of multiple measurement points, the potential
+ requirement for synchronized clocks must be clearly specified. In
+ the specific example of the IP delay variation application metric,
+ the different aspects of synchronized clocks are discussed in
+ [RFC5481].
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 12]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+5.4.3. Informative Parts of Performance Metric Definition
+
+ The informative part of a Performance Metric specification is
+ intended to support the implementation and use of the metric. This
+ part SHOULD provide the following data:
+
+ (i) Implementation
+
+ The implementation description MAY be in the form of text, an
+ algorithm, or example software. The objective of this part of the
+ metric definition is to help implementers achieve consistent
+ results.
+
+ (ii) Verification
+
+ The Performance Metric definition SHOULD provide guidance on
+ verification testing. This may be in the form of test vectors, a
+ formal verification test method, or informal advice.
+
+ (iii) Use and Applications
+
+ The use and applications description is intended to help the
+ "user" understand how, when, and where the metric can be applied,
+ and what significance the value range for the metric may have.
+ This MAY include a definition of the "typical" and "abnormal"
+ range of the Performance Metric, if this was not apparent from the
+ nature of the metric. The description MAY include information
+ about the influence of extreme measurement values, i.e., if the
+ Performance Metric is sensitive to outliers. The Use and
+ Application section SHOULD also include the security implications
+ in the description.
+
+ For example:
+
+ (a) it is fairly intuitive that a lower packet loss ratio would
+ equate to better performance. However, the user may not know
+ the significance of some given packet loss ratio.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 13]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+ (b) the speech level of a telephone signal is commonly expressed
+ in dBm0. If the user is presented with:
+
+ Speech level = -7 dBm0
+
+ this is not intuitively understandable, unless the user
+ is a telephony expert. If the metric definition explains
+ that the typical range is -18 to -28 dBm0, a value higher
+ than -18 means the signal may be too high (loud), and
+ less than -28 means that the signal may be too low
+ (quiet), it is much easier to interpret the metric.
+
+ (iv) Reporting Model
+
+ The reporting model definition is intended to make any
+ relationship between the metric and the reporting model clear.
+ There are often implied relationships between the method of
+ reporting metrics and the metric itself; however, these are often
+ not made apparent to the implementor. For example, if the metric
+ is a short-term running average packet delay variation (e.g., the
+ inter-arrival jitter in [RFC3550]) and this value is reported at
+ intervals of 6-10 seconds, the resulting measurement may have
+ limited accuracy when packet delay variation is non-stationary.
+
+5.4.4. Performance Metric Definition Template
+
+ Normative
+
+ o Metric Name
+
+ o Metric Description
+
+ o Method of Measurement or Calculation
+
+ o Units of Measurement
+
+ o Measurement Point(s) with Potential Measurement Domain
+
+ o Measurement Timing
+
+
+
+
+
+
+
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 14]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+ Informative
+
+ o Implementation
+
+ o Verification
+
+ o Use and Applications
+
+ o Reporting Model
+
+5.4.5. Example: Loss Rate
+
+ The example used is the loss rate metric as specified in RFC 3611
+ [RFC3611].
+
+ Metric Name: LossRate
+
+ Metric Description: The fraction of RTP data packets from the source
+ lost since the beginning of reception.
+
+ Method of Measurement or Calculation: This value is calculated by
+ dividing the total number of packets lost (after the effects of
+ applying any error protection, such as Forward Error Correction
+ (FEC)) by the total number of packets expected, multiplying the
+ result of the division by 256, limiting the maximum value to 255
+ (to avoid overflow), and taking the integer part.
+
+ Units of Measurement: This metric is expressed as a fixed-point
+ number with the binary point at the left edge of the field. For
+ example, a metric value of 12 means a loss rate of
+ approximately 5%.
+
+ Measurement Point(s) with Potential Measurement Domain: This metric
+ is made at the receiving end of the RTP stream sent during a Voice
+ over IP call.
+
+ Measurement Timing: This metric can be used over a wide range of
+ time intervals. Using time intervals of longer than one hour may
+ prevent the detection of variations in the value of this metric
+ due to time-of-day changes in network load. Timing intervals
+ should not vary in duration by more than +/- 2%.
+
+ Implementation: The numbers of duplicated packets and discarded
+ packets do not enter into this calculation. Since receivers
+ cannot be required to maintain unlimited buffers, a receiver MAY
+ categorize late-arriving packets as lost. The degree of lateness
+ that triggers a loss SHOULD be significantly greater than that
+ which triggers a discard.
+
+
+
+Clark & Claise Best Current Practice [Page 15]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+ Verification: The metric value ranges between 0 and 255.
+
+ Use and Applications: This metric is useful for monitoring VoIP
+ calls, more precisely, to detect the VoIP loss rate in the
+ network. This loss rate, along with the rate of packets discarded
+ due to jitter, has some effect on the quality of the voice stream.
+
+ Reporting Model: This metric needs to be associated with a defined
+ time interval, which could be defined by fixed intervals or by a
+ sliding window. In the context of RFC 3611, the metric is
+ measured continuously from the start of the RTP stream, and the
+ value of the metric is sampled and reported in RTCP XR VoIP
+ Metrics reports.
+
+5.5. Dependencies
+
+ This section introduces several Performance Metrics dependencies,
+ which the Performance Metric designer should keep in mind during
+ Performance Metric development. These dependencies, and any others
+ not listed here, SHOULD be documented in the Performance Metric
+ specifications.
+
+5.5.1. Timing Accuracy
+
+ The accuracy of the timing of a measurement may affect the accuracy
+ of the Performance Metric. This may not materially affect a sampled-
+ value metric; however, it would affect an interval-based metric.
+ Some metrics -- for example, the number of events per time interval
+ -- would be directly affected; for example, a 10% variation in time
+ interval would lead directly to a 10% variation in the measured
+ value. Other metrics, such as the average packet loss ratio during
+ some time interval, would be affected to a lesser extent.
+
+ If it is necessary to correlate sampled values or intervals, then it
+ is essential that the accuracy of sampling time and interval start/
+ stop times is sufficient for the application (for example, +/- 2%).
+
+5.5.2. Dependencies of Performance Metric Definitions on Related Events
+ or Metrics
+
+ Performance Metric definitions may explicitly or implicitly rely on
+ factors that may not be obvious. For example, the recognition of a
+ packet as being "lost" relies on having some method of knowing the
+ packet was actually lost (e.g., RTP sequence number), and some time
+ threshold after which a non-received packet is declared lost. It is
+ important that any such dependencies are recognized and incorporated
+ into the metric definition.
+
+
+
+
+Clark & Claise Best Current Practice [Page 16]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+5.5.3. Relationship between Performance Metric and Lower-Layer
+ Performance Metrics
+
+ Lower-layer Performance Metrics may be used to compute or infer the
+ performance of higher-layer applications, potentially using an
+ application performance model. The accuracy of this will depend on
+ many factors, including:
+
+ (i) The completeness of the set of metrics (i.e., are there
+ metrics for all the input values to the application performance
+ model?)
+
+ (ii) Correlation between input variables (being measured) and
+ application performance
+
+ (iii) Variability in the measured metrics and how this variability
+ affects application performance
+
+5.5.4. Middlebox Presence
+
+ Presence of a middlebox [RFC3303], e.g., proxy, network address
+ translation (NAT), redirect server, session border controller (SBC)
+ [RFC5853], and application layer gateway (ALG), may add variability
+ to or restrict the scope of measurements of a metric. For example,
+ an SBC that does not process RTP loopback packets may block or
+ locally terminate this traffic rather than pass it through to its
+ target.
+
+5.6. Organization of Results
+
+ The IPPM Framework [RFC2330] organizes the results of metrics into
+ three related notions:
+
+ o singleton: an elementary instance, or "atomic" value.
+
+ o sample: a set of singletons with some common properties and some
+ varying properties.
+
+ o statistic: a value derived from a sample through deterministic
+ calculation, such as the mean.
+
+ Performance Metrics MAY use this organization for the results, with
+ or without the term names used by the IPPM WG. Section 11 of
+ RFC 2330 [RFC2330] should be consulted for further details.
+
+
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 17]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+5.7. Parameters: the Variables of a Performance Metric
+
+ Metrics are completely defined when all options and input variables
+ have been identified and considered. These variables are sometimes
+ left unspecified in a metric definition, and their general name
+ indicates that the user must set and report them with the results.
+ Such variables are called "parameters" in the IPPM metric template.
+ The scope of the metric, the time at which it was conducted, the
+ length interval of the sliding-window measurement, the settings for
+ timers, and the thresholds for counters are all examples of
+ parameters.
+
+ All documents defining Performance Metrics SHOULD identify all key
+ parameters for each Performance Metric.
+
+6. Performance Metric Development Process
+
+6.1. New Proposals for Performance Metrics
+
+ This process is intended to add more considerations to the processes
+ for adopting new work as described in RFC 2026 [RFC2026] and RFC 2418
+ [RFC2418]. Note that new Performance Metrics work item proposals
+ SHALL be approved using the existing IETF process. The following
+ entry criteria will be considered for each proposal.
+
+ Proposals SHOULD be prepared as Internet-Drafts, describing the
+ Performance Metric and conforming to the qualifications above as much
+ as possible. Proposals SHOULD be deliverables of the corresponding
+ protocol development WG charters. As such, the proposals SHOULD be
+ vetted by that WG prior to discussion by the Performance Metrics
+ Directorate. This aspect of the process includes an assessment of
+ the need for the Performance Metric proposed and assessment of the
+ support for its development in the IETF.
+
+ Proposals SHOULD include an assessment of interaction and/or overlap
+ with work in other Standards Development Organizations (SDOs).
+ Proposals SHOULD identify additional expertise that might be
+ consulted.
+
+ Proposals SHOULD specify the intended audience and users of the
+ Performance Metrics. The development process encourages
+ participation by members of the intended audience.
+
+ Proposals SHOULD identify any security and IANA requirements.
+ Security issues could potentially involve revealing data identifying
+ a user, or the potential misuse of active test tools. IANA
+ considerations may involve the need for a Performance Metrics
+ registry.
+
+
+
+Clark & Claise Best Current Practice [Page 18]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+6.2. Reviewing Metrics
+
+ Each Performance Metric SHOULD be assessed according to the following
+ list of qualifications:
+
+ o Are the performance metrics unambiguously defined?
+
+ o Are the units of measure specified?
+
+ o Does the metric clearly define the measurement interval where
+ applicable?
+
+ o Are significant sources of measurement errors identified and
+ discussed?
+
+ o Does the method of measurement ensure that results are repeatable?
+
+ o Does the metric or method of measurement appear to be
+ implementable (or offer evidence of a working implementation)?
+
+ o Are there any undocumented assumptions concerning the underlying
+ process that would affect an implementation or interpretation of
+ the metric?
+
+ o Can the metric results be related to application performance or
+ user experience, when such a relationship is of value?
+
+ o Is there an existing relationship to metrics defined elsewhere
+ within the IETF or within other SDOs?
+
+ o Do the security considerations adequately address denial-of-
+ service attacks, unwanted interference with the metric/
+ measurement, and user data confidentiality (when measuring live
+ traffic)?
+
+6.3. Performance Metrics Directorate Interaction with Other WGs
+
+ The Performance Metrics Directorate SHALL provide guidance to the
+ related protocol development WG when considering an Internet-Draft
+ that specifies Performance Metrics for a protocol. A sufficient
+ number of individuals with expertise must be willing to consult on
+ the draft. If the related WG has concluded, comments on the proposal
+ should still be sought from key RFC authors and former chairs.
+
+ As with expert reviews performed by other directorates, a formal
+ review is recommended by the time the document is reviewed by the
+ Area Directors or an IETF Last Call is being conducted.
+
+
+
+
+Clark & Claise Best Current Practice [Page 19]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+ Existing mailing lists SHOULD be used; however, a dedicated mailing
+ list MAY be initiated if necessary to facilitate work on a draft.
+
+ In some cases, it will be appropriate to have the IETF session
+ discussion during the related protocol WG session, to maximize
+ visibility of the effort to that WG and expand the review.
+
+6.4. Standards Track Performance Metrics
+
+ The Performance Metrics Directorate will assist with the progression
+ of RFCs along the Standards Track. See [IPPM-STANDARD-ADV-TESTING].
+ This may include the preparation of test plans to examine different
+ implementations of the metrics to ensure that the metric definitions
+ are clear and unambiguous (depending on the final form of the draft
+ mentioned above).
+
+7. Security Considerations
+
+ In general, the existence of a framework for Performance Metric
+ development does not constitute a security issue for the Internet.
+ Performance Metric definitions, however, may introduce security
+ issues, and this framework recommends that persons defining
+ Performance Metrics should identify any such risk factors.
+
+ The security considerations that apply to any active measurement of
+ live networks are relevant here. See [RFC4656].
+
+ The security considerations that apply to any passive measurement of
+ specific packets in live networks are relevant here as well. See the
+ security considerations in [RFC5475].
+
+8. Acknowledgements
+
+ The authors would like to thank Al Morton, Dan Romascanu, Daryl
+ Malas, and Loki Jorgenson for their comments and contributions, and
+ Aamer Akhter, Yaakov Stein, Carsten Schmoll, and Jan Novak for their
+ reviews.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 20]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+9. References
+
+9.1. Normative References
+
+ [RFC2026] Bradner, S., "The Internet Standards Process --
+ Revision 3", BCP 9, RFC 2026, October 1996.
+
+ [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
+ Requirement Levels", BCP 14, RFC 2119, March 1997.
+
+ [RFC2418] Bradner, S., "IETF Working Group Guidelines and
+ Procedures", BCP 25, RFC 2418, September 1998.
+
+ [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M.
+ Zekauskas, "A One-way Active Measurement Protocol
+ (OWAMP)", RFC 4656, September 2006.
+
+9.2. Informative References
+
+ [E.800] "ITU-T Recommendation E.800. E SERIES: OVERALL NETWORK
+ OPERATION, TELEPHONE SERVICE, SERVICE OPERATION AND HUMAN
+ FACTORS", September 2008.
+
+ [G.107] "ITU-T Recommendation G.107. The E-model: a computational
+ model for use in transmission planning", April 2009.
+
+ [IPPM-STANDARD-ADV-TESTING]
+ Geib, R., Ed., Morton, A., Fardid, R., and A. Steinmitz,
+ "IPPM standard advancement testing", Work in Progress,
+ June 2011.
+
+ [P.564] "ITU-T Recommendation P.564. Conformance Testing for
+ Voice over IP Transmission Quality Assessment Models",
+ November 2007.
+
+ [P.800] "ITU-T Recommendation P.800. Methods for subjective
+ determination of transmission quality", August 1996.
+
+ [RFC0793] Postel, J., "Transmission Control Protocol", STD 7,
+ RFC 793, September 1981.
+
+ [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
+ "Framework for IP Performance Metrics", RFC 2330,
+ May 1998.
+
+
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 21]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+ [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
+ A., Peterson, J., Sparks, R., Handley, M., and E.
+ Schooler, "SIP: Session Initiation Protocol", RFC 3261,
+ June 2002.
+
+ [RFC3303] Srisuresh, P., Kuthan, J., Rosenberg, J., Molitor, A., and
+ A. Rayhan, "Middlebox communication architecture and
+ framework", RFC 3303, August 2002.
+
+ [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
+ Jacobson, "RTP: A Transport Protocol for Real-Time
+ Applications", STD 64, RFC 3550, July 2003.
+
+ [RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed.,
+ "RTP Control Protocol Extended Reports (RTCP XR)",
+ RFC 3611, November 2003.
+
+ [RFC4710] Siddiqui, A., Romascanu, D., and E. Golovinsky, "Real-time
+ Application Quality-of-Service Monitoring (RAQMON)
+ Framework", RFC 4710, October 2006.
+
+ [RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol",
+ RFC 4960, September 2007.
+
+ [RFC5101] Claise, B., Ed., "Specification of the IP Flow Information
+ Export (IPFIX) Protocol for the Exchange of IP Traffic
+ Flow Information", RFC 5101, January 2008.
+
+ [RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J.
+ Meyer, "Information Model for IP Flow Information Export",
+ RFC 5102, January 2008.
+
+ [RFC5474] Duffield, N., Ed., Chiou, D., Claise, B., Greenberg, A.,
+ Grossglauser, M., and J. Rexford, "A Framework for Packet
+ Selection and Reporting", RFC 5474, March 2009.
+
+ [RFC5475] Zseby, T., Molina, M., Duffield, N., Niccolini, S., and F.
+ Raspall, "Sampling and Filtering Techniques for IP Packet
+ Selection", RFC 5475, March 2009.
+
+ [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation
+ Applicability Statement", RFC 5481, March 2009.
+
+ [RFC5706] Harrington, D., "Guidelines for Considering Operations and
+ Management of New Protocols and Protocol Extensions",
+ RFC 5706, November 2009.
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 22]
+
+RFC 6390 Guidelines Perf. Metric Devel. October 2011
+
+
+ [RFC5835] Morton, A., Ed., and S. Van den Berghe, Ed., "Framework
+ for Metric Composition", RFC 5835, April 2010.
+
+ [RFC5853] Hautakorpi, J., Ed., Camarillo, G., Penfield, R.,
+ Hawrylyshen, A., and M. Bhatia, "Requirements from Session
+ Initiation Protocol (SIP) Session Border Control (SBC)
+ Deployments", RFC 5853, April 2010.
+
+ [RFC5982] Kobayashi, A., Ed., and B. Claise, Ed., "IP Flow
+ Information Export (IPFIX) Mediation: Problem Statement",
+ RFC 5982, August 2010.
+
+ [RFC6035] Pendleton, A., Clark, A., Johnston, A., and H. Sinnreich,
+ "Session Initiation Protocol Event Package for Voice
+ Quality Reporting", RFC 6035, November 2010.
+
+ [RFC6049] Morton, A. and E. Stephan, "Spatial Composition of
+ Metrics", RFC 6049, January 2011.
+
+ [RFC6183] Kobayashi, A., Claise, B., Muenz, G., and K. Ishibashi,
+ "IP Flow Information Export (IPFIX) Mediation: Framework",
+ RFC 6183, April 2011.
+
+ [RFC6248] Morton, A., "RFC 4148 and the IP Performance Metrics
+ (IPPM) Registry of Metrics Are Obsolete", RFC 6248,
+ April 2011.
+
+Authors' Addresses
+
+ Alan Clark
+ Telchemy Incorporated
+ 2905 Premiere Parkway, Suite 280
+ Duluth, Georgia 30097
+ USA
+
+ EMail: alan.d.clark@telchemy.com
+
+
+ Benoit Claise
+ Cisco Systems, Inc.
+ De Kleetlaan 6a b1
+ Diegem 1831
+ Belgium
+
+ Phone: +32 2 704 5622
+ EMail: bclaise@cisco.com
+
+
+
+
+
+Clark & Claise Best Current Practice [Page 23]
+