diff options
Diffstat (limited to 'doc/rfc/rfc5982.txt')
-rw-r--r-- | doc/rfc/rfc5982.txt | 1403 |
1 files changed, 1403 insertions, 0 deletions
diff --git a/doc/rfc/rfc5982.txt b/doc/rfc/rfc5982.txt new file mode 100644 index 0000000..357f032 --- /dev/null +++ b/doc/rfc/rfc5982.txt @@ -0,0 +1,1403 @@ + + + + + + +Internet Engineering Task Force (IETF) A. Kobayashi, Ed. +Request for Comments: 5982 NTT PF Lab. +Category: Informational B. Claise, Ed. +ISSN: 2070-1721 Cisco Systems, Inc. + August 2010 + + + IP Flow Information Export (IPFIX) Mediation: Problem Statement + +Abstract + + Flow-based measurement is a popular method for various network + monitoring usages. The sharing of flow-based information for + monitoring applications having different requirements raises some + open issues in terms of measurement system scalability, flow-based + measurement flexibility, and export reliability that IP Flow + Information Export (IPFIX) Mediation may help resolve. This document + describes some problems related to flow-based measurement that + network administrators have been facing, and then it describes IPFIX + Mediation applicability examples along with the problems. + +Status of This Memo + + This document is not an Internet Standards Track specification; it is + published for informational purposes. + + This document is a product of the Internet Engineering Task Force + (IETF). It represents the consensus of the IETF community. It has + received public review and has been approved for publication by the + Internet Engineering Steering Group (IESG). Not all documents + approved by the IESG are a candidate for any level of Internet + Standard; see Section 2 of RFC 5741. + + Information about the current status of this document, any errata, + and how to provide feedback on it may be obtained at + http://www.rfc-editor.org/info/rfc5982. + +Copyright Notice + + Copyright (c) 2010 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (http://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + + + +Kobayashi and Claise Informational [Page 1] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the Simplified BSD License. + +Table of Contents + + 1. Introduction ....................................................3 + 2. Terminology and Definitions .....................................3 + 3. IPFIX/PSAMP Documents Overview ..................................5 + 3.1. IPFIX Documents Overview ...................................5 + 3.2. PSAMP Documents Overview ...................................5 + 4. Problem Statement ...............................................5 + 4.1. Coping with IP Traffic Growth ..............................6 + 4.2. Coping with Multipurpose Traffic Measurement ...............7 + 4.3. Coping with Heterogeneous Environments .....................7 + 4.4. Summary ....................................................7 + 5. Mediation Applicability Examples ................................8 + 5.1. Adjusting Flow Granularity .................................8 + 5.2. Collecting Infrastructure ..................................8 + 5.3. Correlation for Data Records ...............................9 + 5.4. Time Composition ...........................................9 + 5.5. Spatial Composition .......................................10 + 5.6. Data Record Anonymization .................................11 + 5.7. Data Retention ............................................11 + 5.8. IPFIX Export from a Branch Office .........................12 + 5.9. Distributing Data Record Types ............................13 + 5.10. Flow-Based Sampling and Selection ........................14 + 5.11. Interoperability between Legacy Protocols and IPFIX ......15 + 6. IPFIX Mediators' Implementation-Specific Problems ..............15 + 6.1. Loss of Original Exporter Information .....................15 + 6.2. Loss of Base Time Information .............................16 + 6.3. Transport Sessions Management .............................16 + 6.4. Loss of Options Template Information ......................16 + 6.5. Template ID Management ....................................17 + 6.6. Consideration for Network Topology ........................18 + 6.7. IPFIX Mediation Interpretation ............................18 + 6.8. Consideration for Aggregation .............................19 + 7. Summary and Conclusion .........................................20 + 8. Security Considerations ........................................20 + 9. Acknowledgements ...............................................21 + 10. References ....................................................22 + 10.1. Normative References .....................................22 + 10.2. Informative References ...................................22 + Contributors ......................................................24 + + + + + + + +Kobayashi and Claise Informational [Page 2] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + +1. Introduction + + An advantage of flow-based measurement is that it allows monitoring + large amounts of traffic observed at distributed Observation Points. + While flow-based measurement can be applied to one of various + purposes and applications, it is difficult for flow-based measurement + to apply to multiple applications with very different requirements in + parallel. Network administrators need to adjust the parameters of + the metering devices to fulfill the requirements of every single + measurement application. Such configurations are often not supported + by the metering devices, either because of functional restrictions or + because of limited computational and memory resources, which inhibit + the metering of large amounts of traffic with the desired setup. IP + Flow Information Export (IPFIX) Mediation fills the gap between + restricted metering capabilities and the requirements of measurement + applications by introducing an intermediate device called the IPFIX + Mediator. + + The IPFIX requirements defined in [RFC3917] mention examples of + intermediate devices located between Exporters and Collectors, such + as IPFIX proxies or concentrators. But, there are no documents + defining a generalized concept for such intermediate devices. This + document addresses that issue by defining IPFIX Mediation -- a + generalized intermediate device concept for IPFIX -- and examining in + detail the motivations behind its application. + + This document is structured as follows: Section 2 describes the + terminology used in this document, Section 3 gives an IPFIX/Packet + Sampling (PSAMP) document overview, Section 4 introduces general + problems related to flow-based measurement, Section 5 describes some + applicability examples where IPFIX Mediation would be beneficial, + and, finally, Section 6 describes some problems an IPFIX Mediation + implementation might face. + +2. Terminology and Definitions + + The IPFIX-specific and PSAMP-specific terminology used in this + document is defined in [RFC5101] and [RFC5476], respectively. In + this document, as in [RFC5101] and [RFC5476], the first letter of + each IPFIX-specific and PSAMP-specific term is capitalized along with + the IPFIX Mediation-specific terms defined here. + + In this document, we call "record stream" a stream of records + carrying flow- or packet-based information. The records may be + encoded as IPFIX Data Records or in any other format. + + + + + + +Kobayashi and Claise Informational [Page 3] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + Original Exporter + + An Original Exporter is an IPFIX Device that hosts the Observation + Points where the metered IP packets are observed. + + IPFIX Mediation + + IPFIX Mediation is the manipulation and conversion of a record + stream for subsequent export using the IPFIX protocol. + + The following terms are used in this document to describe the + architectural entities used by IPFIX Mediation. + + Intermediate Process + + An Intermediate Process takes a record stream as its input from + Collecting Processes, Metering Processes, IPFIX File Readers, + other Intermediate Processes, or other record sources; performs + some transformations on this stream, based upon the content of + each record, states maintained across multiple records, or other + data sources; and passes the transformed record stream as its + output to Exporting Processes, IPFIX File Writers, or other + Intermediate Processes, in order to perform IPFIX Mediation. + Typically, an Intermediate Process is hosted by an IPFIX Mediator. + Alternatively, an Intermediate Process may be hosted by an + Original Exporter. + + IPFIX Mediator + + An IPFIX Mediator is an IPFIX Device that provides IPFIX Mediation + by receiving a record stream from some data sources, hosting one + or more Intermediate Processes to transform that stream, and + exporting the transformed record stream into IPFIX Messages via an + Exporting Process. In the common case, an IPFIX Mediator receives + a record stream from a Collecting Process, but it could also + receive a record stream from data sources not encoded using IPFIX, + e.g., in the case of conversion from the NetFlow V9 protocol + [RFC3954] to the IPFIX protocol. + + Note that the IPFIX Mediator is a generalization of the + concentrator and proxy elements envisioned in the IPFIX + requirements [RFC3917]. IPFIX Mediators running appropriate + Intermediate Processes provide the functionality specified + therein. + + + + + + + +Kobayashi and Claise Informational [Page 4] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + +3. IPFIX/PSAMP Documents Overview + + IPFIX Mediation can be applied to Flow- or packet-based information. + The Flow-based information is encoded as IPFIX Flow Records by the + IPFIX protocol, and the packet-based information is extracted by some + packet selection techniques and then encoded as PSAMP Packet Reports + by the PSAMP protocol. Thus, this section describes relevant + documents for both protocols. + +3.1. IPFIX Documents Overview + + The IPFIX protocol [RFC5101] provides network administrators with + access to IP flow information. The architecture for the export of + measured IP flow information from an IPFIX Exporting Process to a + Collecting Process is defined in [RFC5470], per the requirements + defined in [RFC3917]. The IPFIX protocol [RFC5101] specifies how + IPFIX Data Records and Templates are carried via a number of + transport protocols from IPFIX Exporting Processes to IPFIX + Collecting Processes. IPFIX has a formal description of IPFIX + Information Elements, their names, types, and additional semantic + information, as specified in [RFC5102]. [RFC5815] specifies the + IPFIX Management Information Base. Finally, [RFC5472] describes what + types of applications can use the IPFIX protocol and how they can use + the information provided. Furthermore, it shows how the IPFIX + framework relates to other architectures and frameworks. The storage + of IPFIX Messages in a file is specified in [RFC5655]. + +3.2. PSAMP Documents Overview + + The framework for packet selection and reporting [RFC5474] enables + network elements to select subsets of packets by statistical and + other methods and to export a stream of reports on the selected + packets to a Collector. The set of packet selection techniques + (Sampling and Filtering) standardized by PSAMP is described in + [RFC5475]. The PSAMP protocol [RFC5476] specifies the export of + packet information from a PSAMP Exporting Process to a Collector. + Like IPFIX, PSAMP has a formal description of its Information + Elements, their names, types, and additional semantic information. + The PSAMP information model is defined in [RFC5477]. [PSAMP-MIB] + describes the PSAMP Management Information Base. + +4. Problem Statement + + Network administrators generally face the problems of measurement + system scalability, Flow-based measurement flexibility, and export + reliability, even if some techniques, such as Packet Sampling, + Filtering, Data Records aggregation, and export replication, have + already been developed. The problems consist of adjusting some + + + +Kobayashi and Claise Informational [Page 5] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + parameters of metering devices to resources of the measurement system + while fulfilling appropriate conditions: data accuracy, Flow + granularity, and export reliability. These conditions depend on two + factors. + + o Measurement system capacity: This consists of the bandwidth of the + management network, the storage capacity, and the performances of + the collecting devices and exporting devices. + + o Application requirements: Different applications, such as traffic + engineering, detecting traffic anomalies, and accounting, impose + different Flow Record granularities, and data accuracies. + + The sustained growth of IP traffic has been overwhelming the + capacities of measurement systems. Furthermore, a large variety of + applications (e.g., Quality-of-Service (QoS) measurement, traffic + engineering, security monitoring) and the deployment of measurement + systems in heterogeneous environments have been increasing the demand + and complexity of IP traffic measurements. + +4.1. Coping with IP Traffic Growth + + Enterprise or service provider networks already have multiple 10 Gb/s + links, their total traffic exceeding 100 Gb/s. In the near future, + broadband users' traffic will increase by approximately 40% every + year according to [TRAFGRW]. When administrators monitor IP traffic + sustaining its growth at multiple Exporters, the amount of exported + Flow Records from Exporters could exceed the ability of a single + Collector. + + To deal with this problem, current data reduction techniques (Packet + Sampling and Filtering in [RFC5475], and aggregation of measurement + data) have been generally implemented on Exporters. Note that Packet + Sampling leads to potential loss of small Flows. With both Packet + Sampling and aggregation techniques, administrators might no longer + be able to detect and investigate subtle traffic changes and + anomalies, as this requires detailed Flow information. With + Filtering, only a subset of the Data Records are exported. + + Considering the potential drawbacks of Packet Sampling, Filtering, + and Data Records aggregation, there is a need for a large-scale + collecting infrastructure that does not rely on data reduction + techniques. + + + + + + + + +Kobayashi and Claise Informational [Page 6] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + +4.2. Coping with Multipurpose Traffic Measurement + + Different monitoring applications impose different requirements on + the monitoring infrastructure. Some of them require traffic + monitoring at a Flow level while others need information about + individual packets or just Flow aggregates. + + To fulfill these diverse requirements, an Exporter would need to + perform various complex metering tasks in parallel, which is a + problem due to limited resources. Hence, it can be advantageous to + run the Exporter with a much simpler setup and to perform appropriate + post-processing of the exported Data Records at a later stage. + +4.3. Coping with Heterogeneous Environments + + Network administrators use IPFIX Devices and PSAMP Devices from + various vendors, various software versions, and various device types + (router, switch, or probe) in a single network domain. Even legacy + flow export protocols are still deployed in current networks. This + heterogeneous environment leads to differences in Metering Process + capabilities, Exporting Process capacity (export rate, cache memory, + etc.), and data format. For example, probes and switches cannot + retrieve some derived packet properties from a routing table. + + To deal with this problem, the measurement system needs to mediate + the differences. However, equipping all collecting devices with this + absorption function is difficult. + +4.4. Summary + + Due to resource limitations of the measurement system, it is + important to use traffic data reduction techniques as early as + possible, e.g., at the Exporter. However, this implementation is + made difficult by the heterogeneous environment of exporting devices. + On the other hand, keeping data accuracy and Flow granularity to meet + the requirements of different monitoring applications requires a + scalable and flexible collecting infrastructure. + + This implies that a new Mediation function is required in typical + Exporter-Collector architectures. Based on some applicability + examples, the next section shows the limitation of the typical + Exporter-Collector architecture model and the IPFIX Mediation + benefits. + + + + + + + + +Kobayashi and Claise Informational [Page 7] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + +5. Mediation Applicability Examples + +5.1. Adjusting Flow Granularity + + The simplest set of Flow Keys is a fixed 5-tuple of protocol, source + and destination IP addresses, and source and destination port + numbers. A shorter set of Flow Keys, such as a triple, a double, or + a single property, (for example, network prefix, peering autonomous + system number, or BGP Next-Hop fields), creates more aggregated Flow + Records. This is especially useful for measuring router-level + traffic matrices in a core network domain and for easily adjusting + the performance of Exporters and Collectors. + + Implementation analysis: + + Implementations for this case depend on where Flow granularity is + adjusted. More suitable implementations use configurable Metering + Processes in Original Exporters. The cache in the Metering + Process can specify its own set of Flow Keys and extra fields. + The Original Exporter thus generates Flow Records of the desired + Flow granularity. + + In the case where a Metering Process hosting no ability to change + the Flow Keys in Original Exporters creates Flow Records, or PSAMP + Packet Reports, an IPFIX Mediator can aggregate Data Records based + on a new set of Flow Keys. Even in the case of a Metering Process + hosting this ability, an IPFIX Mediator can further aggregate the + Flow Records. + +5.2. Collecting Infrastructure + + Increasing numbers of IPFIX Exporters, IP traffic growth, and the + variety of treatments expected to be performed on the Data Records + make it more and more difficult to implement all measurement + applications within a single Collector. + + Implementation analysis: + + To increase the collecting (e.g., the bandwidth capacity) and + processing capacity, distributed Collectors close to Exporters + need to be deployed. In such a case, those Collectors would + become IPFIX Mediators, re-exporting Data Records on demand to + centralized applications. To cope with the variety of measurement + applications, one possible implementation uses an Intermediate + Process deciding to which Collector(s) each record is exported. + More specific cases are described in Section 5.9. + + + + + +Kobayashi and Claise Informational [Page 8] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + +5.3. Correlation for Data Records + + The correlation amongst Data Records or between Data Records and + metadata provides new metrics or information, including the + following. + + o One-to-one correlation between Data Records + + * One-way delay from the correlation of PSAMP Packet Reports from + different Exporters along a specific path. For example, one- + way delay is calculated from the correlation of two PSAMP + Packet Reports, including the packet digest and the arrival + time at the Observation Point. This scenario is described in + Section 6.2.1.2 of [RFC5475]. + + * Packet inter-arrival time from the correlation of sequential + PSAMP Packet Reports from an Exporter. + + * Treatment from the correlation of Data Records with common + properties, observed at incoming/outgoing interfaces. Examples + are the rate-limiting ratio, the compression ratio, the + optimization ratio, etc. + + o Correlation amongst Data Records + + Average/maximum/minimum values from correlating multiple Data + Records. Examples are the average/maximum/minimum number of + packets of the measured Flows, the average/maximum/minimum one-way + delay, the average/maximum/minimum number of lost packets, etc. + + o Correlation between Data Records and other metadata + + Examples are some BGP attributes associated with Data Records, as + determined via routing table lookup. + + Implementation analysis: + + One possible implementation for this case uses an Intermediate + Process located between the Metering Processes and Exporting + Processes on the Original Exporter, or alternatively, a separate + IPFIX Mediator located between the Original Exporters and IPFIX + Collectors. + +5.4. Time Composition + + Time composition is defined as the aggregation of consecutive Data + Records with identical Flow Keys. It leads to the same output as + setting a longer active timeout on Original Exporters, with one + + + +Kobayashi and Claise Informational [Page 9] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + advantage: the creation of new metrics such as average, maximum, and + minimum values from Flow Records with a shorter time interval enables + administrators to keep track of changes that might have happened + during the time interval. + + Implementation analysis: + + One possible implementation for this case uses an Intermediate + Process located between the Metering Processes and Exporting + Processes on the Original Exporter, or alternatively a separate + IPFIX Mediator located between the Original Exporters and IPFIX + Collectors. + +5.5. Spatial Composition + + Spatial composition is defined as the aggregation of Data Records in + a set of Observation Points within an Observation Domain, across + multiple Observation Domains from a single Exporter, or even across + multiple Exporters. The spatial composition is divided into four + types. + + o Case 1: Spatial composition within one Observation Domain + + For example, to measure the traffic for a single logical interface + in the case in which link aggregation [IEEE802.3ad] exists, Data + Records metered at physical interfaces belonging to the same trunk + can be merged. + + o Case 2: Spatial composition across Observation Domains, but within + a single Original Exporter + + For example, in the case in which link aggregation exists, Data + Records metered at physical interfaces belonging to the same trunk + grouping beyond the line card can be merged. + + o Case 3: Spatial composition across Exporters + + Data Records metered within an administrative domain, such as the + west area and east area of an ISP network, can be merged. + + o Case 4: Spatial composition across administrative domains + + Data Records metered across administrative domains, such as across + different customer networks or different ISP networks, can be + merged. For example, a unique Collector knows in which customer + network an Exporter exists, and then works out the traffic data + per customer based on the Exporter IP address. + + + + +Kobayashi and Claise Informational [Page 10] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + Implementation analysis: + + One possible implementation for cases 1 and 2 uses an Intermediate + Process located between the Metering Processes and Exporting + Processes on the Original Exporter. A separate IPFIX Mediator + located between the Original Exporters and IPFIX Collectors is a + valid solution for cases 1, 2, 3, and 4. + +5.6. Data Record Anonymization + + IPFIX exports across administrative domains can be used to measure + traffic for wide-area traffic engineering or to analyze Internet + traffic trends, as described in the spatial composition across + administrative domains in the previous subsection. In such a case, + administrators need to adhere to privacy protection policies and + prevent access to confidential traffic measurements by other people. + Typically, anonymization techniques enable the provision of traffic + data to other people without violating these policies. + + Generally, anonymization modifies a data set to protect the identity + of the people or entities described by the data set from being + disclosed. It also attempts to preserve sets of network traffic + properties useful for a given analysis while ensuring the data cannot + be traced back to the specific networks, hosts, or users generating + the traffic. For example, IP address anonymization is particularly + important for avoiding the identification of users, hosts, and + routers. As another example, when an ISP provides traffic monitoring + service to end customers, network administrators take care of + anonymizing interface index fields that could disclose any + information about the vendor or software version of the Exporters. + + Implementation analysis: + + One possible implementation for this case uses an anonymization + function at the Original Exporter. However, this increases the + load on the Original Exporter. A more flexible implementation + uses a separate IPFIX Mediator between the Original Exporter and + Collector. + +5.7. Data Retention + + Data retention refers to the storage of traffic data by service + providers and commercial organizations. Legislative regulations + often require that network operators retain both IP traffic data and + call detail records, in wired and wireless networks, generated by end + + + + + + +Kobayashi and Claise Informational [Page 11] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + users while using a service provider's services. The traffic data is + required for the purpose of the investigation, detection, and + prosecution of serious crime, if necessary. Data retention examples + relevant to IP networks are the following: + + o Internet telephony (includes every multimedia session associated + with IP multimedia services) + + o Internet email + + o Internet access + + Data retention, for these services in particular, requires a + measurement system with reliable export and huge storage, as the data + must be available for a long period of time, typically at least six + months. + + Implementation analysis: + + Regarding export reliability requirement, the most suitable + implementation uses the Stream Control Transmission Protocol + (SCTP) between the Original Exporter and Collector. If an + unreliable transport protocol such as UDP is used, a legacy + exporting device exports Data Records to a nearby IPFIX Mediator + through UDP, and then an IPFIX Mediator could reliably export them + to the IPFIX Collector through SCTP. If an unreliable transport + protocol such as UDP is used and if there is no IPFIX Mediator, + the legacy exporting device should duplicate the exports to + several Collectors to lower the probability of losing Flow + Records. However, it might result in network congestion, unless + dedicated export links are used. + + Regarding huge storage requirements, the collecting infrastructure + is described in Section 5.2. + +5.8. IPFIX Export from a Branch Office + + Generally, in large enterprise networks, Data Records from branch + offices are gathered in a central office. However, in the long- + distance branch office case, the bandwidth for transporting IPFIX is + limited. Therefore, even if multiple Data Record types should be of + interest to the Collector (e.g., IPFIX Flow Records in both + directions, IPFIX Flow Records before and after WAN optimization + techniques, performance metrics associated with the IPFIX Flow + Records exported at regular intervals, etc.), the export bandwidth + limitation is an important factor to pay attention to. + + + + + +Kobayashi and Claise Informational [Page 12] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + Implementation analysis: + + One possible implementation for this case uses an IPFIX Mediator + located in a branch office. The IPFIX Mediator would aggregate + and correlate Data Records to cope with the export bandwidth + limitation. + +5.9. Distributing Data Record Types + + Recently, several networks have shifted towards integrated networks, + such as the pure IP and MPLS networks, which include IPv4, IPv6, and + VPN traffic. Data Record types (IPv4, IPv6, MPLS, and VPN) need to + be analyzed separately and from different perspectives for different + organizations. A single Collector handling all Data Record types + might become a bottleneck in the collecting infrastructure. Data + Records distributed based on their respective types can be exported + to the appropriate Collector, resulting in load distribution amongst + multiple Collectors. + + Implementation analysis: + + One possible implementation for this case uses replication of the + IPFIX Message in an Original Exporter for multiple IPFIX + Collectors. Each Collector then extracts the Data Record required + by its own applications. However, this replication increases the + load of the Exporting Process and the waste of bandwidth between + the Exporter and Collector. + + A more sophisticated implementation uses an Intermediate Process + located between the Metering Processes and Exporting Processes in + an Original Exporter. The Intermediate Process determines to + which Collector a Data Record is exported, depending on certain + field values. If an Original Exporter does not have this + capability, it exports Data Records to a nearby separate IPFIX + Mediator, and then the IPFIX Mediator could distribute them to the + appropriate IPFIX Collectors. + + For example, in the case of distributing a specific customer's + Data Records, an IPFIX Mediator needs to identify the customer + networks. The Route Distinguisher (RD), ingress interface, + peering Autonomous System (AS) number, or BGP Next-Hop, or simply + the network prefix may be evaluated to distinguish different + customer networks. In the following figure, the IPFIX Mediator + reroutes Data Records on the basis of the RD value. This system + enables each customer's traffic to be inspected independently. + + + + + + +Kobayashi and Claise Informational [Page 13] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + .---------. + |Traffic | + .---->|Collector|<==>Customer#A + | |#1 | + | '---------' + RD=100:1 + .----------. .-----------. | + |IPFIX | |IPFIX |----' .---------. + |Exporter#1| |Mediator | RD=100:2 |Traffic | + | |------->| |--------->|Collector|<==>Customer#B + | | | | |#2 | + | | | |----. '---------' + '----------' '-----------' | + RD=100:3 + | .---------. + | |Traffic | + '---->|Collector|<==>Customer#C + |#3 | + '---------' + + Figure A. Distributing Data Records to Collectors + Using IPFIX Mediator + +5.10. Flow-Based Sampling and Selection + + Generally, the distribution of the number of packets per Flow seems + to be heavy tailed. Most types of Flow Records are likely to be + small Flows consisting of a small number of packets. The measurement + system is overwhelmed with a huge amount of these small Flows. If + statistics information of small Flows is exported as merged data by + applying a policy or threshold, the load on the Exporter is reduced. + Furthermore, if the Flow distribution is known, exporting only a + subset of the Data Records might be sufficient. + + Implementation analysis: + + One possible implementation for this case uses an Intermediate + Process located between the Metering Processes and Exporting + Processes on the Original Exporter, or alternatively a separate + IPFIX Mediator located between the Original Exporters and IPFIX + Collectors. A set of IPFIX Mediation functions, such as + Filtering, selecting, and aggregation, is used in the IPFIX + Mediator. + + + + + + + + +Kobayashi and Claise Informational [Page 14] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + +5.11. Interoperability between Legacy Protocols and IPFIX + + During the migration process from a legacy protocol such as NetFlow + [RFC3954] to IPFIX, both NetFlow exporting devices and IPFIX + Exporters are likely to coexist in the same network. Operators need + to continue measuring the traffic data from legacy exporting devices, + even after introducing IPFIX Collectors. + + Implementation analysis: + + One possible implementation for this case uses an IPFIX Mediator + that converts a legacy protocol to IPFIX. + +6. IPFIX Mediators' Implementation-Specific Problems + +6.1. Loss of Original Exporter Information + + Both the Exporter IP address indicated by the source IP address of + the IPFIX Transport Session and the Observation Domain ID included in + the IPFIX Message header are likely to be lost during IPFIX + Mediation. In some cases, an IPFIX Mediator might drop the + information deliberately. In general, however, the Collector must + recognize the origin of the measurement information, such as the IP + address of the Original Exporter, the Observation Domain ID, or even + the Observation Point ID. Note that, if an IPFIX Mediator cannot + communicate the Original Exporter IP address, then the IPFIX + Collector will wrongly deduce that the IP address of the IPFIX + Mediator is that of the Original Exporter. + + In the following figure, a Collector can identify two IP addresses: + 192.0.2.3 (IPFIX Mediator) and 192.0.2.2 (Exporter#2), respectively. + The Collector, however, needs to somehow recognize both Exporter#1 + and Exporter#2, which are the Original Exporters. The IPFIX Mediator + must be able to notify the Collector about the IP address of the + Original Exporter. + + + + + + + + + + + + + + + + +Kobayashi and Claise Informational [Page 15] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + .----------. .--------. + |IPFIX | |IPFIX | + |Exporter#1|--------->|Mediator|---+ + | | | | | + '----------' '--------' | .---------. + IP:192.0.2.1 IP:192.0.2.3 '----->|IPFIX | + ODID:10 ODID:0 |Collector| + +------>| | + .----------. | '---------' + |IPFIX | | + |Exporter#2|-----------------------' + | | + '----------' + IP:192.0.2.2 + ODID:20 + + Figure B. Loss of Original Exporter Information + +6.2. Loss of Base Time Information + + The Export Time field included in the IPFIX Message header represents + a reference timestamp for Data Records. Some IPFIX Information + Elements, described in [RFC5102], carry delta timestamps that + indicate the time difference from the value of the Export Time field. + If the Data Records include any delta time fields and the IPFIX + Mediator overwrites the Export Time field when sending IPFIX + Messages, the delta time fields become meaningless and, because + Collectors cannot recognize this situation, wrong time values are + propagated. + +6.3. Transport Sessions Management + + Maintaining relationships between the incoming Transport Sessions and + the outgoing ones depends on the Mediator's implementation. If an + IPFIX Mediator relays multiple incoming Transport Sessions to a + single outgoing Transport Session, and if the IPFIX Mediator shuts + down its outgoing Transport Session, Data Records of the incoming + Transport Sessions would not be relayed anymore. In the case of + resetting an incoming Transport Session, the behavior of the IPFIX + Mediator needs to be specified. + +6.4. Loss of Options Template Information + + In some cases, depending on the implementation of the IPFIX + Mediators, the information reported in the Data Records defined by + Options Templates could also be lost. If, for example, the Sampling + rate is not communicated from the Mediator to the Collector, the + Collector would miscalculate the traffic volume. This might lead to + + + +Kobayashi and Claise Informational [Page 16] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + crucial problems. Even if an IPFIX Mediator were to simply relay + received Data Records defined by Options Templates, the values of its + scope fields could become meaningless in the content of a different + Transport Session. The minimal information to be communicated by an + IPFIX Mediator must be specified. + +6.5. Template ID Management + + The Template ID is unique on the basis of the Transport Session and + Observation Domain ID. If an IPFIX Mediator is not able to manage + the relationships amongst the Template IDs and the incoming Transport + Session information, and if the Template ID is used in the Options + Template scope, IPFIX Mediators would, for example, relay wrong + values in the scope field and in the Template Withdrawal Message. + The Collector would thus not be able to interpret the Template ID in + the Template Withdrawal Message and in the Options Template scope. + As a consequence, there is a risk that the Collector would then shut + down the IPFIX Transport Session. + + For example, an IPFIX Mediator must maintain the state of the + incoming Transport Sessions in order to manage the Template ID on its + outgoing Transport Session correctly. Even if the Exporter Transport + Session re-initializes, the IPFIX Mediator must manage the + association of Template IDs in a specific Transport Session. In the + following figure, the IPFIX Mediator exports three Templates (256, + 257, and 258), received from Exporter#3, Exporter#2, and Exporter#1, + respectively. If Exporter#1 re-initializes, and the Template ID + value 258 is now replaced with 256, the IPFIX Mediator must correctly + manage the new mapping of (incoming Transport Session, Template ID) + and (outgoing Transport Session, Template ID) without shutting down + its outgoing Transport Session. + + + + + + + + + + + + + + + + + + + + +Kobayashi and Claise Informational [Page 17] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + .----------. OLD: Template ID 258 + |IPFIX | NEW: Template ID 256 + |Exporter#1|----+ + | | | + '----------' X + .----------. | .-----------. .----------. + |IPFIX | '---------->| | | | + |Exporter#2|--------------->|IPFIX |-------------->|IPFIX | + | |Template ID 257 |Mediator |Template ID 258| Collector| + '----------' +---------->| |Template ID 257| | + .----------. | '-----------'Template ID 256'----------' + |IPFIX | | + |Exporter#3|----' + | | Template ID 256 + '----------' + + Figure C. Relaying from Multiple Transport Sessions + to a Single Transport Session + +6.6. Consideration for Network Topology + + While IPFIX Mediation can be applied anywhere, caution should be + taken as to how to aggregate the counters, as there is a potential + risk of double counting. For example, if three Exporters export + PSAMP Packet Reports related to the same flow, the one-way delay can + be calculated, while summing up the number of packets and bytes does + not make sense. Alternatively, if three Exporters export Flow + Records entering an administrative domain, then the sum of the + packets and bytes is a valid operation. Therefore, the possible + function to be applied to Flow Records must take into consideration + the measurement topology. The information such as the network + topology, or at least the Observation Point and measurement + direction, is required for IPFIX Mediation. + +6.7. IPFIX Mediation Interpretation + + In some cases, the IPFIX Collector needs to recognize which specific + function(s) IPFIX Mediation has executed on the Data Records. The + IPFIX Collector cannot distinguish between time composition and + spatial composition, if the IPFIX Mediator does not export the + applied function. Some parameters related to the function also would + need to be exported. For example, in the case of time composition, + the active timeout of original Flow Records is required to interpret + the minimum/maximum counter correctly. In the case of spatial + composition, spatial area information on which Data Records is + aggregated is required. + + + + + +Kobayashi and Claise Informational [Page 18] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + +6.8. Consideration for Aggregation + + Whether the aggregation is based on time or spatial composition, + caution should be taken regarding how to aggregate non-key fields in + IPFIX Mediation. The IPFIX information model [RFC5102] specifies + that the value of non-key fields, which are derived from fields of + packets or from packet treatment and for which the value may change + from packet to packet within a single Flow, is determined by the + first packet observed for the corresponding Flow, unless the + description of the Information Element explicitly specifies a + different semantics. + + However, this simple rule might not be appropriate when aggregating + Flow Records that have different values in a non-key field. For + example, if Differentiated Services Code Point (DSCP) information is + to be exported, the following problem can be observed: if two Flows + with identical Flow Key values are measured at different Observation + Points, they may contain identical packets observed at different + locations in the network and at different points in time. On their + way from the first to the second Observation Point, the DSCP and + potentially some other packet fields may have changed. Hence, if the + Information Element ipDiffServCodePoint is included as a non-key + field, it can be useful to include the DSCP value observed at either + the first or the second Observation Point in the resulting Flow + Record, depending on the application. + + Other potential solutions include removing the Information Element + ipDiffServCodePoint from the Data Record when re-exporting the + aggregate Flow Record, changing the Information Element + ipDiffServCodePoint from a non-key field to a Flow Key when + re-exporting the aggregated Flow Record, or assigning a non-valid + value for the Information Element to express to the Collector that + this Information Element is meaningless. + + If Packet Sampling or Filtering is applied, the IPFIX Mediator must + report an adjusted PSAMP Configured Selection Fraction when + aggregating IPFIX Flow Records with different Sampling rates. + + Finally, special care must be taken when aggregating Flow Records + resulting from different Sampling techniques such as Systematic + Count-Based Sampling and Random n-out-of-N Sampling, for example. + + + + + + + + + + +Kobayashi and Claise Informational [Page 19] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + +7. Summary and Conclusion + + This document describes the problems that network administrators have + been facing, the applicability of IPFIX Mediation to these problems, + and the problems related to the implementation of IPFIX Mediators. + To assist the operations of the Exporters and Collectors, this + document demonstrates that there exist various IPFIX Mediation + functions from which the administrators may select. + + However, there are still some open issues with the use of IPFIX + Mediators. These issues stem from the fact that no standards + regarding IPFIX Mediation have been set. In particular, the minimum + information that should be communicated between Original Exporters + and Collectors, the mapping between different IPFIX Transport + Sessions, and the internal components of IPFIX Mediators should be + standardized. + +8. Security Considerations + + A flow-based measurement system must prevent potential security + threats: the disclosure of confidential traffic data, injection of + incorrect data, and unauthorized access to traffic data. These + security threats of the IPFIX protocol are covered by the Security + Considerations section in [RFC5101] and are still valid for IPFIX + Mediators. + + A measurement system must also prevent the following security threats + related to IPFIX Mediation: + + o Attacks against an IPFIX Mediator + + IPFIX Mediators can be considered as a prime target for attacks, + as an alternative to IPFIX Exporters and Collectors. IPFIX + Proxies or Masquerading Proxies need to prevent unauthorized + access or denial-of-service (DoS) attacks from untrusted public + networks. + + o Man-in-the-middle attack by untrusted IPFIX Mediator + + The Exporter-Mediator-Collector structure model could be misused + for a man-in-the-middle attack. + + o Configuration on IPFIX Mediation + + An accidental misconfiguration and unauthorized access to + configuration data could lead to the crucial problem of disclosure + of confidential traffic data. + + + + +Kobayashi and Claise Informational [Page 20] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + o Unintentional exposure of end-user information + + The probability of collecting fine-grained information on one + arbitrary end user increases with the number of Observation + Points. An IPFIX Mediator facing such a situation may have to + apply appropriate functions (e.g., anonymization or aggregation) + to the Data Records it produces. + + o Multiple-tenancy policy on an IPFIX Mediator + + An IPFIX Mediator handling traffic data from multiple tenants or + customers needs to protect those tenants or customers from one + another's traffic data. For example, an IPFIX Mediator needs to + identify the customer's identifier, e.g., ingress interface index, + network address range, VLAN ID, Media Access Control (MAC) + address, etc., when feeding the customer's traffic data to a + customer's own dedicated IPFIX Collector. If the IPFIX Mediator + cannot identify each customer's traffic data, it may need to drop + the Data Records. In addition, another technique to keep track of + a customer's identifier may be required when customer sites are + movable, e.g., in the case of a virtual machine moving to another + physical machine. + + o Confidentiality protection via an IPFIX Mediator + + To ensure security of Data Records in transit, transport of Data + Records should be confidential and integrity-protected, e.g., by + using Transport Layer Security (TLS) [RFC5246] or Datagram + Transport Layer Security (DTLS) [RFC4347]. However, an IPFIX + Collector cannot know whether received Data Records are + transported as encrypted data between an Original Exporter and an + IPFIX Mediator. If this information is required on the IPFIX + Collector, it must be encoded in the IPFIX Mediator. + + o Certification for an Original Exporter + + An IPFIX Collector communicating via an IPFIX Mediator cannot + verify the identity of an Original Exporter directly. If an + Original Exporter and an IPFIX Collector are located in different + administrative domains, an IPFIX Collector cannot trust its Data + Records. If this information is required on the IPFIX Collector, + it must be encoded in the IPFIX Mediator. + +9. Acknowledgements + + We would like to thank the following persons: Gerhard Muenz for + thorough, detailed review and significant contributions regarding the + improvement of whole sections; Keisuke Ishibashi for contributions + + + +Kobayashi and Claise Informational [Page 21] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + during the initial phases of the document; Brian Trammell for + contributions regarding the improvement of the Terminology and + Definitions section; and Nevil Brownlee, Juergen Schoenwaelder, and + Motonori Shindo for their technical reviews and feedback. + +10. References + +10.1. Normative References + + [RFC5101] Claise, B., Ed., "Specification of the IP Flow + Information Export (IPFIX) Protocol for the Exchange + of IP Traffic Flow Information", RFC 5101, + January 2008. + + [RFC5476] Claise, B., Ed., Johnson, A., and J. Quittek, "Packet + Sampling (PSAMP) Protocol Specifications", RFC 5476, + March 2009. + +10.2. Informative References + + [IEEE802.3ad] IEEE Computer Society, "Link Aggregation", IEEE + Std 802.3ad-2000, March 2000. + + [PSAMP-MIB] Dietz, T., Ed., Claise, B., and J. Quittek, + "Definitions of Managed Objects for Packet Sampling", + Work in Progress, July 2010. + + [RFC3917] Quittek, J., Zseby, T., Claise, B., and S. Zander, + "Requirements for IP Flow Information Export (IPFIX)", + RFC 3917, October 2004. + + [RFC3954] Claise, B., Ed., "Cisco Systems NetFlow Services + Export Version 9", RFC 3954, October 2004. + + [RFC4347] Rescorla, E. and N. Modadugu, "Datagram Transport + Layer Security", RFC 4347, April 2006. + + [RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and + J. Meyer, "Information Model for IP Flow Information + Export", RFC 5102, January 2008. + + [RFC5246] Dierks, T. and E. Rescorla, "The Transport Layer + Security (TLS) Protocol Version 1.2", RFC 5246, August + 2008. + + [RFC5470] Sadasivan, G., Brownlee, N., Claise, B., and J. + Quittek, "Architecture for IP Flow Information + Export", RFC 5470, March 2009. + + + +Kobayashi and Claise Informational [Page 22] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + + [RFC5472] Zseby, T., Boschi, E., Brownlee, N., and B. Claise, + "IP Flow Information Export (IPFIX) Applicability", + RFC 5472, March 2009. + + [RFC5474] Duffield, N., Ed., Chiou, D., Claise, B., Greenberg, + A., Grossglauser, M., and J. Rexford, "A Framework for + Packet Selection and Reporting", RFC 5474, March 2009. + + [RFC5475] Zseby, T., Molina, M., Duffield, N., Niccolini, S., + and F. Raspall, "Sampling and Filtering Techniques for + IP Packet Selection", RFC 5475, March 2009. + + [RFC5477] Dietz, T., Claise, B., Aitken, P., Dressler, F., and + G. Carle, "Information Model for Packet Sampling + Exports", RFC 5477, March 2009. + + [RFC5655] Trammell, B., Boschi, E., Mark, L., Zseby, T., and A. + Wagner, "Specification of the IP Flow Information + Export (IPFIX) File Format", RFC 5655, October 2009. + + [RFC5815] Dietz, T., Ed., Kobayashi, A., Claise, B., and G. + Muenz, "Definitions of Managed Objects for IP Flow + Information Export", RFC 5815, April 2010. + + [TRAFGRW] Cho, K., Fukuda, K., Esaki, H., and A. Kato, "The + Impact and Implications of the Growth in Residential + User-to-User Traffic", SIGCOMM2006, pp. 207-218, Pisa, + Italy, September 2006. + + + + + + + + + + + + + + + + + + + + + + + +Kobayashi and Claise Informational [Page 23] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + +Contributors + + Haruhiko Nishida + NTT Information Sharing Platform Laboratories + 3-9-11 Midori-cho + Musashino-shi, Tokyo 180-8585 + Japan + + Phone: +81-422-59-3978 + EMail: nishida.haruhiko@lab.ntt.co.jp + + + Christoph Sommer + University of Erlangen-Nuremberg + Department of Computer Science 7 + Martensstr. 3 + Erlangen 91058 + Germany + + Phone: +49 9131 85-27993 + EMail: christoph.sommer@informatik.uni-erlangen.de + URI: http://www7.informatik.uni-erlangen.de/~sommer/ + + + Falko Dressler + University of Erlangen-Nuremberg + Department of Computer Science 7 + Martensstr. 3 + Erlangen 91058 + Germany + + Phone: +49 9131 85-27914 + EMail: dressler@informatik.uni-erlangen.de + URI: http://www7.informatik.uni-erlangen.de/~dressler/ + + + Stephan Emile + France Telecom + 2 Avenue Pierre Marzin + Lannion, F-22307 + France + + Fax: +33 2 96 05 18 52 + EMail: emile.stephan@orange-ftgroup.com + + + + + + + +Kobayashi and Claise Informational [Page 24] + +RFC 5982 IPFIX Mediation: Problem Statement August 2010 + + +Authors' Addresses + + Atsushi Kobayashi (editor) + NTT Information Sharing Platform Laboratories + 3-9-11 Midori-cho + Musashino-shi, Tokyo 180-8585 + Japan + + Phone: +81-422-59-3978 + EMail: akoba@nttv6.net + + + Benoit Claise (editor) + Cisco Systems, Inc. + De Kleetlaan 6a b1 + Diegem 1831 + Belgium + + Phone: +32 2 704 5622 + EMail: bclaise@cisco.com + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Kobayashi and Claise Informational [Page 25] + |