IPFIX Working Group A. Kobayashi Internet-Draft H. Nishida Intended status: Informational NTT PF Lab. Expires: April 18, 2008 C. Sommer F. Dressler Univ. Erlangen E. Stephan France Telecom October 16, 2007 Problems with Flow Collection in Large-Scale Networks draft-kobayashi-ipfix-large-ps-00.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on April 18, 2008. Copyright Notice Copyright (C) The IETF Trust (2007). Kobayashi, et al. Expires April 18, 2008 [Page 1] Internet-Draft Problem Statement in large NW October 2007 Abstract Flow measurement techniques have been developed in order to cope with increasing bandwiths. Flow measurement is currently being used as a popular method for monitoring large networks, e.g. at Internet Service Providers or multinational corporations, for accounting, management, and security purposes. To construct the measurement system for such networks, the biggest challenge is scalability. Whereas packet sampling functions in current flow measurement solutions such as NetFlow/sFlow/IPFIX help to approach this issue, some problems still remain. This document describes such open issues, which a flow-based measurement system might encounter in large-scale networks. The results are intended to be used to define and to develope an IPFIX mediator device. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Problem Statement . . . . . . . . . . . . . . . . . . . . . . 4 2.1. Traffic Measurement in Service Provider Networks . . . . . 4 2.2. Flow-Based Measurement in Integrated Networks . . . . . . 4 2.3. Approaches to Scalability . . . . . . . . . . . . . . . . 5 2.3.1. Allocating Multiple Collectors in Parallel . . . . . . 5 2.3.2. Adjusting Sampling Rates . . . . . . . . . . . . . . . 6 2.3.3. Exporting Aggregated Flows from Original Exporters . . 6 2.3.4. Using a Hierarchy of IPFIX Concentrators . . . . . . . 6 3. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 8 4. Security Considerations . . . . . . . . . . . . . . . . . . . 9 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 6. References . . . . . . . . . . . . . . . . . . . . . . . . . . 11 6.1. Normative References . . . . . . . . . . . . . . . . . . . 11 6.2. Informative References . . . . . . . . . . . . . . . . . . 11 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 12 Intellectual Property and Copyright Statements . . . . . . . . . . 14 Kobayashi, et al. Expires April 18, 2008 [Page 2] Internet-Draft Problem Statement in large NW October 2007 1. Introduction Flow-based measurement has dramatically improved the quality of traffic measurement methods available for service provider networks. It is quickly becoming one of the essential operation tools in many networks. Especially monitoring operations in large-scale networks need flow measurement techniques to ensure stable and efficient operations. However, network providers might soon encounter problematic situations because of scalability issues. With traffic volume ever increasing in large networks, the number of exported Flow Records becomes huge. Also, Flow information is required for multiple purposes and it is difficult to maintain the scalability of a flow-based measurement system and at the same time offer Flows in the way required for purposes such as Traffic Engineering, Troubleshooting, Accounting, and Security. This document provides the problem description and, thus, the requirements for an extension to IPFIX, the IPFIX mediator. The IPFIX applicability draft [I-D.ietf-ipfix-as] describes a variety of applications for the IPFIX protocol [I-D.ietf-ipfix-protocol]. In most networks, the flexibility of the IPFIX protocol could meet the requirements of a flow-based measurement system. But, from the viewpoint of large-scale and fast-growing networks, there are still some unresolved problems. This document describes such problematic cases in large-scale networks. Kobayashi, et al. Expires April 18, 2008 [Page 3] Internet-Draft Problem Statement in large NW October 2007 2. Problem Statement 2.1. Traffic Measurement in Service Provider Networks Flow measurement methods are one of the most popular measurement methods of network service providers. Usually, an operator measures traffic on several observation points for a specific purpose, using a certain level of sampling rate such as from 1/10,000 to 1/1,000. This value depends on the several factors, such as the capacity of the management network, the available storage and speed of the Collector, or the load of the routers/switches. This section describes some of use cases of flow-based measurement on the service provider networks. o In gateway routers, exchange traffic between neighboring AS is monitored using flow-based measurement. The result helps with the planing of peering policies and with traffic engineering to optimize the network resources. o In server side routers, the traffic toward a specific customer's server is measured. The collected data is used to detect traffic anomalies, such as distributed denial-of-service attacks and packet scans. Furthermore, this data is valuable for network forensics, when customers ask to investigate their server traffic. o In access routers connected toward the broadband customers, the traffic of each customer is measured to profile them. Operators pay special attention to heavy users that abuse network bandwidth by using P2P file exchange applications. Results give good feedback about the implemented billing policy. As described above, the network providers are making heavy use of Flow measurements. The number of Observation Points in the networks can even be increased to improve on the effectiveness of these methods. In the near future, we anticipate that the advanced features of IPFIX, such as monitoring wide-area traffic exchange behavior and QoS performance will accelerate the utilization. The increasing amount of the traffic that is brought about by broadband users might have a impact on the present measurement parameters, such as the sampling rate or the granularity of Flows. 2.2. Flow-Based Measurement in Integrated Networks Recently several networks seem to shift towards Integrated Networks, such as the Internet and MPLS, which include IPv4, IPv6 and VPN traffic. These sorts of Flow Records need to be analyzed separately and from different perspectives. However, handling them separately without improving the capability of the Collector is difficult. If Kobayashi, et al. Expires April 18, 2008 [Page 4] Internet-Draft Problem Statement in large NW October 2007 the original Exporter can not distribute specific Flow Records based on the Flow contents, a single Collector needs to be able to handle several kinds of Flow Records. Except for the original Exporter, other devices assisting in this function would thus be necessary. 2.3. Approaches to Scalability The volume of traffic on ordinary service provider networks has been increasing year by year. As the sizes of networks become larger, the amount of Flow Records becomes greater. The huge amount of Flow Records has been burdening management networks and Collecting Processes. Maintaining scalability is difficult as a particular network grows. Generally, large-scale networks already have multiple 10Gb/s links, their total traffic exceeding 100Gb/s. In the near future, broadband users' traffic will increase by approximately 50% per year according to [TRAFGRW]. When operators monitor traffic of 500Gb/s with a sampling rate of 1/1000, the amount of exported Flow Records from Exporters could exceed 50k f/s. This value reaches beyond the ability of a single Collector. It should be noted that the current sampling rate might not be able to be applied to the Exporter within the next five years. We consider the problem to be expected to encounter within the next five years in large-scale networks. In addition, network operators need to monitor the overall wide-scale traffic exchange, and investigate detailed traffic behavior when traffic incidents happen. Achieving this seems to be difficult for a single Collector because of the huge amount of Flow Records. To that end, the network operators can consider several solutions. The next sections describe how operators can cope with such a huge amount of Flow Records using available solutions of NetFlow/sFlow/ IPFIX. However, each solution also presents new problems. 2.3.1. Allocating Multiple Collectors in Parallel Generally speaking, allocating multiple Collectors to cope with the amount of Flow Records can resolve many scalability problems. But, in that case, network operators will have difficulties to get an overview of traffic exchange behavior in whole network. Most of Flow-based applications require the observation of flows from the whole network under observation. For example, they correspond to the wide-area traffic matrices that are the router-by-router or PoP-by- PoP traffic matrix. To get the wide-area traffic matrices, single Collector device needs to gather the Flow Records from the whole networks. Kobayashi, et al. Expires April 18, 2008 [Page 5] Internet-Draft Problem Statement in large NW October 2007 2.3.2. Adjusting Sampling Rates Adjusting the sampling rate definitely reduces the amount of Flow Records, and a flow-based measurement system can thus easily adapt to the ability of the Collecting and Exporting Processes. But, in that case, Flows with small traffic volume could easily get lost. When the traffic incidents happen, network operators can no longer investigate traffic behavior. While traffic volume on networks continues to increase, network operators will not be able to maintain the sampling rates currently used. In the near future, it is possible that flow-based measurement systems will not be able to detect traffic anomalies which can currently be detected. 2.3.3. Exporting Aggregated Flows from Original Exporters When using Flow aggregation, the Flow Records are generally aggregated based on network prefix, peering AS (autonomous system) number, or BGP Next-Hop. This solution is especially useful for measurements of wide-area traffic exchange. However, network operators can no longer monitor traffic behavior in-depth, the simplest type of Flows being those comprised of all packets having the same 5-tuple of protocol, source and destination IP addresses and port numbers. As one of solutions that resolve this problem, if the Exporter allows aggregation with multiple granularities, network operators can flexibly adapt the granularity of aggregation according to their requirements. An Exporter might, for example, by default export aggregated Flow Records. If the traffic incident happens, network operators could then change the Flow Key to measure fine-grained Flows. To implement this solution, a system that dynamically changes the Flow Key in cooperation with Exporters and Collectors is required. It is, however, difficult to replace all Exporters in the whole network, as all Exporters in the network will need to implement this solution. 2.3.4. Using a Hierarchy of IPFIX Concentrators In large-scale networks, creating a hierarchical measurement system by using IPFIX Concentrators, as described in [RFC3917], can prove to be very useful. If IPFIX Concentrators store the received Flow Records, and then the stored Flow Records are allowed to be retrieved by other devices, this architecture might actually become a most useful distributed collection system. As described in [I-D.dressler-ipfix-aggregation], in the case of a measurement system consisting of both aggregating Exporters and non-aggregating Exporters, an IPFIX Concentrator that can assist the latter by aggregating received Flow Records to any granularity. Kobayashi, et al. Expires April 18, 2008 [Page 6] Internet-Draft Problem Statement in large NW October 2007 However, IPFIX header information, such as Exporter IP address, Observation Domain ID and Export Time would likely be lost in the process of mediation that is performed by the Concentrator. Exporter IP address and Observation Domain ID indicate the observation point from the viewpoint of entire network domain. Export Time indicates the reference of time for flow information. Such information is necessary for guaranteeing the continuity of the work of the top level Collector. Additionally in some cases the information that is reported by Option Templates, such as the sampling rate, could be lost. It depends on the implementation of the IPFIX Concentrator. It should be noted that the information communicated by IPFIX Concentrator needs to be defined. Kobayashi, et al. Expires April 18, 2008 [Page 7] Internet-Draft Problem Statement in large NW October 2007 3. Conclusion This document has covered the following problems related to flow- based measurement systems in large-scale networks. o Adjusting the sampling rate and granularity of Flow causes small Flows to become invisible. Large-scale networks need another solution that can handle the huge amounts of Flow Records when using current sampling rates, which range from 1/10,000 to 1/1000. o Using IPFIX Concentrators might lose data that should be communicated by the original Exporter. o In Integrated Networks, which mix MPLS VPN with other tunnel traffic and IPv4/IPv6, flow-based measurement might not work well. These problems have their roots in constructing scalable flow-based measurement systems. Although network providers can resolve these problems by utilizing advanced features of Exporters and Collectors, it is difficult to perform an atomic migration of the whole measurement system while the amount of traffic grows. To assist the ability of the Exporters and the Collectors, it should be noted that there are some IPFIX devices for the network providers to select. They correspond to Flow mediation devices. Especially standardized IPFIX Concentrator could have been useful devices. Kobayashi, et al. Expires April 18, 2008 [Page 8] Internet-Draft Problem Statement in large NW October 2007 4. Security Considerations A flow-based measurement system might lead to privacy violation problems, such as the export of flows to an unexpected address, if the system is not confined to the large-scale network under observation. General security issues of the IPFIX protocol are covered by the security considerations section in [I-D.ietf-ipfix-protocol]. Security MUST be considered, if different networks exchange the flow information. As the security of the exchange relies mostly on the protocol used, UDP does not look appropriate to exchange information between networks. Kobayashi, et al. Expires April 18, 2008 [Page 9] Internet-Draft Problem Statement in large NW October 2007 5. IANA Considerations This document has no actions for IANA. Kobayashi, et al. Expires April 18, 2008 [Page 10] Internet-Draft Problem Statement in large NW October 2007 6. References 6.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 6.2. Informative References [I-D.dressler-ipfix-aggregation] Dressler, F., Sommer, C., and G. Munz, "IPFIX Aggregation", draft-dressler-ipfix-aggregation-03.txt (work in progress) , June 2006. [I-D.ietf-ipfix-as] Zseby, T., Boschi, E., Brownlee, N., and B. Claise, "IPFIX Applicability", draft-dressler-ipfix-as-10.txt (work in progress) , June 2006. [I-D.ietf-ipfix-protocol] Claise, B., "IPFIX Protocol Specification", draft-ietf-ipfix-protocol-23 (work in progress) , October 2006. [RFC3917] Quittek, J., Zseby, T., Claise, B., and S. Zander, "Requirements for IP Flow Information Export(IPFIX)", October 2004. [TRAFGRW] Cho, K., Fukuda, K., Esaki, H., and A. Kato, "The Impact and Implications of the Growth in Residential User-to-User Traffic", SIGCOMM2006, pp. 207-218, Pisa, Italy, September 2006. , October 2006. Kobayashi, et al. Expires April 18, 2008 [Page 11] Internet-Draft Problem Statement in large NW October 2007 Authors' Addresses Atsushi Kobayashi NTT Information Sharing Platform Laboratories 3-9-11 Midori-cho Musashino-shi, Tokyo 180-8585 Japan Phone: +81-422-59-3978 Email: akoba@nttv6.net Haruhiko Nishida NTT Information Sharing Platform Laboratories 3-9-11 Midori-cho Musashino-shi, Tokyo 180-8585 Japan Phone: +81-422-59-3978 Email: nishida.haruhiko@lab.ntt.co.jp Christoph Sommer University of Erlangen-Nuremberg Department of Computer Science 7 Martensstr. 3 Erlangen 91058 Germany Phone: +49 9131 85-27993 Email: christoph.sommer@informatik.uni-erlangen.de URI: http://www7.informatik.uni-erlangen.de/~sommer/ Falko Dressler University of Erlangen-Nuremberg Department of Computer Science 7 Martensstr. 3 Erlangen 91058 Germany Phone: +49 9131 85-27914 Email: dressler@informatik.uni-erlangen.de URI: http://www7.informatik.uni-erlangen.de/~dressler/ Kobayashi, et al. Expires April 18, 2008 [Page 12] Internet-Draft Problem Statement in large NW October 2007 Emile Stephan France Telecom Division R&D 2 avenue Pierre Marzin Lannion F-22307 France Phone: +33 2 96 05 18 52 Email: emile.stephan@orange-ftgroup.com Kobayashi, et al. Expires April 18, 2008 [Page 13] Internet-Draft Problem Statement in large NW October 2007 Full Copyright Statement Copyright (C) The IETF Trust (2007). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Acknowledgment Funding for the RFC Editor function is provided by the IETF Administrative Support Activity (IASA). Kobayashi, et al. Expires April 18, 2008 [Page 14]