diff options
Diffstat (limited to 'doc/rfc/rfc8041.txt')
| -rw-r--r-- | doc/rfc/rfc8041.txt | 1683 |
1 files changed, 1683 insertions, 0 deletions
diff --git a/doc/rfc/rfc8041.txt b/doc/rfc/rfc8041.txt new file mode 100644 index 0000000..1895ca6 --- /dev/null +++ b/doc/rfc/rfc8041.txt @@ -0,0 +1,1683 @@ + + + + + + +Internet Engineering Task Force (IETF) O. Bonaventure +Request for Comments: 8041 UCLouvain +Category: Informational C. Paasch +ISSN: 2070-1721 Apple, Inc. + G. Detal + Tessares + January 2017 + + + Use Cases and Operational Experience with Multipath TCP + +Abstract + + This document discusses both use cases and operational experience + with Multipath TCP (MPTCP) in real networks. It lists several + prominent use cases where Multipath TCP has been considered and is + being used. It also gives insight to some heuristics and decisions + that have helped to realize these use cases and suggests possible + improvements. + +Status of This Memo + + This document is not an Internet Standards Track specification; it is + published for informational purposes. + + This document is a product of the Internet Engineering Task Force + (IETF). It represents the consensus of the IETF community. It has + received public review and has been approved for publication by the + Internet Engineering Steering Group (IESG). Not all documents + approved by the IESG are a candidate for any level of Internet + Standard; see Section 2 of RFC 7841. + + Information about the current status of this document, any errata, + and how to provide feedback on it may be obtained at + http://www.rfc-editor.org/info/rfc8041. + + + + + + + + + + + + + + + + +Bonaventure, et al. Informational [Page 1] + +RFC 8041 MPTCP Experience January 2017 + + +Copyright Notice + + Copyright (c) 2017 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (http://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the Simplified BSD License. + +Table of Contents + + 1. Introduction ....................................................3 + 2. Use Cases .......................................................4 + 2.1. Datacenters ................................................4 + 2.2. Cellular/WiFi Offload ......................................5 + 2.3. Multipath TCP Proxies ......................................8 + 3. Operational Experience ..........................................9 + 3.1. Middlebox Interference .....................................9 + 3.2. Congestion Control ........................................11 + 3.3. Subflow Management ........................................12 + 3.4. Implemented Subflow Managers ..............................13 + 3.5. Subflow Destination Port ..................................15 + 3.6. Closing Subflows ..........................................16 + 3.7. Packet Schedulers .........................................17 + 3.8. Segment Size Selection ....................................18 + 3.9. Interactions with the Domain Name System ..................19 + 3.10. Captive Portals ..........................................20 + 3.11. Stateless Webservers .....................................20 + 3.12. Load-Balanced Server Farms ...............................21 + 4. Security Considerations ........................................21 + 5. References .....................................................23 + 5.1. Normative References ......................................23 + 5.2. Informative References ....................................23 + Acknowledgements ..................................................30 + Authors' Addresses ................................................30 + + + + + + + + + + +Bonaventure, et al. Informational [Page 2] + +RFC 8041 MPTCP Experience January 2017 + + +1. Introduction + + Multipath TCP was specified in [RFC6824] and five independent + implementations have been developed. As of November 2016, Multipath + TCP has been or is being implemented on the following platforms: + + o Linux kernel [MultipathTCP-Linux] + + o Apple iOS and macOS + + o Citrix load balancers + + o FreeBSD [FreeBSD-MPTCP] + + o Oracle Solaris + + The first three implementations are known to interoperate. Three of + these implementations are open source (Linux kernel, FreeBSD and + Apple's iOS and macOS). Apple's implementation is widely deployed. + + Since the publication of [RFC6824] as an Experimental RFC, experience + has been gathered by various network researchers and users about the + operational issues that arise when Multipath TCP is used in today's + Internet. + + When the MPTCP working group was created, several use cases for + Multipath TCP were identified [RFC6182]. Since then, other use cases + have been proposed and some have been tested and even deployed. We + describe these use cases in Section 2. + + Section 3 focuses on the operational experience with Multipath TCP. + Most of this experience comes from the utilization of the Multipath + TCP implementation in the Linux kernel [MultipathTCP-Linux]. This + open-source implementation has been downloaded and implemented by + thousands of users all over the world. Many of these users have + provided direct or indirect feedback by writing documents (scientific + articles or blog messages) or posting to the mptcp-dev mailing list + (see https://listes-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev). This + Multipath TCP implementation is actively maintained and continuously + improved. It is used on various types of hosts, ranging from + smartphones or embedded routers to high-end servers. + + The Multipath TCP implementation in the Linux kernel is not, by far, + the most widespread deployment of Multipath TCP. Since September + 2013, Multipath TCP is also supported on smartphones and tablets + beginning with iOS7 [IETFJ]. There are likely hundreds of millions + of MPTCP-enabled devices. This Multipath TCP implementation is + + + + +Bonaventure, et al. Informational [Page 3] + +RFC 8041 MPTCP Experience January 2017 + + + currently only used to support the Siri voice recognition/control + application. Some lessons learned from this deployment are described + in [IETFJ]. + + Section 3 is organized as follows. Supporting the middleboxes was + one of the difficult issues in designing the Multipath TCP protocol. + We explain in Section 3.1 which types of middleboxes the Linux Kernel + implementation of Multipath TCP supports and how it reacts upon + encountering these. Section 3.2 summarizes the MPTCP-specific + congestion controls that have been implemented. Sections 3.3 to 3.7 + discuss heuristics and issues with respect to subflow management as + well as the scheduling across the subflows. Section 3.8 explains + some problems that occurred with subflows having different Maximum + Segment Size (MSS) values. Section 3.9 presents issues with respect + to content delivery networks and suggests a solution to this issue. + Finally, Section 3.10 documents an issue with captive portals where + MPTCP will behave suboptimally. + +2. Use Cases + + Multipath TCP has been tested in several use cases. There is already + an abundant amount of scientific literature on Multipath TCP + [MPTCPBIB]. Several of the papers published in the scientific + literature have identified possible improvements that are worth being + discussed here. + +2.1. Datacenters + + A first, although initially unexpected, documented use case for + Multipath TCP has been in datacenters [HotNets][SIGCOMM11]. Today's + datacenters are designed to provide several paths between single- + homed servers. The multiplicity of these paths comes from the + utilization of Equal-Cost Multipath (ECMP) and other load-balancing + techniques inside the datacenter. Most of the deployed load- + balancing techniques in datacenters rely on hashes computed over the + five tuple. Thus, all packets from the same TCP connection follow + the same path: so they are not reordered. The results in [HotNets] + demonstrate by simulations that Multipath TCP can achieve a better + utilization of the available network by using multiple subflows for + each Multipath TCP session. Although [RFC6182] assumes that at least + one of the communicating hosts has several IP addresses, [HotNets] + demonstrates that Multipath TCP is beneficial when both hosts are + single-homed. This idea is analyzed in more details in [SIGCOMM11], + where the Multipath TCP implementation in the Linux kernel is + modified to be able to use several subflows from the same IP address. + Measurements in a public datacenter show the quantitative benefits of + Multipath TCP [SIGCOMM11] in this environment. + + + + +Bonaventure, et al. Informational [Page 4] + +RFC 8041 MPTCP Experience January 2017 + + + Although ECMP is widely used inside datacenters, this is not the only + environment where there are different paths between a pair of hosts. + ECMP and other load-balancing techniques such as Link Aggregation + Groups (LAGs) are widely used in today's networks; having multiple + paths between a pair of single-homed hosts is becoming the norm + instead of the exception. Although these multiple paths often have + the same cost (from an IGP metrics viewpoint), they do not + necessarily have the same performance. For example, [IMC13c] reports + the results of a long measurement study showing that load-balanced + Internet paths between that same pair of hosts can have huge delay + differences. + +2.2. Cellular/WiFi Offload + + A second use case that has been explored by several network + researchers is the cellular/WiFi offload use case. Smartphones or + other mobile devices equipped with two wireless interfaces are a very + common use case for Multipath TCP. In September 2015, this is also + the largest deployment of MPTCP-enabled devices [IETFJ]. It has been + briefly discussed during IETF 88 [IETF88], but there is no published + paper or report that analyses this deployment. For this reason, we + only discuss published papers that have mainly used the Multipath TCP + implementation in the Linux kernel for their experiments. + + The performance of Multipath TCP in wireless networks was briefly + evaluated in [NSDI12]. One experiment analyzes the performance of + Multipath TCP on a client with two wireless interfaces. This + evaluation shows that when the receive window is large, Multipath TCP + can efficiently use the two available links. However, if the window + becomes smaller, then packets sent on a slow path can block the + transmission of packets on a faster path. In some cases, the + performance of Multipath TCP over two paths can become lower than the + performance of regular TCP over the best performing path. Two + heuristics, reinjection and penalization, are proposed in [NSDI12] to + solve this identified performance problem. These two heuristics have + since been used in the Multipath TCP implementation in the Linux + kernel. [CONEXT13] explored the problem in more detail and revealed + some other scenarios where Multipath TCP can have difficulties in + efficiently pooling the available paths. Improvements to the + Multipath TCP implementation in the Linux kernel are proposed in + [CONEXT13] to cope with some of these problems. + + The first experimental analysis of Multipath TCP in a public wireless + environment was presented in [Cellnet12]. These measurements explore + the ability of Multipath TCP to use two wireless networks (real WiFi + and 3G networks). Three modes of operation are compared. The first + mode of operation is the simultaneous use of the two wireless + networks. In this mode, Multipath TCP pools the available resources + + + +Bonaventure, et al. Informational [Page 5] + +RFC 8041 MPTCP Experience January 2017 + + + and uses both wireless interfaces. This mode provides fast handover + from WiFi to cellular or the opposite when the user moves. + Measurements presented in [CACM14] show that the handover from one + wireless network to another is not an abrupt process. When a host + moves, there are regions where the quality of one of the wireless + networks is weaker than the other, but the host considers this + wireless network to still be up. When a mobile host enters such + regions, its ability to send packets over another wireless network is + important to ensure a smooth handover. This is clearly illustrated + from the packet trace discussed in [CACM14]. + + Many cellular networks use volume-based pricing; users often prefer + to use unmetered WiFi networks when available instead of metered + cellular networks. [Cellnet12] implements support for the MP_PRIO + option to explore two other modes of operation. + + In the backup mode, Multipath TCP opens a TCP subflow over each + interface, but the cellular interface is configured in backup mode. + This implies that data flows only over the WiFi interface when both + interfaces are considered to be active. If the WiFi interface fails, + then the traffic switches quickly to the cellular interface, ensuring + a smooth handover from the user's viewpoint [Cellnet12]. The cost of + this approach is that the WiFi and cellular interfaces are likely to + remain active all the time since all subflows are established over + the two interfaces. + + The single-path mode is slightly different. This mode benefits from + the break-before-make capability of Multipath TCP. When an MPTCP + session is established, a subflow is created over the WiFi interface. + No packet is sent over the cellular interface as long as the WiFi + interface remains up [Cellnet12]. This implies that the cellular + interface can remain idle and battery capacity is preserved. When + the WiFi interface fails, a new subflow is established over the + cellular interface in order to preserve the established Multipath TCP + sessions. Compared to the backup mode described earlier, + measurements reported in [Cellnet12] indicate that this mode of + operation is characterized by a throughput drop while the cellular + interface is brought up and the subflows are reestablished. + + From a protocol viewpoint, [Cellnet12] discusses the problem posed by + the unreliability of the REMOVE_ADDR option and proposes a small + protocol extension to allow hosts to reliably exchange this option. + It would be useful to analyze packet traces to understand whether the + unreliability of the REMOVE_ADDR option poses an operational problem + in real deployments. + + + + + + +Bonaventure, et al. Informational [Page 6] + +RFC 8041 MPTCP Experience January 2017 + + + Another study of the performance of Multipath TCP in wireless + networks was reported in [IMC13b]. This study uses laptops connected + to various cellular ISPs and WiFi hotspots. It compares various file + transfer scenarios. [IMC13b] observes that 4-path MPTCP outperforms + 2-path MPTCP, especially for larger files. However, for three + congestion-control algorithms (LIA, OLIA, and Reno -- see + Section 3.2), there is no significant performance difference for file + sizes smaller than 4 MB. + + A different study of the performance of Multipath TCP with two + wireless networks is presented in [INFOCOM14]. In this study the two + networks had different qualities: a good network and a lossy network. + When using two paths with different packet-loss ratios, the Multipath + TCP congestion-control scheme moves traffic away from the lossy link + that is considered to be congested. However, [INFOCOM14] documents + an interesting scenario that is summarized hereafter. + + client ----------- path1 -------- server + | | + +--------------- path2 ------------+ + + Figure 1: Simple network topology + + Initially, the two paths in Figure 1 have the same quality and + Multipath TCP distributes the load over both of them. During the + transfer, the path2 becomes lossy, e.g., because the client moves. + Multipath TCP detects the packet losses and they are retransmitted + over path1. This enables the data transfer to continue over this + path. However, the subflow over path2 is still up and transmits one + packet from time to time. Although the N packets have been + acknowledged over the first subflow (at the MPTCP level), they have + not been acknowledged at the TCP level over the second subflow. To + preserve the continuity of the sequence numbers over the second + subflow, TCP will continue to retransmit these segments until either + they are acknowledged or the maximum number of retransmissions is + reached. This behavior is clearly inefficient and may lead to + blocking since the second subflow will consume window space to be + able to retransmit these packets. [INFOCOM14] proposes a new + Multipath TCP option to solve this problem. In practice, a new TCP + option is probably not required. When the client detects that the + data transmitted over the second subflow has been acknowledged over + the first subflow, it could decide to terminate the second subflow by + sending a RST segment. If the interface associated to this subflow + is still up, a new subflow could be immediately reestablished. It + would then be immediately usable to send new data and would not be + forced to first retransmit the previously transmitted data. As of + this writing, this dynamic management of the subflows is not yet + implemented in the Multipath TCP implementation in the Linux kernel. + + + +Bonaventure, et al. Informational [Page 7] + +RFC 8041 MPTCP Experience January 2017 + + + Some studies have started to analyze the performance of Multipath TCP + on smartphones with real applications. In contrast with the bulk + transfers that are used by many publications, many deployed + applications do not exchange huge amounts of data and mainly use + small connections. [COMMAG2016] proposes a software testing + framework that allows to automate Android applications to study their + interactions with Multipath TCP. [PAM2016] analyses a one-month + packet trace of all the packets exchanged by a dozen of smartphones + utilized by regular users. This analysis reveals that short + connections are important on smartphones and that the main benefit of + using Multipath TCP on smartphones is the ability to perform seamless + handovers between different wireless networks. Long connections + benefit from these handovers. + +2.3. Multipath TCP Proxies + + As Multipath TCP is not yet widely deployed on both clients and + servers, several deployments have used various forms of proxies. Two + families of solutions are currently being used or tested. + + A first use case is when an MPTCP-enabled client wants to use several + interfaces to reach a regular TCP server. A typical use case is a + smartphone that needs to use both its WiFi and its cellular interface + to transfer data. Several types of proxies are possible for this use + case. An HTTP proxy deployed on a MPTCP-capable server would enable + the smartphone to use Multipath TCP to access regular web servers. + Obviously, this solution only works for applications that rely on + HTTP. Another possibility is to use a proxy that can convert any + Multipath TCP connection into a regular TCP connection. MPTCP- + specific proxies have been proposed [HotMiddlebox13b] [HAMPEL]. + + Another possibility leverages the SOCKS protocol [RFC1928]. SOCKS is + often used in enterprise networks to allow clients to reach external + servers. For this, the client opens a TCP connection to the SOCKS + server that relays it to the final destination. If both the client + and the SOCKS server use Multipath TCP, but not the final + destination, then Multipath TCP can still be used on the path between + the clients and the SOCKS server. At IETF 93, Korea Telecom + announced that they have deployed (in June 2015) a commercial service + that uses Multipath TCP on smartphones. These smartphones access + regular TCP servers through a SOCKS proxy. This enables them to + achieve throughputs of up to 850 Mbps [KT]. + + + + + + + + + +Bonaventure, et al. Informational [Page 8] + +RFC 8041 MPTCP Experience January 2017 + + + Measurements performed with Android smartphones [Mobicom15] show that + popular applications work correctly through a SOCKS proxy and MPTCP- + enabled smartphones. Thanks to Multipath TCP, long-lived connections + can be spread over the two available interfaces. However, for short- + lived connections, most of the data is sent over the initial subflow + that is created over the interface corresponding to the default route + and the second subflow is almost not used [PAM2016]. + + A second use case is when Multipath TCP is used by middleboxes, + typically inside access networks. Various network operators are + discussing and evaluating solutions for hybrid access networks + [TR-348]. Such networks arise when a network operator controls two + different access network technologies, e.g., wired and cellular, and + wants to combine them to improve the bandwidth offered to the end + users [HYA-ARCH]. Several solutions are currently investigated for + such networks [TR-348]. Figure 2 shows the organization of such a + network. When a client creates a normal TCP connection, it is + intercepted by the Hybrid CPE (HPCE) that converts it in a Multipath + TCP connection so that it can use the available access networks (DSL + and LTE in the example). The Hybrid Access Gateway (HAG) does the + opposite to ensure that the regular server sees a normal TCP + connection. Some of the solutions currently discussed for hybrid + networks use Multipath TCP on the HCPE and the HAG. Other solutions + rely on tunnels between the HCPE and the HAG [GRE-NOTIFY]. + + client --- HCPE ------ DSL ------- HAG --- internet --- server + | | + +------- LTE -----------+ + + Figure 2: Hybrid Access Network + +3. Operational Experience + +3.1. Middlebox Interference + + The interference caused by various types of middleboxes has been an + important concern during the design of the Multipath TCP protocol. + Three studies on the interactions between Multipath TCP and + middleboxes are worth discussing. + + + + + + + + + + + + +Bonaventure, et al. Informational [Page 9] + +RFC 8041 MPTCP Experience January 2017 + + + The first analysis appears in [IMC11]. This paper was the main + motivation for Multipath TCP incorporating various techniques to cope + with middlebox interference. More specifically, Multipath TCP has + been designed to cope with middleboxes that: + + o change source or destination addresses + + o change source or destination port numbers + + o change TCP sequence numbers + + o split or coalesce segments + + o remove TCP options + + o modify the payload of TCP segments + + These middlebox interferences have all been included in the MBtest + suite [MBTest]. This test suite is used in [HotMiddlebox13] to + verify the reaction of the Multipath TCP implementation in the Linux + kernel [MultipathTCP-Linux] when faced with middlebox interference. + The test environment used for this evaluation is a dual-homed client + connected to a single-homed server. The middlebox behavior can be + activated on any of the paths. The main results of this analysis + are: + + o the Multipath TCP implementation in the Linux kernel is not + affected by a middlebox that performs NAT or modifies TCP sequence + numbers + + o when a middlebox removes the MP_CAPABLE option from the initial + SYN segment, the Multipath TCP implementation in the Linux kernel + falls back correctly to regular TCP + + o when a middlebox removes the DSS option from all data segments, + the Multipath TCP implementation in the Linux kernel falls back + correctly to regular TCP + + o when a middlebox performs segment coalescing, the Multipath TCP + implementation in the Linux kernel is still able to accurately + extract the data corresponding to the indicated mapping + + o when a middlebox performs segment splitting, the Multipath TCP + implementation in the Linux kernel correctly reassembles the data + corresponding to the indicated mapping. [HotMiddlebox13] shows, + in Figure 4 in Section 3.3, a corner case with segment splitting + that may lead to a desynchronization between the two hosts. + + + + +Bonaventure, et al. Informational [Page 10] + +RFC 8041 MPTCP Experience January 2017 + + + The interactions between Multipath TCP and real deployed middleboxes + are also analyzed in [HotMiddlebox13]; a particular scenario with the + FTP Application Level Gateway running on a NAT is described. + + Middlebox interference can also be detected by analyzing packet + traces on MPTCP-enabled servers. A closer look at the packets + received on the multipath-tcp.org server [TMA2015] shows that among + the 184,000 Multipath TCP connections, only 125 of them were falling + back to regular TCP. These connections originated from 28 different + client IP addresses. These include 91 HTTP connections and 34 FTP + connections. The FTP interference is expected since Application + Level Gateways used for FTP modify the TCP payload and the DSS + Checksum detects these modifications. The HTTP interference appeared + only on the direction from server to client and could have been + caused by transparent proxies deployed in cellular or enterprise + networks. A longer trace is discussed in [COMCOM2016] and similar + conclusions about the middlebox interference are provided. + + From an operational viewpoint, knowing that Multipath TCP can cope + with various types of middlebox interference is important. However, + there are situations where the network operators need to gather + information about where a particular middlebox interference occurs. + The tracebox software [tracebox] described in [IMC13a] is an + extension of the popular traceroute software that enables network + operators to check at which hop a particular field of the TCP header + (including options) is modified. It has been used by several network + operators to debug various middlebox interference problems. + Experience with tracebox indicates that supporting the ICMP extension + defined in [RFC1812] makes it easier to debug middlebox problems in + IPv4 networks. + + Users of the Multipath TCP implementation have reported some + experience with middlebox interference. The strangest scenario has + been a middlebox that accepts the Multipath TCP options in the SYN + segment but later replaces Multipath TCP options with a TCP EOL + option [StrangeMbox]. This causes Multipath TCP to perform a + fallback to regular TCP without any impact on the application. + +3.2. Congestion Control + + Congestion control has been an important challenge for Multipath TCP. + The coupled congestion-control scheme defined in [RFC6356] in an + adaptation of the NewReno algorithm. A detailed description of this + coupled algorithm is provided in [NSDI11]. It is the default scheme + in the Linux implementation of Multipath TCP, but Linux supports + other schemes. + + + + + +Bonaventure, et al. Informational [Page 11] + +RFC 8041 MPTCP Experience January 2017 + + + The second congestion-control scheme is OLIA [CONEXT12]. It is also + an adaptation of the NewReno single path congestion-control scheme to + support multiple paths. Simulations [CONEXT12] and measurements + [CONEXT13] have shown that it provides some performance benefits + compared to the default coupled congestion-control scheme. + + The delay-based scheme proposed in [ICNP12] has also been ported to + the Multipath TCP implementation in the Linux kernel. It has been + evaluated by using simulations [ICNP12] and measurements [PaaschPhD]. + + BALIA, defined in [BALIA], provides a better balance between TCP + friendliness, responsiveness, and window oscillation. + + These different congestion-control schemes have been compared in + several articles. [CONEXT13] and [PaaschPhD] compare these + algorithms in an emulated environment. The evaluation showed that + the delay-based congestion-control scheme is less able to efficiently + use the available links than the three other schemes. + +3.3. Subflow Management + + The multipath capability of Multipath TCP comes from the utilization + of one subflow per path. The Multipath TCP architecture [RFC6182] + and the protocol specification [RFC6824] define the basic usage of + the subflows and the protocol mechanisms that are required to create + and terminate them. However, there are no guidelines on how subflows + are used during the lifetime of a Multipath TCP session. Most of the + published experiments with Multipath TCP have been performed in + controlled environments. Still, based on the experience running them + and discussions on the mptcp-dev mailing list, interesting lessons + have been learned about the management of these subflows. + + From a subflow viewpoint, the Multipath TCP protocol is completely + symmetrical. Both the clients and the server have the capability to + create subflows. However, in practice, the existing Multipath TCP + implementations have opted for a strategy where only the client + creates new subflows. The main motivation for this strategy is that + often the client resides behind a NAT or a firewall, preventing + passive subflow openings on the client. Although there are + environments such as datacenters where this problem does not occur, + as of this writing, no precise requirement has emerged for allowing + the server to create new subflows. + + + + + + + + + +Bonaventure, et al. Informational [Page 12] + +RFC 8041 MPTCP Experience January 2017 + + +3.4. Implemented Subflow Managers + + The Multipath TCP implementation in the Linux kernel includes several + strategies to manage the subflows that compose a Multipath TCP + session. The basic subflow manager is the full-mesh. As the name + implies, it creates a full-mesh of subflows between the communicating + hosts. + + The most frequent use case for this subflow manager is a multihomed + client connected to a single-homed server. In this case, one subflow + is created for each interface on the client. The current + implementation of the full-mesh subflow manager is static. The + subflows are created immediately after the creation of the initial + subflow. If one subflow fails during the lifetime of the Multipath + TCP session (e.g., due to excessive retransmissions or the loss of + the corresponding interface), it is not always reestablished. There + is ongoing work to enhance the full-mesh path manager to deal with + such events. + + When the server is multihomed, using the full-mesh subflow manager + may lead to a large number of subflows being established. For + example, consider a dual-homed client connected to a server with + three interfaces. In this case, even if the subflows are only + created by the client, six subflows will be established. This may be + excessive in some environments, in particular when the client and/or + the server have a large number of interfaces. Implementations should + limit the number of subflows that are used. + + Creating subflows between multihomed clients and servers may + sometimes lead to operational issues as observed by discussions on + the mptcp-dev mailing list. In some cases, the network operators + would like to have a better control on how the subflows are created + by Multipath TCP [MPTCP-MAX-SUB]. This might require the definition + of policy rules to control the operation of the subflow manager. The + two scenarios below illustrate some of these requirements. + + host1 ---------- switch1 ----- host2 + | | | + +-------------- switch2 --------+ + + Figure 3: Simple Switched Network Topology + + + + + + + + + + +Bonaventure, et al. Informational [Page 13] + +RFC 8041 MPTCP Experience January 2017 + + + Consider the simple network topology shown in Figure 3. From an + operational viewpoint, a network operator could want to create two + subflows between the communicating hosts. From a bandwidth + utilization viewpoint, the most natural paths are host1-switch1-host2 + and host1-switch2-host2. However, a Multipath TCP implementation + running on these two hosts may sometimes have difficulties to obtain + this result. + + To understand the difficulty, let us consider different allocation + strategies for the IP addresses. A first strategy is to assign two + subnets: subnetA (resp. subnetB) contains the IP addresses of host1's + interface to switch1 (resp. switch2) and host2's interface to switch1 + (resp. switch2). In this case, a Multipath TCP subflow manager + should only create one subflow per subnet. To enforce the + utilization of these paths, the network operator would have to + specify a policy that prefers the subflows in the same subnet over + subflows between addresses in different subnets. It should be noted + that the policy should probably also specify how the subflow manager + should react when an interface or subflow fails. + + A second strategy is to use a single subnet for all IP addresses. In + this case, it becomes more difficult to specify a policy that + indicates which subflows should be established. + + The second subflow manager that is currently supported by the + Multipath TCP implementation in the Linux kernel is the ndiffport + subflow manager. This manager was initially created to exploit the + path diversity that exists between single-homed hosts due to the + utilization of flow-based load-balancing techniques [SIGCOMM11]. + This subflow manager creates N subflows between the same pair of IP + addresses. The N subflows are created by the client and differ only + in the source port selected by the client. It was not designed to be + used on multihomed hosts. + + A more flexible subflow manager has been proposed, implemented and + evaluated in [CONEXT15]. This subflow manager exposes various kernel + events to a user space daemon that decides when subflows need to be + created and terminated based on various policies. + + + + + + + + + + + + + +Bonaventure, et al. Informational [Page 14] + +RFC 8041 MPTCP Experience January 2017 + + +3.5. Subflow Destination Port + + The Multipath TCP protocol relies on the token contained in the + MP_JOIN option to associate a subflow to an existing Multipath TCP + session. This implies that there is no restriction on the source + address, destination address and source or destination ports used for + the new subflow. The ability to use different source and destination + addresses is key to support multihomed servers and clients. The + ability to use different destination port numbers is worth discussing + because it has operational implications. + + For illustration, consider a dual-homed client that creates a second + subflow to reach a single-homed server as illustrated in Figure 4. + + client ------- r1 --- internet --- server + | | + +----------r2-------+ + + + Figure 4: Multihomed-Client Connected to Single-Homed Server + + When the Multipath TCP implementation in the Linux kernel creates the + second subflow, it uses the same destination port as the initial + subflow. This choice is motivated by the fact that the server might + be protected by a firewall and only accept TCP connections (including + subflows) on the official port number. Using the same destination + port for all subflows is also useful for operators that rely on the + port numbers to track application usage in their network. + + There have been suggestions from Multipath TCP users to modify the + implementation to allow the client to use different destination ports + to reach the server. This suggestion seems mainly motivated by + traffic-shaping middleboxes that are used in some wireless networks. + In networks where different shaping rates are associated with + different destination port numbers, this could allow Multipath TCP to + reach a higher performance. This behavior is valid according to the + Multipath TCP specification [RFC6824]. An application could use an + enhanced socket API [SOCKET] to behave in this way. + + However, from an implementation point-of-view supporting different + destination ports for the same Multipath TCP connection can cause + some issues. A legacy implementation of a TCP stack creates a + listening socket to react upon incoming SYN segments. The listening + socket is handling the SYN segments that are sent on a specific port + number. Demultiplexing incoming segments can thus be done solely by + looking at the IP addresses and the port numbers. With Multipath TCP + however, incoming SYN segments may have an MP_JOIN option with a + different destination port. This means that all incoming segments + + + +Bonaventure, et al. Informational [Page 15] + +RFC 8041 MPTCP Experience January 2017 + + + that did not match on an existing listening-socket or an already + established socket must be parsed for an eventual MP_JOIN option. + This imposes an additional cost on servers, previously not existent + on legacy TCP implementations. + +3.6. Closing Subflows + + client server + | | + MPTCP: ESTABLISHED | | MPTCP: ESTABLISHED + Sub: ESTABLISHED | | Sub: ESTABLISHED + | | + | DATA_FIN | + MPTCP: CLOSE-WAIT | <------------------------ | close() (step 1) + Sub: ESTABLISHED | DATA_ACK | + | ------------------------> | MPTCP: FIN-WAIT-2 + | | Sub: ESTABLISHED + | | + | DATA_FIN + subflow-FIN | + close()/shutdown() | ------------------------> | MPTCP: TIME-WAIT + (step 2) | DATA_ACK | Sub: CLOSE-WAIT + MPTCP: CLOSED | <------------------------ | + Sub: FIN-WAIT-2 | | + | | + | subflow-FIN | + MPTCP: CLOSED | <------------------------ | subflow-close() + Sub: TIME-WAIT | subflow-ACK | + (step 3) | ------------------------> | MPTCP: TIME-WAIT + | | Sub: CLOSED + | | + + + Figure 5: Multipath TCP may not be able to avoid time-wait state on + the subflow (indicated as Sub in the drawing), even if enforced by + the application on the client-side. + + Figure 5 shows a very particular issue within Multipath TCP. Many + high-performance applications try to avoid TIME-WAIT state by + deferring the closure of the connection until the peer has sent a + FIN. That way, the client on the left of Figure 5 does a passive + closure of the connection, transitioning from CLOSE-WAIT to Last-ACK + and finally freeing the resources after reception of the ACK of the + FIN. An application running on top of an MPTCP-enabled Linux kernel + might also use this approach. The difference here is that the + close() of the connection (step 1 in Figure 5) only triggers the + + + + + + +Bonaventure, et al. Informational [Page 16] + +RFC 8041 MPTCP Experience January 2017 + + + sending of a DATA_FIN. Nothing guarantees that the kernel is ready + to combine the DATA_FIN with a subflow-FIN. The reception of the + DATA_FIN will make the application trigger the closure of the + connection (step 2), trying to avoid TIME-WAIT state with this late + closure. This time, the kernel might decide to combine the DATA_FIN + with a subflow-FIN. This decision will be fatal, as the subflow's + state machine will not transition from CLOSE_WAIT to Last-ACK, but + rather go through FIN_WAIT-2 into TIME-WAIT state. The TIME-WAIT + state will consume resources on the host for at least 2 MSL (Maximum + Segment Lifetime). Thus, a smart application that tries to avoid + TIME-WAIT state by doing late closure of the connection actually ends + up with one of its subflows in TIME-WAIT state. A high-performance + Multipath TCP kernel implementation should honor the desire of the + application to do passive closure of the connection and successfully + avoid TIME-WAIT state -- even on the subflows. + + The solution to this problem lies in an optimistic assumption that a + host doing active-closure of a Multipath TCP connection by sending a + DATA_FIN will soon also send a FIN on all its subflows. Thus, the + passive closer of the connection can simply wait for the peer to send + exactly this FIN -- enforcing passive closure even on the subflows. + Of course, to avoid consuming resources indefinitely, a timer must + limit the time our implementation waits for the FIN. + +3.7. Packet Schedulers + + In a Multipath TCP implementation, the packet scheduler is the + algorithm that is executed when transmitting each packet to decide on + which subflow it needs to be transmitted. The packet scheduler + itself does not have any impact on the interoperability of Multipath + TCP implementations. However, it may clearly impact the performance + of Multipath TCP sessions. The Multipath TCP implementation in the + Linux kernel supports a pluggable architecture for the packet + scheduler [PaaschPhD]. As of this writing, two schedulers have been + implemented: round-robin and lowest-rtt-first. The second scheduler + relies on the round-trip time (rtt) measured on each TCP subflow and + sends first segments over the subflow having the lowest round-trip + time. They are compared in [CSWS14]. The experiments and + measurements described in [CSWS14] show that the lowest-rtt-first + scheduler appears to be the best compromise from a performance + viewpoint. Another study of the packet schedulers is presented in + [PAMS2014]. This study relies on simulations with the Multipath TCP + implementation in the Linux kernel. They compare the lowest-rtt- + first with the round-robin and a random scheduler. They show some + situations where the lowest-rtt-first scheduler does not perform as + well as the other schedulers, but there are many scenarios where the + + + + + +Bonaventure, et al. Informational [Page 17] + +RFC 8041 MPTCP Experience January 2017 + + + opposite is true. [PAMS2014] notes that "it is highly likely that + the optimal scheduling strategy depends on the characteristics of the + paths being used." + +3.8. Segment Size Selection + + When an application performs a write/send system call, the kernel + allocates a packet buffer (sk_buff in Linux) to store the data the + application wants to send. The kernel will store at most one MSS + (Maximum Segment Size) of data per buffer. As the MSS can differ + amongst subflows, an MPTCP implementation must select carefully the + MSS used to generate application data. The Linux kernel + implementation had various ways of selecting the MSS: minimum or + maximum amongst the different subflows. However, these heuristics of + MSS selection can cause significant performance issues in some + environments. Consider the following example. An MPTCP connection + has two established subflows that respectively use an MSS of 1420 and + 1428 bytes. If MPTCP selects the maximum, then the application will + generate segments of 1428 bytes of data. An MPTCP implementation + will have to split the segment in two (1420-byte and 8-byte) segments + when pushing on the subflow with the smallest MSS. The latter + segment will introduce a large overhead as this single data segment + will use 2 slots in the congestion window (in packets) therefore + reducing by roughly twice the potential throughput (in bytes/s) of + this subflow. Taking the smallest MSS does not solve the issue as + there might be a case where the subflow with the smallest MSS only + sends a few packets, therefore reducing the potential throughput of + the other subflows. + + The Linux implementation recently took another approach [DetalMSS]. + Instead of selecting the minimum and maximum values, it now + dynamically adapts the MSS based on the contribution of all the + subflows to the connection's throughput. For each subflow, it + computes the potential throughput achieved by selecting each MSS + value and by taking into account the lost space in the congestion + window. It then selects the MSS that allows to achieve the highest + potential throughput. + + Given the prevalence of middleboxes that clamp the MSS, Multipath TCP + implementations must be able to efficiently support subflows with + different MSS values. The strategy described above is a possible + solution to this problem. + + + + + + + + + +Bonaventure, et al. Informational [Page 18] + +RFC 8041 MPTCP Experience January 2017 + + +3.9. Interactions with the Domain Name System + + Multihomed clients such as smartphones can send DNS queries over any + of their interfaces. When a single-homed client performs a DNS + query, it receives from its local resolver the best answer for its + request. If the client is multihomed, the answer in response to the + DNS query may vary with the interface over which it has been sent. + + cdn1 + | + client -- cellular -- internet -- cdn3 + | | + +----- wifi --------+ + | + cdn2 + + + Figure 6: Simple Network Topology + + If the client sends a DNS query over the WiFi interface, the answer + will point to the cdn2 server while the same request sent over the + cellular interface will point to the cdn1 server. This might cause + problems for CDN providers that locate their servers inside ISP + networks and have contracts that specify that the CDN server will + only be accessed from within this particular ISP. Assume now that + both the client and the CDN servers support Multipath TCP. In this + case, a Multipath TCP session from cdn1 or cdn2 would potentially use + both the cellular network and the WiFi network. Serving the client + from cdn2 over the cellular interface could violate the contract + between the CDN provider and the network operators. A similar + problem occurs with regular TCP if the client caches DNS replies. + For example, the client obtains a DNS answer over the cellular + interface and then stops this interface and starts to use its WiFi + interface. If the client retrieves data from cdn1 over its WiFi + interface, this may also violate the contract between the CDN and the + network operators. + + A possible solution to prevent this problem would be to modify the + DNS resolution on the client. The client subnet Extension Mechanisms + for DNS (EDNS) defined in [RFC7871] could be used for this purpose. + When the client sends a DNS query from its WiFi interface, it should + also send the client subnet corresponding to the cellular interface + in this request. This would indicate to the resolver that the answer + should be valid for both the WiFi and the cellular interfaces (e.g., + the cdn3 server). + + + + + + +Bonaventure, et al. Informational [Page 19] + +RFC 8041 MPTCP Experience January 2017 + + +3.10. Captive Portals + + Multipath TCP enables a host to use different interfaces to reach a + server. In theory, this should ensure connectivity when at least one + of the interfaces is active. However, in practice, there are some + particular scenarios with captive portals that may cause operational + problems. The reference environment is shown in Figure 7. + + client ----- network1 + | + +------- internet ------------- server + + Figure 7: Issue with Captive Portal + + The client is attached to two networks: network1 that provides + limited connectivity and the entire Internet through the second + network interface. In practice, this scenario corresponds to an open + WiFi network with a captive portal for network1 and a cellular + service for the second interface. On many smartphones, the WiFi + interface is preferred over the cellular interface. If the + smartphone learns a default route via both interfaces, it will + typically prefer to use the WiFi interface to send its DNS request + and create the first subflow. This is not optimal with Multipath + TCP. A better approach would probably be to try a few attempts on + the WiFi interface and then, upon failure of these attempts, try to + use the second interface for the initial subflow as well. + +3.11. Stateless Webservers + + MPTCP has been designed to interoperate with webservers that benefit + from SYN-cookies to protect against SYN-flooding attacks [RFC4987]. + MPTCP achieves this by echoing the keys negotiated during the + MP_CAPABLE handshake in the third ACK of the three-way handshake. + Reception of this third ACK then allows the server to reconstruct the + state specific to MPTCP. + + However, one caveat to this mechanism is the unreliable nature of the + third ACK. Indeed, when the third ACK gets lost, the server will not + be able to reconstruct the MPTCP state. MPTCP will fall back to + regular TCP in this case. This is in contrast to regular TCP. When + the client starts sending data, the first data segment also includes + the SYN-cookie, which allows the server to reconstruct the TCP-state. + Further, this data segment will be retransmitted by the client in + case it gets lost and thus is resilient against loss. MPTCP does not + include the keys in this data segment and thus the server cannot + reconstruct the MPTCP state. + + + + + +Bonaventure, et al. Informational [Page 20] + +RFC 8041 MPTCP Experience January 2017 + + + This issue might be considered as a minor one for MPTCP. Losing the + third ACK should only happen when packet loss is high; in this case, + MPTCP provides a lot of benefits as it can move traffic away from the + lossy link. It is undesirable that MPTCP has a higher chance to fall + back to regular TCP in those lossy environments. + + [MPTCP-DEPLOY] discusses this issue and suggests a modified handshake + mechanism that ensures reliable delivery of the MP_CAPABLE, following + the three-way handshake. This modification will make MPTCP reliable, + even in lossy environments when servers need to use SYN-cookies to + protect against SYN-flooding attacks. + +3.12. Load-Balanced Server Farms + + Large-scale server farms typically deploy thousands of servers behind + a single virtual IP (VIP). Steering traffic to these servers is done + through Layer 4 load-balancers that ensure that a TCP-flow will + always be routed to the same server [Presto08]. + + As Multipath TCP uses multiple different TCP subflows to steer the + traffic across the different paths, load-balancers need to ensure + that all these subflows are routed to the same server. This implies + that the load-balancers need to track the MPTCP-related state, + allowing them to parse the token in the MP_JOIN and assign those + subflows to the appropriate server. However, server farms typically + deploy several load-balancers for reliability and capacity reasons. + As a TCP subflow might get routed to any of these load-balancers, + they would need to synchronize the MPTCP-related state -- a solution + that is not feasible on a large scale. + + The token (carried in the MP_JOIN) contains the information + indicating to which MPTCP-session the subflow belongs. As the token + is a hash of the key, servers are not able to generate the token in + such a way that the token can provide the necessary information to + the load-balancers, which would allow them to route TCP subflows to + the appropriate server. [MPTCP-LOAD] discusses this issue in detail + and suggests two alternative MP_CAPABLE handshakes to overcome these. + +4. Security Considerations + + This informational document discusses use cases and operational + experience with Multipath TCP. An extensive analysis of the + remaining security issues in the Multipath TCP specification has been + published in [RFC7430], together with suggestions for possible + solutions. + + + + + + +Bonaventure, et al. Informational [Page 21] + +RFC 8041 MPTCP Experience January 2017 + + + From a security viewpoint, it is important to note that Multipath + TCP, like other multipath solutions such as SCTP, has the ability to + send packets belonging to a single connection over different paths. + This design feature of Multipath TCP implies that middleboxes that + have been deployed on-path assuming that they would observe all the + packets exchanged for a given connection in both directions may not + function correctly anymore. A typical example are firewalls, + Intrusion Detection System (IDS) or deep packet inspections (DPIs) + deployed in enterprise networks. Those devices expect to observe all + the packets from all TCP connections. With Multipath TCP, those + middleboxes may not observe anymore all packets since some of them + may follow a different path. The two examples below illustrate + typical deployments of such middleboxes. The first example, + Figure 8, shows an MPTCP-enabled smartphone attached to both an + enterprise and a cellular network. If a Multipath TCP connection is + established by the smartphone towards a server, some of the packets + sent by the smartphone or the server may be transmitted over the + cellular network and thus be invisible for the enterprise middlebox. + + smartphone +----- enterprise net --- MBox----+------ server + | | + +----- cellular net -------------+ + + Figure 8: Enterprise Middlebox May Not Observe + All Packets from Multihomed Host + + The second example, Figure 9, shows a possible issue when multiple + middleboxes are deployed inside a network. For simplicity, we assume + that network1 is the default IPv4 path while network2 is the default + IPv6 path. A similar issue could occur with per-flow load-balancing + such as ECMP [RFC2992]. With regular TCP, all packets from each + connection would either pass through Mbox1 or Mbox2. With Multipath + TCP, the client can easily establish a subflow over network1 and + another over network2 and each middlebox would only observe a part of + the traffic of the end-to-end Multipath TCP connection. + + client ----R-- network1 --- MBox1 -----R------------- server + | | + +-- network2 --- MBox2 -----+ + + Figure 9: Interactions between + Load-Balancing and Security Middleboxes + + In these two cases, it is possible for an attacker to evade some + security measures operating on the TCP byte stream and implemented on + the middleboxes by controlling the bytes that are actually sent over + each subflow and there are tools that ease those kinds of evasion + [PZ15] [PT14]. This is not a security issue for Multipath TCP itself + + + +Bonaventure, et al. Informational [Page 22] + +RFC 8041 MPTCP Experience January 2017 + + + since Multipath TCP behaves correctly. However, this demonstrates + the difficulty of enforcing security policies by relying only on + on-path middleboxes instead of enforcing them directly on the + endpoints. + +5. References + +5.1. Normative References + + [RFC6182] Ford, A., Raiciu, C., Handley, M., Barre, S., and J. + Iyengar, "Architectural Guidelines for Multipath TCP + Development", RFC 6182, DOI 10.17487/RFC6182, March 2011, + <http://www.rfc-editor.org/info/rfc6182>. + + [RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, + "TCP Extensions for Multipath Operation with Multiple + Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013, + <http://www.rfc-editor.org/info/rfc6824>. + +5.2. Informative References + + [BALIA] Peng, Q., Walid, A., Hwang, J., and S. Low, "Multipath + TCP: analysis, design, and implementation", IEEE/ACM + Trans. on Networking (TON), Volume 24, Issue 1, February + 2016. + + [CACM14] Paasch, C. and O. Bonaventure, "Multipath TCP", + Communications of the ACM, 57(4):51-57, April 2014, + <http://inl.info.ucl.ac.be/publications/multipath-tcp>. + + [Cellnet12] + Paasch, C., Detal, G., Duchene, F., Raiciu, C., and O. + Bonaventure, "Exploring Mobile/WiFi Handover with + Multipath TCP", ACM SIGCOMM workshop on Cellular + Networks (Cellnet12), August 2012, + <http://inl.info.ucl.ac.be/publications/ + exploring-mobilewifi-handover-multipath-tcp>. + + [COMCOM2016] + Tran, V., De Coninck, Q., Hesmans, B., Sadre, R., and O. + Bonaventure, "Observing real Multipath TCP traffic", + Computer Communications, DOI 10.1016/j.comcom.2016.01.014, + April 2016, <http://inl.info.ucl.ac.be/publications/ + observing-real-multipath-tcp-traffic>. + + + + + + + +Bonaventure, et al. Informational [Page 23] + +RFC 8041 MPTCP Experience January 2017 + + + [COMMAG2016] + De Coninck, Q., Baerts, M., Hesmans, B., and O. + Bonaventure, "Observing Real Smartphone Applications over + Multipath TCP", IEEE Communications Magazine Network + Testing Series, 54(3), March 2016, + <http://inl.info.ucl.ac.be/publications/observing-real- + smartphone-applications-over-multipath-tcp>. + + [CONEXT12] Khalili, R., Gast, N., Popovic, M., Upadhyay, U., and J. + Leboudec, "MPTCP is not Pareto-Optimal: Performance Issues + and a Possible Solution", CoNEXT '12: Proceedings of the + 8th international conference on Emerging networking + experiments and technologies, DOI 10.1145/2413176.2413178, + December 2012. + + [CONEXT13] Paasch, C., Khalili, R., and O. Bonaventure, "On the + Benefits of Applying Experimental Design to Improve + Multipath TCP", Conference on emerging Networking + EXperiments and Technologies (CoNEXT), + DOI 10.1145/2535372.2535403, December 2013, + <http://inl.info.ucl.ac.be/publications/benefits-applying- + experimental-design-improve-multipath-tcp>. + + [CONEXT15] Hesmans, B., Detal, G., Barre, S., Bauduin, R., and O. + Bonaventure, "SMAPP: Towards Smart Multipath TCP-enabled + APPlications", Proc. Conext 2015, Heidelberg, Germany, + December 2015, <http://inl.info.ucl.ac.be/publications/ + smapp-towards-smart-multipath-tcp-enabled-applications>. + + [CSWS14] Paasch, C., Ferlin, S., Alay, O., and O. Bonaventure, + "Experimental evaluation of multipath TCP schedulers", + CSWS '14: Proceedings of the 2014 ACM SIGCOMM workshop on + Capacity sharing workshop, DOI 10.1145/2630088.2631977, + August 2014. + + [DetalMSS] Detal, G., "dynamically adapt mss value", Post on the + mptcp-dev mailing list, September 2014, + <https://listes-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev/ + 2014-09/msg00130.html>. + + [FreeBSD-MPTCP] + Williams, N., "Multipath TCP For FreeBSD Kernel Patch + v0.5", <http://caia.swin.edu.au/urp/newtcp/mptcp>. + + + + + + + + +Bonaventure, et al. Informational [Page 24] + +RFC 8041 MPTCP Experience January 2017 + + + [GRE-NOTIFY] + Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M. + Zhang, "GRE Notifications for Hybrid Access", Work in + Progress, draft-lhwxz-gre-notifications-hybrid-access-01, + January 2015. + + [HAMPEL] Hampel, G., Rana, A., and T. Klein, "Seamless TCP mobility + using lightweight MPTCP proxy", MobiWac '13: Proceedings + of the 11th ACM international symposium on Mobility + management and wireless access, + DOI 10.1145/2508222.2508226, November 2013. + + [HotMiddlebox13] + Hesmans, B., Duchene, F., Paasch, C., Detal, G., and O. + Bonaventure, "Are TCP Extensions Middlebox-proof?", CoNEXT + workshop Hot Middlebox, December 2013, + <http://inl.info.ucl.ac.be/publications/ + are-tcp-extensions-middlebox-proof>. + + [HotMiddlebox13b] + Detal, G., Paasch, C., and O. Bonaventure, "Multipath in + the Middle(Box)", HotMiddlebox '13, December 2013, + <http://inl.info.ucl.ac.be/publications/ + multipath-middlebox>. + + [HotNets] Raiciu, C., Pluntke, C., Barre, S., Greenhalgh, A., + Wischik, D., and M. Handley, "Data center networking with + multipath TCP", Hotnetx-IX: Proceedings of the 9th ACM + SIGCOMM Workshop on Hot Topics in Networks Article No. 10, + DOI 10.1145/1868447.1868457, October 2010, + <http://doi.acm.org/10.1145/1868447.1868457>. + + [HYA-ARCH] Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M. + Zhang, "Hybrid Access Network Architecture", Work in + Progress, draft-lhwxz-hybrid-access-network- + architecture-02, January 2015. + + [ICNP12] Cao, Y., Xu, M., and X. Fu, "Delay-based congestion + control for multipath TCP", 20th IEEE International + Conference on Network Protocols (INCP), + DOI 10.1109/ICNP.2012.6459978, October 2012. + + [IETF88] Stewart, L., "IETF 88 Meeting minutes of the MPTCP working + group", November 2013, <https://www.ietf.org/proceedings/ + 88/minutes/minutes-88-mptcp>. + + + + + + +Bonaventure, et al. Informational [Page 25] + +RFC 8041 MPTCP Experience January 2017 + + + [IETFJ] Bonaventure, O. and S. Seo, "Multipath TCP Deployments", + IETF Journal, Vol. 12, Issue 2, November 2016. + + [IMC11] Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A., + Handley, M., and H. Tokuda, "Is it still possible to + extend TCP?", IMC '11: Proceedings of the 2011 ACM SIGCOMM + conference on Internet measurement conference, + DOI 10.1145/2068816.2068834, November 2011, + <http://doi.acm.org/10.1145/2068816.2068834>. + + [IMC13a] Detal, G., Hesmans, B., Bonaventure, O., Vanaubel, Y., and + B. Donnet, "Revealing Middlebox Interference with + Tracebox", Proceedings of the 2013 ACM SIGCOMM conference + on Internet measurement conference, + DOI 10.1145/2504730.2504757, October 2013, + <http://inl.info.ucl.ac.be/publications/ + revealing-middlebox-interference-tracebox>. + + [IMC13b] Chen, Y., Lim, Y., Gibbens, R., Nahum, E., Khalili, R., + and D. Towsley, "A measurement-based study of MultiPath + TCP performance over wireless network", ICM '13: + Proceedings of the 2013 conference on Internet + measurement conference, DOI 10.1145/2504730.2504751, + October 2013, + <http://doi.acm.org/10.1145/2504730.2504751>. + + [IMC13c] Pelsser, C., Cittadini, L., Vissicchio, S., and R. Bush, + "From Paris to Tokyo: on the suitability of ping to + measure latency", IMC '13: Proceedings of the 2013 + conference on Internet measurement Conference, + DOI 10.1145/2504730.2504765, October 2013, + <http://doi.acm.org/10.1145/2504730.2504765>. + + [INFOCOM14] + Lim, Y., Chen, Y., Nahum, E., Towsley, D., and K. Lee, + "Cross-layer path management in multi-path transport + protocol for mobile devices", IEEE INFOCOM'14, + DOI 10.1109/INFOCOM.2014.6848120, April 2014. + + [KT] Seo, S., "KT's GiGA LTE", July 2015, + <https://www.ietf.org/proceedings/93/slides/ + slides-93-mptcp-3.pdf>. + + [MBTest] Hesmans, B., "MBTest", October 2013, + <https://bitbucket.org/bhesmans/mbtest>. + + + + + + +Bonaventure, et al. Informational [Page 26] + +RFC 8041 MPTCP Experience January 2017 + + + [Mobicom15] + De Coninck, Q., Baerts, M., Hesmans, B., and O. + Bonaventure, "Poster - Evaluating Android Applications + with Multipath TCP", Mobicom 2015 (Poster), + DOI 10.1145/2789168.2795165, September 2015. + + [MPTCP-DEPLOY] + Paasch, C., Biswas, A., and D. Haas, "Making Multipath TCP + robust for stateless webservers", Work in Progress, + draft-paasch-mptcp-syncookies-02, October 2015. + + [MPTCP-LOAD] + Paasch, C., Greenway, G., and A. Ford, "Multipath TCP + behind Layer-4 loadbalancers", Work in Progress, + draft-paasch-mptcp-loadbalancer-00, September 2015. + + [MPTCP-MAX-SUB] + Boucadair, M. and C. Jacquenet, "Negotiating the Maximum + Number of Multipath TCP (MPTCP) Subflows", Work in + Progress draft-boucadair-mptcp-max-subflow-02, May 2016. + + [MPTCPBIB] Bonaventure, O., "Multipath TCP - Annotated bibliography", + Technical report, April 2015, + <https://github.com/obonaventure/mptcp-bib>. + + [MultipathTCP-Linux] + Paasch, C., Barre, S., and . et al, "Multipath TCP - Linux + Kernel implementation", <http://www.multipath-tcp.org>. + + [NSDI11] Wischik, D., Raiciu, C., Greenhalgh, A., and M. Handley, + "Design, implementation and evaluation of congestion + control for multipath TCP", NSDI11: In Proceedings of the + 8th USENIX conference on Networked systems design + and implementation, 2011. + + [NSDI12] Raiciu, C., Paasch, C., Barre, S., Ford, A., Honda, M., + Duchene, F., Bonaventure, O., and M. Handley, "How Hard + Can It Be? Designing and Implementing a Deployable + Multipath TCP", NSDI '12: USENIX Symposium of Networked + Systems Design and implementation, April 2012, + <http://inl.info.ucl.ac.be/publications/how-hard-can-it- + be-designing-and-implementing-deployable-multipath-tcp>. + + [PaaschPhD] + Paasch, C., "Improving Multipath TCP", Ph.D. Thesis , + November 2014, <http://inl.info.ucl.ac.be/publications/ + improving-multipath-tcp>. + + + + +Bonaventure, et al. Informational [Page 27] + +RFC 8041 MPTCP Experience January 2017 + + + [PAM2016] De Coninck, Q., Baerts, M., Hesmans, B., and O. + Bonaventure, "A First Analysis of Multipath TCP on + Smartphones", 17th International Passive and Active + Measurements Conference (PAM2016) volume 17, March 2016, + <http://inl.info.ucl.ac.be/publications/ + first-analysis-multipath-tcp-smartphones>. + + [PAMS2014] Arzani, B., Gurney, A., Cheng, S., Guerin, R., and B. Loo, + "Impact of Path Selection and Scheduling Policies on MPTCP + Performance", PAMS2014, DOI 10.1109/WAINA.2014.121, May + 2014. + + [Presto08] Greenberg, A., Lahiri, P., Maltz, D., Patel, P., and S. + Sengupta, "Towards a next generation data center + architecture: scalability and commoditization", ACM + PRESTO 2008, DOI 10.1145/1397718.1397732, August 2008, + <http://dl.acm.org/citation.cfm?id=1397732>. + + [PT14] Pearce, C. and P. Thomas, "Multipath TCP Breaking Today's + Networks with Tomorrow's Protocols", Proc. + Blackhat Briefings, 2014, <http://www.blackhat.com/docs/ + us-14/materials/us-14-Pearce-Multipath-TCP-Breaking- + Todays-Networks-With-Tomorrows-Protocols-WP.pdf>. + + [PZ15] Pearce, C. and S. Zeadally, "Ancillary Impacts of + Multipath TCP on Current and Future Network Security", + IEEE Internet Computing, vol. 19, no. 5, pp. 58-65, + DOI 10.1109/MIC.2015.70, September 2015. + + [RFC1812] Baker, F., Ed., "Requirements for IP Version 4 Routers", + RFC 1812, DOI 10.17487/RFC1812, June 1995, + <http://www.rfc-editor.org/info/rfc1812>. + + [RFC1928] Leech, M., Ganis, M., Lee, Y., Kuris, R., Koblas, D., and + L. Jones, "SOCKS Protocol Version 5", RFC 1928, + DOI 10.17487/RFC1928, March 1996, + <http://www.rfc-editor.org/info/rfc1928>. + + [RFC2992] Hopps, C., "Analysis of an Equal-Cost Multi-Path + Algorithm", RFC 2992, DOI 10.17487/RFC2992, November 2000, + <http://www.rfc-editor.org/info/rfc2992>. + + [RFC4987] Eddy, W., "TCP SYN Flooding Attacks and Common + Mitigations", RFC 4987, DOI 10.17487/RFC4987, August 2007, + <http://www.rfc-editor.org/info/rfc4987>. + + + + + + +Bonaventure, et al. Informational [Page 28] + +RFC 8041 MPTCP Experience January 2017 + + + [RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled + Congestion Control for Multipath Transport Protocols", + RFC 6356, DOI 10.17487/RFC6356, October 2011, + <http://www.rfc-editor.org/info/rfc6356>. + + [RFC7430] Bagnulo, M., Paasch, C., Gont, F., Bonaventure, O., and C. + Raiciu, "Analysis of Residual Threats and Possible Fixes + for Multipath TCP (MPTCP)", RFC 7430, + DOI 10.17487/RFC7430, July 2015, + <http://www.rfc-editor.org/info/rfc7430>. + + [RFC7871] Contavalli, C., van der Gaast, W., Lawrence, D., and W. + Kumari, "Client Subnet in DNS Queries", RFC 7871, + DOI 10.17487/RFC7871, May 2016, + <http://www.rfc-editor.org/info/rfc7871>. + + [SIGCOMM11] + Raiciu, C., Barre, S., Pluntke, C., Greenhalgh, A., + Wischik, D., and M. Handley, "Improving datacenter + performance and robustness with multipath TCP", SIGCOMM + '11: Proceedings of the ACM SIGCOMM 2011 conference, + DOI 10.1145/2018436.2018467, August 2011, + <http://doi.acm.org/10.1145/2018436.2018467>. + + [SOCKET] Hesmans, B. and O. Bonaventure, "An enhanced socket API + for Multipath TCP", Proceedings of the 2016 Applied + Networking Research Workshop, DOI 10.1145/2959424.2959433, + July 2016, <http://doi.acm.org/10.1145/2959424.2959433>. + + [StrangeMbox] + Bonaventure, O., "Multipath TCP through a strange + middlebox", Blog post, January 2015, + <http://blog.multipath-tcp.org/blog/html/2015/01/30/ + multipath_tcp_through_a_strange_middlebox.html>. + + [TMA2015] Hesmans, B., Tran Viet, H., Sadre, R., and O. Bonaventure, + "A First Look at Real Multipath TCP Traffic", Traffic + Monitoring and Analysis, 2015, + <http://inl.info.ucl.ac.be/publications/ + first-look-real-multipath-tcp-traffic>. + + [TR-348] Broadband Forum, ., "TR 348 - Hybrid Access Broadband + Network Architecture", Issue: 1, July 2016, + <https://www.broadband-forum.org/technical/download/ + TR-348.pdf>. + + + + + + +Bonaventure, et al. Informational [Page 29] + +RFC 8041 MPTCP Experience January 2017 + + + [tracebox] Detal, G. and O. Tilmans, "Tracebox: A Middlebox Detection + Tool", 2013, <http://www.tracebox.org>. + +Acknowledgements + + This work was partially supported by the FP7-Trilogy2 project. We + would like to thank all the implementers and users of the Multipath + TCP implementation in the Linux kernel. This document has benefited + from the comments of John Ronan, Yoshifumi Nishida, Phil Eardley, + Jaehyun Hwang, Mirja Kuehlewind, Benoit Claise, Jari Arkko, Qin Wu, + Spencer Dawkins, and Ben Campbell. + +Authors' Addresses + + Olivier Bonaventure + UCLouvain + + Email: Olivier.Bonaventure@uclouvain.be + + + Christoph Paasch + Apple, Inc. + + Email: cpaasch@apple.com + + + Gregory Detal + Tessares + + Email: Gregory.Detal@tessares.net + + + + + + + + + + + + + + + + + + + + + +Bonaventure, et al. Informational [Page 30] + |