doc: Add RFC documents

1 files changed, 843 insertions, 0 deletions

diff --git a/doc/rfc/rfc5532.txt b/doc/rfc/rfc5532.txt
new file mode 100644
index 0000000..a38cec0
--- /dev/null
+++ b/doc/rfc/rfc5532.txt
@@ -0,0 +1,843 @@
+
+
+
+
+
+
+Network Working Group                                          T. Talpey
+Request for Comments: 5532                                   C. Juszczak
+Category: Informational                                         May 2009
+
+
+     Network File System (NFS) Remote Direct Memory Access (RDMA)
+                           Problem Statement
+
+Status of This Memo
+
+   This memo provides information for the Internet community.  It does
+   not specify an Internet standard of any kind.  Distribution of this
+   memo is unlimited.
+
+Copyright Notice
+
+   Copyright (c) 2009 IETF Trust and the persons identified as the
+   document authors.  All rights reserved.
+
+   This document is subject to BCP 78 and the IETF Trust's Legal
+   Provisions Relating to IETF Documents in effect on the date of
+   publication of this document (http://trustee.ietf.org/license-info).
+   Please review these documents carefully, as they describe your rights
+   and restrictions with respect to this document.
+
+   This document may contain material from IETF Documents or IETF
+   Contributions published or made publicly available before November
+   10, 2008.  The person(s) controlling the copyright in some of this
+   material may not have granted the IETF Trust the right to allow
+   modifications of such material outside the IETF Standards Process.
+   Without obtaining an adequate license from the person(s) controlling
+   the copyright in such materials, this document may not be modified
+   outside the IETF Standards Process, and derivative works of it may
+   not be created outside the IETF Standards Process, except to format
+   it for publication as an RFC or to translate it into languages other
+   than English.
+
+Abstract
+
+   This document addresses enabling the use of Remote Direct Memory
+   Access (RDMA) by the Network File System (NFS) protocols.  NFS
+   implementations historically incur significant overhead due to data
+   copies on end-host systems, as well as other processing overhead.
+   This document explores the potential benefits of RDMA to these
+   implementations and evaluates the reasons why RDMA is especially
+   well-suited to NFS and network file protocols in general.
+
+
+
+
+
+Talpey & Juszczak            Informational                      [Page 1]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+Table of Contents
+
+   1. Introduction ....................................................2
+      1.1. Background .................................................3
+   2. Problem Statement ...............................................4
+   3. File Protocol Architecture ......................................5
+   4. Sources of Overhead .............................................7
+      4.1. Savings from TOE ...........................................8
+      4.2. Savings from RDMA ..........................................9
+   5. Application of RDMA to NFS .....................................10
+   6. Conclusions ....................................................10
+   7. Security Considerations ........................................11
+   8. Acknowledgments ................................................12
+   9. References .....................................................12
+      9.1. Normative References ......................................12
+      9.2. Informative References ....................................13
+
+1.  Introduction
+
+   The Network File System (NFS) protocol (as described in [RFC1094],
+   [RFC1813], and [RFC3530]) is one of several remote file access
+   protocols used in the class of processing architecture sometimes
+   called Network-Attached Storage (NAS).
+
+   Historically, remote file access has proven to be a convenient,
+   cost-effective way to share information over a network, a concept
+   proven over time by the popularity of the NFS protocol.  However,
+   there are issues in such a deployment.
+
+   As compared to a local (direct-attached) file access architecture,
+   NFS removes the overhead of managing the local on-disk file system
+   state and its metadata, but interposes at least a transport network
+   and two network endpoints between an application process and the
+   files it is accessing.  To date, this trade-off has usually resulted
+   in a net performance loss as a result of reduced bandwidth, increased
+   application server CPU utilization, and other overheads.
+
+   Several classes of applications, including those directly supporting
+   enterprise activities in high-performance domains such as database
+   applications and shared clusters, have therefore encountered issues
+   with moving to NFS architectures.  While this has been due
+   principally to the performance costs of NFS versus direct-attached
+   files, other reasons are relevant, such as the lack of strong
+   consistency guarantees being provided by NFS implementations.
+
+   Replication of local file access performance on NAS using traditional
+   network protocol stacks has proven difficult, not because of protocol
+   processing overheads, but because of data copy costs in the network
+
+
+
+Talpey & Juszczak            Informational                      [Page 2]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+   endpoints.  This is especially true since host buses are now often
+   the main bottleneck in NAS architectures [MOG03] [CHA+01].
+
+   The External Data Representation [RFC4506] employed beneath NFS and
+   the Remote Procedure Call (RPC) [RFC5531] can add more data copies,
+   exacerbating the problem.
+
+   Data copy-avoidance designs have not been widely adopted for a
+   variety of reasons.  [BRU99] points out that "many copy avoidance
+   techniques for network I/O are not applicable or may even backfire if
+   applied to file I/O".  Other designs that eliminate unnecessary
+   copies, such as [PAI+00], are incompatible with existing APIs and
+   therefore force application changes.
+
+   In recent years, an effort to standardize a set of protocols for
+   Remote Direct Memory Access (RDMA) over the standard Internet
+   Protocol Suite has been chartered [RDDP].  A complete IP-based RDMA
+   protocol suite is available in the published Standards Track
+   specifications.
+
+   RDMA is a general solution to the problem of CPU overhead incurred
+   due to data copies, primarily at the receiver.  Substantial research
+   has addressed this and has borne out the efficacy of the approach.
+   An overview of this is the "Remote Direct Memory Access (RDMA) over
+   IP Problem Statement" [RFC4297].
+
+   In addition to the per-byte savings of offloading data copies, RDMA-
+   enabled NICs (RNICS) offload the underlying protocol layers as well
+   (e.g., TCP), further reducing CPU overhead due to NAS processing.
+
+1.1.  Background
+
+   The RDDP Problem Statement [RFC4297] asserts:
+
+      High costs associated with copying are an issue primarily for
+      large scale systems ... with high bandwidth feeds, usually
+      multiprocessors and clusters, that are adversely affected by
+      copying overhead.  Examples of such machines include all varieties
+      of servers: database servers, storage servers, application servers
+      for transaction processing, for e-commerce, and web serving,
+      content distribution, video distribution, backups, data mining and
+      decision support, and scientific computing.
+
+      Note that such servers almost exclusively service many concurrent
+      sessions (transport connections), which, in aggregate, are
+      responsible for > 1 Gbits/s of communication.  Nonetheless, the
+      cost of copying overhead for a particular load is the same whether
+      from few or many sessions.
+
+
+
+Talpey & Juszczak            Informational                      [Page 3]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+   Note that each of the servers listed above could be accessing their
+   file data as an NFS client, or as NFS serving the data to such
+   clients, or acting as both.
+
+   The CPU overhead of the NFS and TCP/IP protocol stacks (including
+   data copies or reduced copy workarounds) becomes a significant matter
+   in these clients and servers.  File access using locally attached
+   disks imposes relatively low overhead due to the highly optimized I/O
+   path and direct memory access afforded to the storage controller.
+   This is not the case with NFS, which must pass data to, and
+   especially from, the network and network processing stack to the NFS
+   stack.  Frequently, data copies are imposed on this transfer; in some
+   cases, several such copies are imposed in each direction.
+
+   Copies are potentially encountered in an NFS implementation
+   exchanging data to and from user address spaces, within kernel buffer
+   caches, in eXternal Data Representation (XDR) marshalling and
+   unmarshalling, and within network stacks and network drivers.  Other
+   overheads such as serialization among multiple threads of execution
+   sharing a single NFS mount point and transport connection are
+   additionally encountered.
+
+   Numerous upper-layer protocols achieve extremely high bandwidth and
+   low overhead through the use of RDMA.  [MAF+02] shows that the RDMA-
+   based Direct Access File System (with a user-level implementation of
+   the file system client) can outperform even a zero-copy
+   implementation of NFS [CHA+01] [CHA+99] [GAL+99] [KM02].  Also, file
+   data access implies the use of large Unequal Loss Protection (ULP)
+   messages.  These large messages tend to amortize any increase in
+   per-message costs due to the offload of protocol processing incurred
+   when using RNICs while gaining the benefits of reduced per-byte
+   costs.  Finally, the direct memory addressing afforded by RDMA avoids
+   many sources of contention on network resources.
+
+2.  Problem Statement
+
+   The principal performance problem encountered by NFS implementations
+   is the CPU overhead required to implement the protocol.  Primary
+   among the sources of this overhead is the movement of data from NFS
+   protocol messages to its eventual destination in user buffers or
+   aligned kernel buffers.  Due to the nature of the RPC and XDR
+   protocols, the NFS data payload arrives at arbitrary alignment,
+   necessitating a copy at the receiver, and the NFS requests are
+   completed in an arbitrary sequence.
+
+   The data copies consume system bus bandwidth and CPU time, reducing
+   the available system capacity for applications [RFC4297].  To date,
+   achieving zero-copy with NFS has required sophisticated, version-
+
+
+
+Talpey & Juszczak            Informational                      [Page 4]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+   specific "header cracking" hardware and/or extensive platform-
+   specific virtual memory mapping tricks.  Such approaches become even
+   more difficult for NFS version 4 due to the existence of the COMPOUND
+   operation and presence of Kerberos and other security information,
+   which further reduce alignment and greatly complicate ULP offload.
+
+   Furthermore, NFS is challenged by high-speed network fabrics such as
+   10 Gbits/s Ethernet.  Performing even raw network I/O such as TCP is
+   an issue at such speeds with today's hardware.  The problem is
+   fundamental in nature and has led the IETF to explore RDMA [RFC4297].
+
+   Zero-copy techniques benefit file protocols extensively, as they
+   enable direct user I/O, reduce the overhead of protocol stacks,
+   provide perfect alignment into caches, etc.  Many studies have
+   already shown the performance benefits of such techniques [SKE+01]
+   [DCK+03] [FJNFS] [FJDAFS] [KM02] [MAF+02].
+
+   RDMA is compelling here for another reason; hardware-offloaded
+   networking support in itself does not avoid data copies, without
+   resorting to implementing part of the NFS protocol in the Network
+   Interface Card (NIC).  Support of RDMA by NFS enables the highest
+   performance at the architecture level rather than by implementation;
+   this enables ubiquitous and interoperable solutions.
+
+   By providing file access performance equivalent to that of local file
+   systems, NFS over RDMA will enable applications running on a set of
+   client machines to interact through an NFS file system, just as
+   applications running on a single machine might interact through a
+   local file system.
+
+3.  File Protocol Architecture
+
+   NFS runs as an Open Network Computing (ONC) RPC [RFC5531]
+   application.  Being a file access protocol, NFS is very "rich" in
+   data content (versus control information).
+
+   NFS messages can range from very small (under 100 bytes) to very
+   large (from many kilobytes to a megabyte or more).  They are all
+   contained within an RPC message and follow a variable-length RPC
+   header.  This layout provides an alignment challenge for the data
+   items contained in an NFS call (request) or reply (response) message.
+
+   In addition to the control information in each NFS call or reply
+   message, sometimes there are large "chunks" of application file data,
+   for example, read and write requests.  With NFS version 4 (due to the
+   existence of the COMPOUND operation), there can be several of these
+   data chunks interspersed with control information.
+
+
+
+
+Talpey & Juszczak            Informational                      [Page 5]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+   ONC RPC is a remote procedure call protocol that has been run over a
+   variety of transports.  Most implementations today use UDP or TCP.
+   RPC messages are defined in terms of an eXternal Data Representation
+   (XDR) [RFC4506], which provides a canonical data representation
+   across a variety of host architectures.  An XDR data stream is
+   conveyed differently on each type of transport.  On UDP, RPC messages
+   are encapsulated inside datagrams, while on a TCP byte stream, RPC
+   messages are delineated by a record-marking protocol.  An RDMA
+   transport also conveys RPC messages in a unique fashion that must be
+   fully described if client and server implementations are to
+   interoperate.
+
+   The RPC transport is responsible for conveying an RPC message from a
+   sender to a receiver.  An RPC message is either an RPC call from a
+   client to a server, or an RPC reply from the server back to the
+   client.  An RPC message contains an RPC call header followed by
+   arguments if the message is an RPC call, or an RPC reply header
+   followed by results if the message is an RPC reply.  The call header
+   contains a transaction ID (XID) followed by the program and procedure
+   number as well as a security credential.  An RPC reply header begins
+   with an XID that matches that of the RPC call message, followed by a
+   security verifier and results.  All data in an RPC message is XDR
+   encoded.
+
+   The encoding of XDR data into transport buffers is referred to as
+   "marshalling", and the decoding of XDR data contained within
+   transport buffers and into destination RPC procedure result buffers,
+   is referred to as "unmarshalling".  Therefore, the process of
+   marshalling takes place at the sender of any particular message, be
+   it an RPC request or an RPC response.  Unmarshalling, of course,
+   takes place at the receiver.
+
+   Normally, any bulk data is moved (copied) as a result of the
+   unmarshalling process, because the destination address is not known
+   until the RPC code receives control and subsequently invokes the XDR
+   unmarshalling routine.  In other words, XDR-encoded data is not
+   self-describing, and it carries no placement information.  This
+   results in a data copy in most NFS implementations.
+
+   One mechanism by which the RPC layer may overcome this is for each
+   request to include placement information, to be used for direct
+   placement during XDR encode.  This "write chunk" can avoid sending
+   bulk data inline in an RPC message and generally results in one or
+   more RDMA Write operations.
+
+   Similarly, a "read chunk", where placement information referring to
+   bulk data that may be directly fetched via one or more RDMA Read
+   operations during XDR decode, may be conveyed.  The "read chunk" will
+
+
+
+Talpey & Juszczak            Informational                      [Page 6]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+   therefore be useful in both RPC calls and replies, while the "write
+   chunk" is used solely in replies.
+
+   These "chunks" are the key concept in an existing proposal [RPCRDMA].
+   They convey what are effectively pointers to remote memory across the
+   network.  They allow cooperating peers to exchange data outside of
+   XDR encodings but still use XDR for describing the data to be
+   transferred.  And, finally, through use of XDR they maintain a large
+   degree of on-the-wire compatibility.
+
+   The central concept of the RDMA transport is to provide the
+   additional encoding conventions to convey this placement information
+   in transport-specific encoding, and to modify the XDR handling of
+   bulk data.
+
+                           Block Diagram
+
+   +------------------------+-----------------------------------+
+   |         NFS            |            NFS + RDMA             |
+   +------------------------+----------------------+------------+
+   |           Operations / Procedures             |            |
+   +-----------------------------------------------+            |
+   |                   RPC/XDR                     |            |
+   +--------------------------------+--------------+            |
+   |       Stream Transport         |      RDMA Transport       |
+   +--------------------------------+---------------------------+
+
+4.  Sources of Overhead
+
+   Network and file protocol costs can be categorized as follows:
+
+   o  per-byte costs - data touching costs such as checksum or data
+      copy.  Today's network interface hardware commonly offloads the
+      checksum, which leaves the other major source of per-byte
+      overhead, data copy.
+
+   o  per-packet costs - interrupts and lower-layer processing (LLP).
+      Today's network interface hardware also commonly coalesce
+      interrupts to reduce per-packet costs.
+
+   o  per-message (request or response) costs - LLP and ULP processing.
+
+   Improvement from optimization becomes more important if the overhead
+   it targets is a larger share of the total cost.  As other sources of
+   overhead, such as the checksumming and interrupt handling above are
+   eliminated, the remaining overheads (primarily data copy) loom
+   larger.
+
+
+
+
+Talpey & Juszczak            Informational                      [Page 7]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+   With copies crossing the bus twice per copy, network processing
+   overhead is high whenever network bandwidth is large in comparison to
+   CPU and memory bandwidths.  Generally, with today's end-systems, the
+   effects are observable at network speeds at or above 1 Gbit/s.
+
+   A common question is whether an increase in CPU processing power
+   alleviates the problem of high processing costs of network I/O.  The
+   answer is no, it is the memory bandwidth that is the issue.  Faster
+   CPUs do not help if the CPU spends most of its time waiting for
+   memory [RFC4297].
+
+   TCP offload engine (TOE) technology aims to offload the CPU by moving
+   TCP/IP protocol processing to the NIC.  However, TOE technology by
+   itself does nothing to avoid necessary data copies within upper-layer
+   protocols.  [MOG03] provides a description of the role TOE can play
+   in reducing per-packet and per-message costs.  Beyond the offloads
+   commonly provided by today's network interface hardware, TOE alone
+   (without RDMA) helps in protocol header processing, but this has been
+   shown to be a minority component of the total protocol processing
+   overhead. [CHA+01]
+
+   Numerous software approaches to the optimization of network
+   throughput have been made.  Experience has shown that network I/O
+   interacts with other aspects of system processing such as file I/O
+   and disk I/O [BRU99] [CHU96].  Zero-copy optimizations based on page
+   remapping [CHU96] can be dependent upon machine architecture, and are
+   not scalable to multi-processor architectures.  Correct buffer
+   alignment and sizing together are needed to optimize the performance
+   of zero-copy movement mechanisms [SKE+01].  The NFS message layout
+   described above does not facilitate the splitting of headers from
+   data nor does it facilitate providing correct data buffer alignment.
+
+4.1.  Savings from TOE
+
+   The expected improvement of TOE specifically for NFS protocol
+   processing can be quantified and shown to be fundamentally limited.
+   [SHI+03] presents a set of "LAWS" parameters that serve to illustrate
+   the issues.  In the TOE case, the copy cost can be viewed as part of
+   the application processing "a".  Application processing increases the
+   LAWS "gamma", which is shown by the paper to result in a diminished
+   benefit for TOE.
+
+   For example, if the overhead is 20% TCP/IP, 30% copy, and 50% real
+   application work, then gamma is 80/20 or 4, which means the maximum
+   benefit of TOE is 1/gamma, or only 25%.
+
+   For RDMA (with embedded TOE) and the same example, the "overhead" (o)
+   offloaded or eliminated is 50% (20% + 30%).  Therefore, in the RDMA
+
+
+
+Talpey & Juszczak            Informational                      [Page 8]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+   case, gamma is 50/50 or 1, and the inverse gives the potential
+   benefit of 1 (100%), a factor of two.
+
+                    CPU Overhead Reduction Factor
+
+               No Offload   TCP Offload   RDMA Offload
+               -----------+-------------+-------------
+                  1.00x        1.25x         2.00x
+
+   The analysis in the paper shows that RDMA could improve throughput by
+   the same factor of two, even when the host is (just) powerful enough
+   to drive the full network bandwidth without RDMA.  It can also be
+   shown that the speedup may be higher if network bandwidth grows
+   faster than Moore's Law, although the higher benefits will apply to a
+   narrow range of applications.
+
+4.2.  Savings from RDMA
+
+   Performance measurements directly comparing an NFS-over-RDMA
+   prototype with conventional network-based NFS processing are
+   described in [CAL+03].  Comparisons of Read throughput and CPU
+   overhead were performed on two types of Gigabit Ethernet adapters,
+   one type being a conventional adapter, and another type with RDMA
+   capability.  The prototype RDMA protocol performed all transfers via
+   RDMA Read.  The NFS layer in the study was measured while performing
+   read transfers, varying the transfer size and readahead depth across
+   ranges used by typical NFS deployments.
+
+   In these results, conventional network-based throughput was severely
+   limited by the client's CPU being saturated at 100% for all
+   transfers.  Read throughput reached no more than 60 MBytes/s.
+
+      I/O Type        Size     Read Throughput     CPU Utilization
+      Conventional    2 KB         20 MB/s              100%
+      Conventional   16 KB         40 MB/s              100%
+      Conventional  256 KB         60 MB/s              100%
+
+   However, over RDMA, throughput rose to the theoretical maximum
+   throughput of the platform, while saturating the single-CPU system
+   only at maximum throughput.
+
+       I/O Type       Size     Read Throughput     CPU Utilization
+      RDMA            2 KB         10 MB/s               45%
+      RDMA           16 KB         40 MB/s               70%
+      RDMA          256 KB        100 MB/s              100%
+
+   The lower relative throughput of the RDMA prototype at the small
+   blocksize may be attributable to the RDMA Read imposed by the
+
+
+
+Talpey & Juszczak            Informational                      [Page 9]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+   prototype protocol, which reduced the operation rate since it
+   introduces additional latency.  As well, it may reflect the relative
+   increase of per-packet setup costs within the DMA portion of the
+   transfer.
+
+5.  Application of RDMA to NFS
+
+   Efficient file protocols require efficient data positioning and
+   movement.  The client system knows the client memory address where
+   the application has data to be written or wants read data deposited.
+   The server system knows the server memory address where the local
+   file system will accept write data or has data to be read.  Neither
+   peer however is aware of the others' data destination in the current
+   NFS, RPC, or XDR protocols.  Existing NFS implementations have
+   struggled with the performance costs of data copies when using
+   traditional Ethernet transports.
+
+   With the onset of faster networks, the network I/O bottleneck will
+   worsen.  Fortunately, new transports that support RDMA have emerged.
+   RDMA excels at bulk transfer efficiency; it is an efficient way to
+   deliver direct data placement and remove a major part of the problem:
+   data copies.  RDMA also addresses other overheads, e.g., underlying
+   protocol offload, and offers separation of control information from
+   data.
+
+   The current NFS message layout provides the performance-enhancing
+   opportunity for an NFS-over-RDMA protocol that separates the control
+   information from data chunks while meeting the alignment needs of
+   both.  The data chunks can be copied "directly" between the client
+   and server memory addresses above (with a single occurrence on each
+   memory bus) while the control information can be passed "inline".
+   [RPCRDMA] describes such a protocol.
+
+6.  Conclusions
+
+   NFS version 4 [RFC3530] has been granted "Proposed Standard" status.
+   The NFSv4 protocol was developed along several design points,
+   important among them: effective operation over wide-area networks,
+   including the Internet itself; strong security integrated into the
+   protocol; extensive cross-platform interoperability including
+   integrated locking semantics compatible with multiple operating
+   systems; and (this is key), protocol extension.
+
+   NFS version 4 is an excellent base on which to add the needed
+   performance enhancements and improved semantics described above.  The
+   minor versioning support defined in NFS version 4 was designed to
+   support protocol improvements without disruption to the installed
+   base [NFSv4.1].  Evolutionary improvement of the protocol via minor
+
+
+
+Talpey & Juszczak            Informational                     [Page 10]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+   versioning is a conservative and cautious approach to current and
+   future problems and shortcomings.
+
+   Many arguments can be made as to the efficacy of the file abstraction
+   in meeting the future needs of enterprise data service and the
+   Internet.  Fine grained Quality of Service (QoS) policies (e.g., data
+   delivery, retention, availability, security, etc.) are high among
+   them.
+
+   It is vital that the NFS protocol continue to provide these benefits
+   to a wide range of applications, without its usefulness being
+   compromised by concerns about performance and semantic inadequacies.
+   This can reasonably be addressed in the existing NFS protocol
+   framework.  A cautious evolutionary improvement of performance and
+   semantics allows building on the value already present in the NFS
+   protocol, while addressing new requirements that have arisen from the
+   application of networking technology.
+
+7.  Security Considerations
+
+   The NFS protocol, in conjunction with its layering on RPC, provides a
+   rich and widely interoperable security model to applications and
+   systems.  Any layering of NFS-over-RDMA transports must address the
+   NFS security requirements, and additionally must ensure that no new
+   vulnerabilities are introduced.  For RDMA, the integrity, and any
+   privacy, of the data stream are of particular importance.
+
+   The core goals of an NFS-to-RDMA binding are to reduce overhead and
+   to enable high performance.  To support these goals while maintaining
+   required NFS security protection presents a special challenge.
+   Historically, the provision of integrity and privacy have been
+   implemented within the RPC layer, and their operation requires local
+   processing of messages exchanged with the RPC peer.  This processing
+   imposes memory and processing overhead on a per-message basis,
+   exactly the overhead that RDMA is designed to avoid.
+
+   Therefore, it is a requirement that the RDMA transport binding
+   provide a means to delegate the integrity and privacy processing to
+   the RDMA hardware, in order to maintain the high level of performance
+   desired from the approach, while simultaneously providing the
+   existing highest levels of security required by the NFS protocol.
+   This in turn requires a means by which the RPC layer may invoke these
+   services from the RDMA provider, and for the NFS layer to negotiate
+   their use end-to-end.
+
+   The "Channel Binding" concept [RFC5056] together with "IPsec Channel
+   Connection Latching" [BTNSLATCH] provide a means by which the RPC and
+   NFS layers may delegate their session protection to the lower RDMA
+
+
+
+Talpey & Juszczak            Informational                     [Page 11]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+   layers.  An extension to the RPCSEC_GSS protocol [RFC5403] may be
+   employed to negotiate the use of these bindings, and to establish the
+   shared secrets necessary to protect the sessions.
+
+   The protocol described in [RPCRDMA] specifies the use of these
+   mechanisms, and they are required to implement the protocol.
+
+   An additional consideration is protection of the integrity and
+   privacy of local memory by the RDMA transport itself.  The use of
+   RDMA by NFS must not introduce any vulnerabilities to system memory
+   contents, or to memory owned by user processes.  These protections
+   are provided by the RDMA layer specifications, and specifically their
+   security models.  It is required that any RDMA provider used for NFS
+   transport be conformant to the requirements of [RFC5042] in order to
+   satisfy these protections.
+
+8.  Acknowledgments
+
+   The authors wish to thank Jeff Chase who provided many useful
+   suggestions.
+
+9.  References
+
+9.1.  Normative References
+
+   [RFC3530]   Shepler, S., Callaghan, B., Robinson, D., Thurlow, R.,
+               Beame, C., Eisler, M., and D. Noveck, "Network File
+               System (NFS) version 4 Protocol", RFC 3530, April 2003.
+
+   [RFC5531]   Thurlow, R., "RPC: Remote Procedure Call Protocol
+               Specification Version 2", RFC 5531, May 2009.
+
+   [RFC4506]   Eisler, M., Ed., "XDR: External Data Representation
+               Standard", STD 67, RFC 4506, May 2006.
+
+   [RFC1813]   Callaghan, B., Pawlowski, B., and P. Staubach, "NFS
+               Version 3 Protocol Specification", RFC 1813, June 1995.
+
+   [RFC5403]   Eisler, M., "RPCSEC_GSS Version 2", RFC 5403, February
+               2009.
+
+   [RFC5056]   Williams, N., "On the Use of Channel Bindings to Secure
+               Channels", RFC 5056, November 2007.
+
+   [RFC5042]   Pinkerton, J. and E. Deleganes, "Direct Data Placement
+               Protocol (DDP) / Remote Direct Memory Access Protocol
+               (RDMAP) Security", RFC 5042, October 2007.
+
+
+
+
+Talpey & Juszczak            Informational                     [Page 12]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+9.2.  Informative References
+
+   [BRU99]     J. Brustoloni, "Interoperation of copy avoidance in
+               network and file I/O", in Proc. INFOCOM '99, pages 534-
+               542, New York, NY, Mar. 1999., IEEE.  Also available from
+               http://www.cs.pitt.edu/~jcb/publs.html.
+
+   [BTNSLATCH] Williams, N., "IPsec Channels: Connection Latching", Work
+               in Progress, November 2008.
+
+   [CAL+03]    B. Callaghan, T. Lingutla-Raj, A. Chiu, P. Staubach, O.
+               Asad, "NFS over RDMA", in Proceedings of ACM SIGCOMM
+               Summer 2003 NICELI Workshop.
+
+   [CHA+01]    J. S. Chase, A. J. Gallatin, K. G. Yocum, "Endsystem
+               optimizations for high-speed TCP", IEEE Communications,
+               39(4):68-74, April 2001.
+
+   [CHA+99]    J. S. Chase, D. C. Anderson, A. J. Gallatin, A. R.
+               Lebeck, K.  G. Yocum, "Network I/O with Trapeze", in 1999
+               Hot Interconnects Symposium, August 1999.
+
+   [CHU96]     H.K. Chu, "Zero-copy TCP in Solaris", Proc. of the USENIX
+               1996 Annual Technical Conference, San Diego, CA, January
+               1996.
+
+   [DCK+03]    M. DeBergalis, P. Corbett, S. Kleiman, A. Lent, D.
+               Noveck, T.  Talpey, M. Wittle, "The Direct Access File
+               System", in Proceedings of 2nd USENIX Conference on File
+               and Storage Technologies (FAST '03), San Francisco, CA,
+               March 31 - April 2, 2003.
+
+   [FJDAFS]    Fujitsu Prime Software Technologies, "Meet the DAFS
+               Performance with DAFS/VI Kernel Implementation using
+               cLAN", available from
+               http://www.pst.fujitsu.com/english/dafsdemo/index.html,
+               2001.
+
+   [FJNFS]     Fujitsu Prime Software Technologies, "An Adaptation of
+               VIA to NFS on Linux", available from
+               http://www.pst.fujitsu.com/english/nfs/index.html, 2000.
+
+   [GAL+99]    A. Gallatin, J. Chase, K. Yocum, "Trapeze/IP: TCP/IP at
+               Near-Gigabit Speeds", 1999 USENIX Technical Conference
+               (Freenix Track), June 1999.
+
+
+
+
+
+
+Talpey & Juszczak            Informational                     [Page 13]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+   [KM02]      K. Magoutis, "Design and Implementation of a Direct
+               Access File System (DAFS) Kernel Server for FreeBSD", in
+               Proceedings of USENIX BSDCon 2002 Conference, San
+               Francisco, CA, February 11-14, 2002.
+
+   [MAF+02]    K. Magoutis, S. Addetia, A. Fedorova, M. Seltzer, J.
+               Chase, D.  Gallatin, R. Kisley, R. Wickremesinghe, E.
+               Gabber, "Structure and Performance of the Direct Access
+               File System (DAFS)", in Proceedings of 2002 USENIX Annual
+               Technical Conference, Monterey, CA, June 9-14, 2002.
+
+   [MOG03]     J. Mogul, "TCP offload is a dumb idea whose time has
+               come", 9th Workshop on Hot Topics in Operating Systems
+               (HotOS IX), Lihue, HI, May 2003. USENIX.
+
+   [NFSv4.1]   Shepler, S., Eisler, M., and D. Noveck, "NFSv4 Minor
+               Version 1", Work in Progress, September 2008.
+
+   [PAI+00]    V. S. Pai, P. Druschel, W. Zwaenepoel, "IO-Lite: a
+               unified I/O buffering and caching system", ACM Trans.
+               Computer Systems, 18(1):37-66, Feb. 2000.
+
+   [RDDP]      RDDP Working Group charter,
+               http://www.ietf.org/html.charters/rddpcharter.html.
+
+   [RFC4297]   Romanow, A., Mogul, J., Talpey, T., and S. Bailey,
+               "Remote Direct Memory Access (RDMA) over IP Problem
+               Statement", RFC 4297, December 2005.
+
+   [RFC1094]   Sun Microsystems, "NFS: Network File System Protocol
+               specification", RFC 1094, March 1989.
+
+   [RPCRDMA]   Talpey, T. and B. Callaghan, "Remote Direct Memory Access
+               Transport for Remote Procedure Call", Work in Progress,
+               April 2008.
+
+   [SHI+03]    P. Shivam, J. Chase, "On the Elusive Benefits of Protocol
+               Offload", Proceedings of ACM SIGCOMM Summer 2003 NICELI
+               Workshop, also available from
+               http://issg.cs.duke.edu/publications/niceli03.pdf.
+
+   [SKE+01]    K.-A. Skevik, T. Plagemann, V. Goebel, P. Halvorsen,
+               "Evaluation of a Zero-Copy Protocol Implementation", in
+               Proceedings of the 27th Euromicro Conference - Multimedia
+               and Telecommunications Track (MTT'2001), Warsaw, Poland,
+               September 2001.
+
+
+
+
+
+Talpey & Juszczak            Informational                     [Page 14]
+
+RFC 5532               NFS RDMA Problem Statement               May 2009
+
+
+Authors' Addresses
+
+   Tom Talpey
+   170 Whitman St.
+   Stow, MA 01775 USA
+
+   Phone: +1 978 821-8577
+   EMail: tmtalpey@gmail.com
+
+
+   Chet Juszczak
+   P.O. Box 1467
+   Merrimack, NH 03054
+
+   Phone: +1 603 253-6602
+   EMail: chetnh@earthlink.net
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Talpey & Juszczak            Informational                     [Page 15]
+