doc: Add RFC documents

1 files changed, 1907 insertions, 0 deletions

diff --git a/doc/rfc/rfc7306.txt b/doc/rfc/rfc7306.txt
new file mode 100644
index 0000000..e3b1a7b
--- /dev/null
+++ b/doc/rfc/rfc7306.txt
@@ -0,0 +1,1907 @@
+
+
+
+
+
+
+Internet Engineering Task Force (IETF)                           H. Shah
+Request for Comments: 7306                          Broadcom Corporation
+Category: Standards Track                                       F. Marti
+ISSN: 2070-1721                                            W. Noureddine
+                                                            A. Eiriksson
+                                            Chelsio Communications, Inc.
+                                                                R. Sharp
+                                                       Intel Corporation
+                                                               June 2014
+
+
+         Remote Direct Memory Access (RDMA) Protocol Extensions
+
+Abstract
+
+   This document specifies extensions to the IETF Remote Direct Memory
+   Access Protocol (RDMAP) as specified in RFC 5040.  RDMAP provides
+   read and write services directly to applications and enables data to
+   be transferred directly into Upper-Layer Protocol (ULP) Buffers
+   without intermediate data copies.  The extensions specified in this
+   document provide the following capabilities and/or improvements:
+   Atomic Operations and Immediate Data.
+
+Status of This Memo
+
+   This is an Internet Standards Track document.
+
+   This document is a product of the Internet Engineering Task Force
+   (IETF).  It represents the consensus of the IETF community.  It has
+   received public review and has been approved for publication by the
+   Internet Engineering Steering Group (IESG).  Further information on
+   Internet Standards is available in Section 2 of RFC 5741.
+
+   Information about the current status of this document, any errata,
+   and how to provide feedback on it may be obtained at
+   http://www.rfc-editor.org/info/rfc7306.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Shah, et al.                 Standards Track                    [Page 1]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+Copyright Notice
+
+   Copyright (c) 2014 IETF Trust and the persons identified as the
+   document authors.  All rights reserved.
+
+   This document is subject to BCP 78 and the IETF Trust's Legal
+   Provisions Relating to IETF Documents
+   (http://trustee.ietf.org/license-info) in effect on the date of
+   publication of this document.  Please review these documents
+   carefully, as they describe your rights and restrictions with respect
+   to this document.  Code Components extracted from this document must
+   include Simplified BSD License text as described in Section 4.e of
+   the Trust Legal Provisions and are provided without warranty as
+   described in the Simplified BSD License.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Shah, et al.                 Standards Track                    [Page 2]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+Table of Contents
+
+   1. Introduction ....................................................4
+      1.1. Discovery of RDMAP Extensions ..............................5
+   2. Requirements Language ...........................................5
+   3. Glossary ........................................................6
+   4. Header Format Extensions ........................................7
+      4.1. RDMAP Control and Invalidate STag Fields ...................7
+      4.2. RDMA Message Definitions ...................................9
+   5. Atomic Operations ...............................................9
+      5.1. Atomic Operation Details ..................................10
+           5.1.1. FetchAdd ...........................................10
+           5.1.2. CmpSwap ............................................12
+      5.2. Atomic Operations .........................................13
+           5.2.1. Atomic Operation Request Message ...................14
+           5.2.2. Atomic Operation Response Message ..................17
+      5.3. Atomicity Guarantees ......................................18
+      5.4. Atomic Operations Ordering and Completion Rules ...........18
+   6. Immediate Data .................................................20
+      6.1. RDMAP Interactions with ULP for Immediate Data ............20
+      6.2. Immediate Data Header Format ..............................21
+      6.3. Immediate Data or Immediate Data with SE Message ..........21
+      6.4. Ordering and Completions ..................................22
+   7. Ordering and Completions Table .................................22
+   8. Error Processing ...............................................25
+      8.1. Errors Detected at the Local Peer .........................25
+      8.2. Errors Detected at the Remote Peer ........................26
+   9. Security Considerations ........................................26
+   10. IANA Considerations ...........................................27
+      10.1. RDMAP Message Atomic Operation Subcodes ..................27
+      10.2. RDMAP Queue Numbers ......................................28
+   11. References ....................................................29
+      11.1. Normative References .....................................29
+      11.2. Informative References ...................................29
+   12. Acknowledgments ...............................................30
+   Appendix A. DDP Segment Formats for RDMA Messages .................31
+      A.1. DDP Segment for Atomic Operation Request ..................32
+      A.2. DDP Segment for Atomic Response ...........................33
+      A.3. DDP Segment for Immediate Data and Immediate Data with SE .33
+
+
+
+
+
+
+
+
+
+
+
+
+Shah, et al.                 Standards Track                    [Page 3]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+1.  Introduction
+
+   The RDMA Protocol [RFC5040] provides capabilities for zero-copy data
+   communications that preserve memory protection semantics, enabling
+   more efficient network protocol implementations.  The RDMA Protocol
+   is part of the iWARP family of specifications which also include RFC
+   5041 [RFC5041], RFC 5044 [RFC5044], and RFC 6581 [RFC6581].  This
+   document specifies the following extensions to the RDMA Protocol
+   (RDMAP):
+
+   o  Atomic Operations can be performed on remote memory locations.
+      Support for Atomic Operations enhances the usability of RDMAP in
+      distributed shared-memory environments.
+
+   o  Immediate Data messages allow the ULP at the sender to provide a
+      small amount of data.  When an Immediate Data message is sent
+      following an RDMA Write Message, the combination of the two
+      messages is an implementation of RDMA Write with Immediate message
+      that is found in other RDMA transport protocols.
+
+   Other RDMA transport protocols define the functionality added by
+   these extensions leading to differences in RDMA applications and/or
+   Upper-Layer Protocols.  Removing these differences in the transport
+   protocols simplifies these applications and ULPs, and that is the
+   main motivation for the extensions specified in this document.
+
+   RSockets [RSOCKETS] is an example of RDMA-enabled middleware that
+   provides a socket interface as the upper-edge interface and utilizes
+   RDMA to provide more efficient networking for socket-based
+   applications.  RSockets is aware of Immediate Data support in
+   InfiniBand [IB].  RSockets cannot utilize the RDMA Write with
+   Immediate Data operation from InfiniBand.  The addition of the
+   Immediate Data operation specified in this document will alleviate
+   this difference in RSockets when running on InfiniBand and iWARP.
+
+   Structured high-performance computing applications based on the
+   Message-Passing Interface [MPI] may use Atomic Operations defined in
+   this specification.  DAT Atomics [DAT_ATOMICS] is an example of RDMA-
+   enabled middleware that provides a portable RDMA programming
+   interface for various RDMA transport protocols.  DAT Atomics includes
+   a primitive for InfiniBand that is not supported by iWARP RDMA-
+   enabled Network Interface Controllers or RNICs.  The addition of
+   Atomic Operations as specified in this document will allow Atomic
+   Operations in DAT Atomics to work for both InfiniBand and RNICs
+   interchangeably.
+
+
+
+
+
+
+Shah, et al.                 Standards Track                    [Page 4]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   For more background on RDMA Protocol applicability, see
+   "Applicability of Remote Direct Memory Access Protocol (RDMA) and
+   Direct Data Placement Protocol (DDP)" [RFC5045].
+
+1.1.  Discovery of RDMAP Extensions
+
+   Today there are RDMA applications and/or ULPs that are aware of the
+   existence of Atomic and Immediate Data operations for RDMA transports
+   such as InfiniBand and application programming interfaces such as
+   Open Fabrics Verbs [OFAVERBS].  Today, these applications need to be
+   aware that RDMAP does not support certain of these operations.
+   Typically, the availability of these capabilities is exposed to the
+   applications through adapter query interfaces in software.
+   Applications then have to decide to use or not use Immediate Data or
+   Atomic Operations based on the results of the query interfaces.  Such
+   query interfaces typically return the scope of atomicity guarantees,
+   not the individual Atomic Operations supported.  Therefore, this
+   specification requires all Atomic Operations defined within to be
+   supported if an RNIC supports any Atomic Operations.
+
+   In cases where heterogeneous hardware, with differing support for
+   Atomic Operations and Immediate Data Operations, is deployed for use
+   by RDMA applications and/or ULPs, applications are either statically
+   configured to use or not use optional features or use application-
+   specific negotiation mechanisms.  For the extensions covered by this
+   document, it is RECOMMENDED that RDMA applications and/or ULPs
+   negotiate at the application or ULP level the usage of these
+   extensions.  The definition of such application-specific mechanisms
+   is outside the scope of this specification.  For backward
+   compatibility, existing applications and/or ULPs should not assume
+   that these extensions are supported.
+
+   In the absence of application-specific negotiation of the features
+   defined within this specification, the new operations can be
+   attempted, and reported errors can be used to determine a remote
+   peer's capabilities.  In the case of Atomics, a FetchAdd operation
+   with "Add Data" set to 0 can safely be used to determine the
+   existence of Atomic Operations without modifying the content of a
+   remote peer's memory.  A Remote Operation Error or Unexpected OpCode
+   error will be reported by the remote peer if there is an Immediate
+   Data or Atomic Operation that is not supported by the remote peer.
+
+2.  Requirements Language
+
+   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
+   document are to be interpreted as described in RFC 2119 [RFC2119].
+
+
+
+
+Shah, et al.                 Standards Track                    [Page 5]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+3.  Glossary
+
+   This document is an extension of RFC 5040, and key words are defined
+   in the glossary of that document.
+
+   Atomic Operation - an operation that results in an execution of a
+      memory operation at a specific ULP Buffer address on a remote node
+      using the Tagged Buffer data transfer model.  The consumer can use
+      Atomic Operations to read, modify, and write memory at the
+      destination ULP Buffer address, while at the same time
+      guaranteeing that no other Atomic Operation read or write accesses
+      to the ULP Buffer address targeted by the Atomic Operation will
+      occur across any other RDMAP Streams on an RNIC at the Responder.
+
+   Atomic Operation Request - an RDMA Message used by the Data Source to
+      perform an Atomic Operation at the Responder.
+
+   Atomic Operation Response - an RDMA Message used by the Responder to
+      describe the completion of an Atomic Operation at the Responder.
+
+   CmpSwap - an Atomic Operation that is used to compare and swap a
+      value at a specific address on a remote node.
+
+   FetchAdd - an Atomic Operation that is used to atomically increment a
+      value at a specific ULP Buffer address on a remote node.
+
+   Immediate Data - a small fixed-size portion of data sent from the
+      Data Source to a Data Sink.
+
+   Immediate Data Message - an RDMA Message used by the Data Source to
+      send Immediate Data to the Data Sink.
+
+   Immediate Data with Solicited Event (SE) Message - an RDMA Message
+      used by the Data Source to send Immediate Data with Solicited
+      Event to the Data Sink.
+
+   iWARP - a suite of wire protocols comprised of RFC 5040, RFC 5041,
+      RFC 5044, and RFC 6581.
+
+   Requester - the sender of an RDMA Atomic Operation request.
+
+   Responder - the receiver of an RDMA Atomic Operation request.
+
+   RNIC - RDMA-enabled Network Interface Controller.  In this context,
+      this would be a network I/O adapter or embedded controller with
+      iWARP functionality.
+
+
+
+
+
+Shah, et al.                 Standards Track                    [Page 6]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   ULP - Upper-Layer Protocol.  The protocol layer above the one
+      currently being referenced.  The ULP for RFC 5040 / RFC 5041 is
+      expected to be an OS, Application, adaptation layer, or
+      proprietary device.  The RFC 5040 / RFC 5041 documents do not
+      specify a ULP -- they provide a set of semantics that allow a ULP
+      to be designed to utilize RFC 5040 / RFC 5041.
+
+4.  Header Format Extensions
+
+   The control information of RDMA Messages is included in header fields
+   defined in RFC 5041, the Direct Data Placement (DDP) protocol.  RFC
+   5040 defines the RDMAP header formats layered on the DDP header
+   definition.  This specification extends RFC 5040 with the following
+   new formats:
+
+   o  Four new RDMA Messages carry additional RDMAP headers.  The
+      Immediate Data operation and Immediate Data with Solicited Event
+      operation each include 8 bytes of data following the RDMAP header.
+      Atomic Operations include Atomic Request or Atomic Response
+      headers following the RDMAP header.  The RDMAP header for Atomic
+      Request messages is 52 bytes long as specified in Figure 4.  The
+      RDMAP header for Atomic Response Messages is 32 bytes long as
+      specified in Figure 5.
+
+   o  Introduction of a new queue for untagged Buffers (QN=3) used for
+      Atomic Response tracking.
+
+4.1.  RDMAP Control and Invalidate STag Fields
+
+   For reference, Figure 1 depicts the format of the DDP Control and
+   RDMAP Control Fields, in the style and convention of RFC 5040:
+
+   0                   1                   2                   3
+   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |T|L| Resrv | DV| RV|Rsv| Opcode|
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                     Invalidate STag                           |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+            Figure 1: DDP Control and RDMAP Control Fields
+
+   The DDP Control Field consists of the T (Tagged), L (Last), Resrv,
+   and DV (DDP protocol Version) fields [RFC5041].  The RDMAP Control
+   Field consists of the RV (RDMA Version), Rsv, and Opcode fields
+   [RFC5040].
+
+
+
+
+
+Shah, et al.                 Standards Track                    [Page 7]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   This specification adds values for the RDMA Opcode field to those
+   specified in RFC 5040.  Figure 2 defines the new values of the RDMA
+   Opcode field that are used for the RDMA Messages defined in this
+   specification.
+
+   As shown in Figure 2, STag (Steering Tag) and Tagged Offset are not
+   applicable for the RDMA Messages defined in this specification.
+   Figure 2 also shows the appropriate Queue Number for each Opcode.
+
+   All RDMA Messages defined in this specification MUST have:
+
+   The RDMA Version (RV) field: 01b.
+
+   Opcode field: Set to one of the values in Figure 2.
+
+   Invalidate STag: Set to zero by the sender, ignored by the receiver.
+
+   -------+-----------+-------+------+-------+---------+-------------
+   RDMA   | Message   | Tagged| STag | Queue | In-     | Message
+   Opcode | Type      | Flag  | and  | Number| validate| Length
+          |           |       | TO   |       | STag    | Communicated
+          |           |       |      |       |         | between DDP
+          |           |       |      |       |         | and RDMAP
+   -------+-----------+-------+------+-------+---------+-------------
+   1000b  | Immediate | 0     | N/A  | 0     | N/A     | Yes
+          | Data      |       |      |       |         |
+   -------+-----------+----------------------------------------------
+   1001b  | Immediate | 0     | N/A  | 0     | N/A     | Yes
+          | Data with |       |      |       |         |
+          | SE        |       |      |       |         |
+   -------+-----------+----------------------------------------------
+   1010b  | Atomic    | 0     | N/A  | 1     | N/A     | Yes
+          | Request   |       |      |       |         |
+   -------+-----------+----------------------------------------------
+   1011b  | Atomic    | 0     | N/A  | 3     | N/A     | Yes
+          | Response  |       |      |       |         |
+   -------+-----------+----------------------------------------------
+
+              Figure 2: Additional RDMA Usage of DDP Fields
+
+   Note:  N/A means Not Applicable.
+
+   This extension defines RDMAP use of Queue Number 3 for Untagged
+   Buffers for Atomic Responses.  This queue is used for tracking
+   outstanding Atomic Requests.
+
+   All other DDP and RDMAP Control Fields are set as described in RFC
+   5040.
+
+
+
+Shah, et al.                 Standards Track                    [Page 8]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+4.2.  RDMA Message Definitions
+
+   The following figure defines which RDMA Headers are used on each new
+   RDMA Message and which new RDMA Messages are allowed to carry ULP
+   payload.
+
+   -------+-----------+-------------------+-------------------------
+   RDMA   | Message   | RDMA Header Used  | ULP Message allowed in
+   Message| Type      |                   | the RDMA Message
+   OpCode |           |                   |
+          |           |                   |
+   -------+-----------+-------------------+-------------------------
+   1000b  | Immediate | Immediate Data    | No
+          | Data      | Header            |
+   -------+-----------+-------------------+-------------------------
+   1001b  | Immediate | Immediate Data    | No
+          | Data with | Header            |
+          | SE        |                   |
+   -------+-----------+-------------------+-------------------------
+   1010b  | Atomic    | Atomic Request    | No
+          | Request   | Header            |
+   -------+-----------+-------------------+-------------------------
+   1011b  | Atomic    | Atomic Response   | No
+          | Response  | Header            |
+   -------+-----------+-------------------+-------------------------
+
+                     Figure 3: RDMA Message Definitions
+
+5.  Atomic Operations
+
+   The RDMA Protocol Specification in RFC 5040 does not include support
+   for Atomic Operations, which are an important building block for
+   implementing distributed shared memory.
+
+   This document extends the RDMA Protocol specification with a set of
+   basic Atomic Operations and specifies their resource and ordering
+   rules.  The Atomic Operations specified in this document provide
+   equivalent functionality to the InfiniBand RDMA transport as well as
+   extended Atomic Operations defined in Open Fabrics Verbs, to allow
+   applications that use these primitives to work interchangeably over
+   iWARP.  Other operations are left for future consideration.
+
+   Atomic Operations as specified in this document execute a 64-bit
+   memory operation at a specified destination ULP Buffer address on a
+   Responder node using the Tagged Buffer data transfer model.  The
+   operations atomically read, modify, and write back the contents of
+   the destination ULP Buffer address and guarantee that Atomic
+   Operations on this ULP Buffer address by other RDMAP Streams on the
+
+
+
+Shah, et al.                 Standards Track                    [Page 9]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   same RNIC do not occur between the read and the write caused by the
+   Atomic Operation.  Therefore, the Responder RNIC MUST implement
+   mechanisms to prevent Atomic Operations to a memory registered for
+   Atomic Operations while an Atomic Operation targeting the memory is
+   in progress.  The Requester of an Atomic Operation cannot rely on
+   Atomic Operation behavior at the Responder across multiple RNICs or
+   with respect to other applications/ULPs running at the Responder that
+   can access the ULP Buffer.  It is OPTIONAL for an RNIC to provide
+   such behavior when implementing the Atomic Operations specified in
+   this document.  An RNIC that supports Atomic Operations as specified
+   in this document MUST implement both the FetchAdd operation as
+   specified in Section 5.1.1 and the CmpSwap operation as specified in
+   Section 5.1.2.  The advertisement of Tagged Buffer information for
+   Atomic Operations is outside the scope of this specification and is
+   handled by the ULPs.
+
+   Implementation note: It is RECOMMENDED that the applications do not
+   use the ULP Buffer addresses used for Atomic Operations for other
+   RDMA operations due to the lack of atomicity guarantees between
+   operations other than Atomic Operations.
+
+   Implementation note: Errors related to the alignment in the following
+   sections cover Atomic Operations targeted at a ULP Buffer address
+   that is not aligned to a 64-bit boundary.
+
+   Atomic Operation Request Messages use the same remote addressing
+   mechanism as RDMA Reads and Writes.  The ULP Buffer address specified
+   in the request is in the address space of the Remote Peer to which
+   the Atomic Operation is targeted.
+
+   Atomic Operation Response Messages MUST use the Untagged Buffer model
+   with QN=3.  Queue number 3 will be used to track outstanding Atomic
+   Operation Request messages at the Requester.  When the Atomic
+   Operation Response message is received, the Message Sequence Number
+   (MSN) will be used to locate the corresponding Atomic Operation
+   request in order to complete the Atomic Operation request.
+
+5.1.  Atomic Operation Details
+
+   The following subsections describe the Atomic Operations in more
+   detail.
+
+5.1.1.  FetchAdd
+
+   The FetchAdd Atomic Operation requests the Responder to read a 64-bit
+   Original Remote Data Value at a 64-bit aligned ULP Buffer address in
+   the Responder's memory, perform the FetchAdd operation on multiple
+   fields of selectable length specified by 64-bit "Add Mask", and write
+
+
+
+Shah, et al.                 Standards Track                   [Page 10]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   the result back to the same ULP Buffer address.  The Atomic addition
+   is performed independently on each one of these fields.  A bit set in
+   the Add Mask field specifies the field boundary; for each field, a
+   bit is set at the most significant bit position for each field,
+   causing any carry out of that bit position to be discarded when the
+   addition is performed.
+
+   FetchAdd Atomic Operations MUST target ULP Buffer addresses that are
+   64-bit aligned.  FetchAdd Atomic Operations that target ULP Buffer
+   addresses that are not 64-bit aligned MUST be surfaced as errors, and
+   the Responder's memory MUST NOT be modified in such cases.
+   Additionally, an error MUST be surfaced and a terminate message MUST
+   be generated.  The setting of the Add Mask field to
+   0x0000000000000000 results in Atomic Add of 64-bit Original Remote
+   Data Value and 64-bit "Add Data".
+
+   The pseudocode below describes a masked FetchAdd Atomic Operation.
+
+   bit_location = 1
+
+   carry = 0
+
+   Remote Data Value = 0
+
+   for bit = 0 to 63
+
+   {
+
+      if (bit != 0 ) bit_location = bit_location << 1
+
+      val1 = (Original Remote Data Value & bit_location) >> bit
+
+      val2 = (Add Data & bit_location) >> bit
+      sum = carry + val1 + val2
+
+      carry = (sum & 2) >> 1
+
+      sum = sum & 1
+
+      if (sum)
+
+         Remote Data Value |= bit_location
+
+      carry = ((carry) && (!(Add Mask & bit_location)))
+
+   }
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 11]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   The FetchAdd operation is performed in the endian format of the
+   target memory.  The "Original Remote Data Value" is converted from
+   the endian format of the target memory for return and returned to the
+   Requester.  The fields are in big-endian format on the wire.
+
+   The Requester specifies:
+
+   o  Remote STag
+
+   o  Remote Tagged Offset
+
+   o  Add Data
+
+   o  Add Mask
+
+   The Responder returns:
+
+   o  Original Remote Data
+
+5.1.2.  CmpSwap
+
+   The CmpSwap Atomic Operation requires the Responder to read a 64-bit
+   value at a ULP Buffer address that is 64-bit aligned in the
+   Responder's memory, to perform an AND logical operation using the
+   64-bit Compare Mask field in the Atomic Operation Request header,
+   then to compare it with the result of a logical AND operation of the
+   Compare Mask and the Compare Data fields in the header.  If the two
+   values are equal, the Responder is required to swap masked bits in
+   the same ULP Buffer address with the masked Swap Data.  If the two
+   masked compare values are not equal, the contents of the Responder's
+   memory are not changed.  In either case, the original value read from
+   the ULP Buffer address is converted from the endian format of the
+   target memory for return and returned to the Requester.  The fields
+   are in big-endian format on the wire.
+
+   The Requester specifies:
+
+   o  Remote STag
+
+   o  Remote Tagged Offset
+
+   o  Swap Data
+
+   o  Swap Mask
+
+   o  Compare Data
+
+   o  Compare Mask
+
+
+
+Shah, et al.                 Standards Track                   [Page 12]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   The Responder returns:
+
+   o  Original Remote Data Value
+
+   The following pseudocode describes the masked CmpSwap operation
+   result.
+
+      if (!((Compare Data ^ Original Remote Data Value) &
+
+            Compare Mask))
+
+      then
+
+         Remote Data Value =
+
+           (Original Remote Data Value & ~(Swap Mask))
+
+                             | (Swap Data & Swap Mask)
+
+      else
+
+         Remote Data Value = Original Remote Data Value
+
+   After the operation, the remote data Buffer MUST contain the
+   "Original Remote Data Value" (if comparison did not match) or the
+   masked "Swap Data" (if the comparison did match).  CmpSwap Atomic
+   Operations MUST target ULP Buffer addresses that are 64-bit aligned.
+
+   If a CmpSwap Atomic Operation is attempted on a target ULP Buffer
+   address that is not 64-bit aligned:
+
+   o  The operation MUST NOT be performed,
+
+   o  The Responder's memory MUST NOT be modified,
+
+   o  The result MUST be surfaced as an error, and
+
+   o  A terminate message MUST be generated.  (See Section 8.2 for the
+      contents of the terminate message.)
+
+5.2.  Atomic Operations
+
+   The Atomic Operation Request and Response are RDMA Messages.  An
+   Atomic Operation makes use of the DDP Untagged Buffer Model.  Atomic
+   Operation Request messages MUST use the same Queue Number as RDMA
+   Read Requests (QN=1).  Reusing the same Queue Number for Atomic
+   Request messages allows the Atomic Operations to reuse the same
+   infrastructure (e.g., Outbound and Inbound RDMA Read Queue Depth
+
+
+
+Shah, et al.                 Standards Track                   [Page 13]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   (ORD/IRD) flow control) as defined for RDMA Read Requests.  Atomic
+   Operation Response messages MUST set Queue Number (QN) to 3 in the
+   DDP header.
+
+   The RDMA Message OpCode for an Atomic Request Message is 1010b.  The
+   RDMA Message OpCode for an Atomic Response Message is 1011b.
+
+5.2.1.  Atomic Operation Request Message
+
+   The Atomic Operation Request Message carries an Atomic Operation
+   Header that describes the ULP Buffer address in the Responder's
+   memory.  The Atomic Operation Request header immediately follows the
+   DDP header.  The RDMAP layer passes to the DDP layer a RDMAP Control
+   Field.  The following figure depicts the Atomic Operation Request
+   Header that is used for all Atomic Operation Request Messages:
+
+    0                   1                   2                   3
+    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                      Reserved (Not Used)              |AOpCode|
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                       Request Identifier                      |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                          Remote STag                          |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                      Remote Tagged Offset                     |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                       Add or Swap Data                        |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                       Add or Swap Mask                        |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                         Compare Data                          |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                         Compare Mask                          |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+              Figure 4: Atomic Operation Request Header
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 14]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+      Reserved (Not Used): 28 bits
+
+         This field is set to zero on transmit, ignored on receive.
+
+      Atomic Operation Code (AOpCode): 4 bits.
+
+         See Figure 5.  All Atomic Operation Codes from Figure 5 MUST be
+         implemented by an RNIC that supports Atomic Operations.
+
+      Request Identifier: 32 bits.
+
+         The Request Identifier specifies a number that is used to
+         identify the Atomic Operation Request Message.  The value used
+         in this field is selected by the RNIC that sends the message,
+         and it is reflected back to the Local Peer in the Atomic
+         Operation Response message.
+
+      Remote STag: 32 bits.
+
+         The Remote STag identifies the Remote Peer's Tagged Buffer
+         targeted by the Atomic Operation.  The Remote STag is
+         associated with the RDMAP Stream through a mechanism that is
+         outside the scope of the RDMAP specification.
+
+      Remote Tagged Offset: 64 bits.
+
+         The Remote Tagged Offset specifies the starting offset, in
+         octets, from the base of the Remote Peer's Tagged Buffer
+         targeted by the Atomic Operation.  The Remote Tagged Offset MAY
+         start at an arbitrary offset but MUST represent a ULP Buffer
+         address that is 64-bit aligned.
+
+      Add or Swap Data: 64 bits.
+
+         The Add or Swap Data field specifies the 64-bit "Add Data"
+         value in an Atomic FetchAdd Operation or the 64-bit "Swap Data"
+         value in an Atomic Swap or CmpSwap Operation.
+
+      Add or Swap Mask: 64 bits
+
+         This field is used in masked Atomic Operations (FetchAdd and
+         CmpSwap) to perform a bitwise logical AND operation as
+         specified in the definition of these operations.  For non-
+         masked Atomic Operations (Swap), this field is set to
+         ffffffffffffffffh on transmit and ignored by the receiver.
+
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 15]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+      Compare Data: 64 bits.
+
+         The Compare Data field specifies the 64-bit "Compare Data"
+         value in an Atomic CmpSwap Operation.  For Atomic Operations
+         FetchAdd and Atomic Swap, the Compare Data field is set to zero
+         on transmit and ignored by the receiver.
+
+      Compare Mask: 64 bits
+
+         This field is used in masked Atomic Operation CmpSwap to
+         perform a bitwise logical AND operation as specified in the
+         definition of these operations.  For Atomic Operations FetchAdd
+         and Swap, this field is set to ffffffffffffffffh on transmit
+         and ignored by the receiver.
+
+   ---------+-----------+----------+----------+---------+---------
+   Atomic   | Atomic    | Add or   | Add or   | Compare | Compare
+   Operation| Operation | Swap     | Swap     | Data    | Mask
+   Code     |           | Data     | Mask     |         |
+   ---------+-----------+----------+----------+---------+---------
+   0000b    | FetchAdd  | Add Data | Add Mask | N/A     | N/A
+   ---------+-----------+----------+----------+---------+---------
+   0010b    | CmpSwap   | Swap Data| Swap Mask| Valid   | Valid
+   ---------+-----------+-----------------------------------------
+
+            Figure 5: Atomic Operation Message Definitions
+
+   The Atomic Operation Request Message has the following semantics:
+
+   1. An Atomic Operation Request Message MUST reference an Untagged
+      Buffer.  That is, the Local Peer's RDMAP layer MUST request that
+      the DDP mark the Message as Untagged.
+
+   2. One Atomic Operation Request Message MUST consume one Untagged
+      Buffer.
+
+   3. The Responder's RDMAP layer MUST process an Atomic Operation
+      Request Message.  A valid Atomic Operation Request Message MUST
+      NOT be delivered to the Responder's ULP (i.e., it is processed by
+      the RDMAP layer).
+
+   4. At the Responder, an error MUST be surfaced in response to
+      delivery to the Remote Peer's RDMAP layer of an Atomic Operation
+      Request Message with an Atomic Operation Code that the RNIC does
+      not support.
+
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 16]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   5. An Atomic Operation Request Message MUST reference the RDMA Read
+      Request Queue.  That is, the Requester's RDMAP layer MUST request
+      that the DDP layer set the Queue Number field to one.
+
+   6. The Requester MUST pass to the DDP layer Atomic Operation Request
+      Messages in the order they were submitted by the ULP.
+
+   7. The Responder MUST process the Atomic Operation Request Messages
+      in the order they were sent.
+
+   8. If the Responder receives a valid Atomic Operation Request
+      Message, it MUST respond with a valid Atomic Operation Response
+      Message.
+
+5.2.2.  Atomic Operation Response Message
+
+   The Atomic Operation Response Message carries an Atomic Operation
+   Response Header that contains the "Original Request Identifier" and
+   "Original Remote Data Value".  The Atomic Operation Response Header
+   immediately follows the DDP header.  The RDMAP layer passes to the
+   DDP layer a RDMAP Control Field.  The following figure depicts the
+   Atomic Operation Response header that is used for all Atomic
+   Operation Response Messages:
+
+    0                   1                   2                   3
+    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                    Original Request Identifier                |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                    Original Remote Data Value                 |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+                Figure 6: Atomic Operation Response Header
+
+   Original Request Identifier: 32 bits.
+
+      The Original Request Identifier is set to the value specified in
+      the Request Identifier field that was originally provided in the
+      corresponding Atomic Operation Request Message.
+
+   Original Remote Data Value: 64 bits.
+
+      The Original Remote Value specifies the original 64-bit value
+      stored at the ULP Buffer address targeted by the Atomic Operation.
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 17]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   The Atomic Operation Response Message has the following semantics:
+
+   1. The Atomic Operation Response Message for the associated Atomic
+      Operation Request Message travels in the opposite direction.
+
+   2. An Atomic Operation Response Message MUST consume an Untagged
+      Buffer.  That is, the Responder RDMAP layer MUST request that the
+      DDP mark the Message as Untagged.
+
+   3. An Atomic Operation Response Message MUST reference the Queue
+      Number 3.  That is, the Responder's RDMAP layer MUST request that
+      the DDP layer set the Queue Number field to 3.
+
+   4. The Responder MUST ensure that a sufficient number of Untagged
+      Buffers are available on the RDMA Read Request Queue (Queue with
+      DDP Queue Number 1) to support the maximum number of Atomic
+      Operation Requests negotiated by the ULP in addition to the
+      maximum number of RDMA Read Requests negotiated by the ULP.
+
+   5. The Requester MUST ensure that a sufficient number of Untagged
+      Buffers are available on the RDMA Atomic Response Queue (Queue
+      with DDP Queue Number 3) to support the maximum number of Atomic
+      Operation Requests negotiated by the ULP.
+
+   6. The RDMAP layer MUST Deliver the Atomic Operation Response Message
+      to the ULP.
+
+   7. At the Requester, when an invalid Atomic Operation Response
+      Message is delivered to the Remote Peer's RDMAP layer, an error is
+      surfaced.
+
+   8. When the Responder receives Atomic Operation Request messages, the
+      Responder RDMAP layer MUST pass Atomic Operation Response Messages
+      to the DDP layer, in the order that the Atomic Operation Request
+      Messages were received by the RDMAP layer, at the Responder.
+
+5.3.  Atomicity Guarantees
+
+   Atomicity of the Read-Modify-Write (RMW) on the Responder's node by
+   the Atomic Operation MUST be assured in the context of concurrent
+   atomic accesses by other RDMAP Streams on the same RNIC.
+
+5.4.  Atomic Operations Ordering and Completion Rules
+
+   In addition to the ordering and completion rules described in RFC
+   5040, the following rules apply to implementations of the Atomic
+   Operations.
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 18]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   1. For an Atomic Operation, the Requester MUST NOT consider the
+      contents of the Tagged Buffer at the Responder to be modified by
+      that specific Atomic Operation until the Atomic Operation Response
+      Message has been Delivered to RDMAP at the Requester.
+
+   2. Atomicity guarantees MUST be provided within the scope of a single
+      RNIC.
+
+      Implementation Note: This requirement for atomicity among
+      operations is limited to the scope of a single RNIC.  Atomicity
+      guarantees are OPTIONAL with respect to access to the Tagged
+      Buffer by any other method than an Atomic Operation via the same
+      RNIC.  Examples of such accesses that may not be atomic with
+      respect to an Atomic Operation include accesses via other RNICs
+      and local processor memory access to the Tagged Buffer.
+
+   3. Atomic Operation Request Messages MUST NOT start processing at the
+      Responder until they have been Delivered to RDMAP by DDP.
+
+   4. Atomic Operation Response Messages MAY be generated at the
+      Responder after subsequent RDMA Write Messages or Send Messages
+      have been Placed or Delivered.
+
+   5. Atomic Operation Response Message processing at the Responder MUST
+      be started only after the Atomic Operation Request Message has
+      been Delivered by the DDP layer (thus, all previous RDMA Messages
+      on that DDP Stream have been Delivered).
+
+   6. Send Messages MAY be Completed at the Responder before prior
+      incoming Atomic Operation Request Messages have completed their
+      response processing.
+
+   7. An Atomic Operation MUST NOT be Completed at the Requester until
+      the DDP layer Delivers the associated incoming Atomic Operation
+      Response Message.
+
+   8. If more than one outstanding Atomic Request Message is supported
+      by both peers, the Atomic Operation Request Messages MUST be
+      processed in the order they were delivered by the DDP layer on the
+      Responder.  Atomic Operation Response Messages MUST be submitted
+      to the DDP layer on the Responder in the order the Atomic
+      Operation Request Messages were Delivered by DDP.
+
+
+
+
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 19]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+6.  Immediate Data
+
+   The Immediate Data operation is typically used in conjunction with an
+   RDMA Write Operation to improve ULP processing efficiency.  The
+   efficiency is gained by causing an RDMA Completion to be generated
+   immediately following the RDMA Write operation.  This RDMA Completion
+   delivers 8 bytes of Immediate Data at the Remote Peer.  The
+   combination of an RDMA Write Message followed by an Immediate Data
+   Operation has the same behavior as the RDMA Write with Immediate Data
+   operation found in InfiniBand.  An Immediate Data operation that is
+   not preceded by an RDMA Write operation causes an RDMA Completion.
+
+6.1.  RDMAP Interactions with ULP for Immediate Data
+
+   For Immediate Data operations, the following are the interactions
+   between the RDMAP Layer and the ULP:
+
+   o  At the Data Source:
+
+      -  The ULP passes to the RDMAP Layer the following:
+
+         *  8 bytes of ULP Immediate Data
+
+      -  When the Immediate Data operation Completes, an indication of
+         the Completion results.
+
+   o  At the Data Sink:
+
+      -  If the Immediate Data operation is Completed successfully, the
+         RDMAP Layer passes the following information to the ULP Layer:
+
+         *  8 bytes of Immediate Data
+
+         *  An Event, if the Data Sink is configured to generate an
+            Event.
+
+      -  If the Immediate Data operation is Completed in error, the Data
+         Sink RDMAP Layer will pass up the corresponding error
+         information to the Data Sink ULP and send a Terminate Message
+         to the Data Source RDMAP Layer.  The Data Source RDMAP Layer
+         will then pass up the Terminate Message to the ULP.
+
+
+
+
+
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 20]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+6.2.  Immediate Data Header Format
+
+   The Immediate Data and Immediate Data with SE Messages carry
+   Immediate Data as shown in Figure 7.  The RDMAP layer passes to the
+   DDP layer an RDMAP Control Field and 8 bytes of Immediate Data.  The
+   first 8 bytes of the data following the DDP header contains the
+   Immediate Data.  See Appendix A.3 for the DDP segment format of an
+   Immediate Data or Immediate Data with SE Message.
+
+    0                   1                   2                   3
+    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                        Immediate Data                         |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+   Figure 7: Immediate Data or Immediate Data with SE Message Header
+
+   Immediate Data: 64 bits.
+      8 bytes of data transferred from the Data Source to an untagged
+      Buffer at the Data Sink.
+
+6.3.  Immediate Data or Immediate Data with SE Message
+
+   The Immediate Data or Immediate Data with SE Message uses the DDP
+   Untagged Buffer Model to transfer Immediate Data from the Data Source
+   to the Data Sink.
+
+   o  An Immediate Data or Immediate Data with SE Message MUST reference
+      an Untagged Buffer.  That is, the Local Peer's RDMAP Layer MUST
+      request that the DDP layer mark the Message as Untagged.
+
+   o  One Immediate Data or Immediate Data with SE Message MUST consume
+      one Untagged Buffer.
+
+   o  At the Remote Peer, the Immediate Data and Immediate Data with SE
+      Messages MUST be Delivered to the Remote Peer's ULP in the order
+      they were sent.
+
+   o  For an Immediate Data or Immediate Data with SE Message, the Local
+      Peer's RDMAP Layer MUST request that the DDP layer set the Queue
+      Number field to zero.
+
+   o  For an Immediate Data or Immediate Data with SE Message, the Local
+      Peer's RDMAP Layer MUST request that the DDP layer transmit 8
+      bytes of data.
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 21]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   o  The Local Peer MUST issue Immediate Data and Immediate Data with
+      SE Messages in the order they were submitted by the ULP.
+
+   o  The Remote Peer MUST check that Immediate Data and Immediate Data
+      with SE Messages include exactly 8 bytes of data from the DDP
+      layer.  The DDP header carries the length field that is reported
+      by the DDP layer.
+
+6.4.  Ordering and Completions
+
+   Ordering and completion rules for Immediate Data are the same as
+   those for a Send operation as described in Section 5.5 of RFC 5040.
+
+7.  Ordering and Completions Table
+
+   The following table summarizes the ordering relationships for Atomic
+   and Immediate Data operations from the standpoint of the Local Peer
+   issuing the Operations.  Note that in the table that follows, Send
+   includes Send, Send with Invalidate, Send with Solicited Event, and
+   Send with Solicited Event and Invalidate.  Also note that in the
+   table below, Immediate Data includes Immediate Data and Immediate
+   Data with Solicited Event.
+
+   ---------+----------+-------------+-------------+------------------
+   First    | Second   | Placement   | Placement   | Ordering
+   Operation| Operation| Guarantee at| Guarantee at| Guarantee at
+            |          | Remote Peer | Local Peer  | Remote Peer
+   ---------+----------+-------------+-------------+------------------
+   Immediate| Send     | No Placement| Not         | Completed in
+   Data     |          | Guarantee   | Applicable  | Order
+            |          | between Send|             |
+            |          | Payload and |             |
+            |          | Immediate   |             |
+            |          | Data        |             |
+   ---------+----------+-------------+-------------+------------------
+   Immediate| RDMA     | No Placement| Not         | Not
+   Data     | Write    | Guarantee   | Applicable  | Applicable
+            |          | between RDMA|             |
+            |          | Write       |             |
+            |          | Payload and |             |
+            |          | Immediate   |             |
+            |          | Data        |             |
+
+
+
+
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 22]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   ---------+----------+-------------+-------------+------------------
+   Immediate| RDMA     | No Placement| RDMA Read   | RDMA Read
+   Data     | Read     | Guarantee   | Response    | Response
+            |          | between     | will not be | Message will
+            |          | Immediate   | Placed until| not be
+            |          | Data and    | Immediate   | generated
+            |          | RDMA Read   | Data is     | until
+            |          | Request     | Placed at   | Immediate Data
+            |          |             | Remote Peer | has been
+            |          |             |             | Completed
+   ---------+----------+-------------+-------------+------------------
+   Immediate| Atomic   | No Placement| Atomic      | Atomic
+   Data     |          | Guarantee   | Response    | Response
+            |          | between     | will not be | Message will
+            |          | Immediate   | Placed until| not be
+            |          | Data and    | Immediate   | generated
+            |          | Atomic      | Data is     | until
+            |          | Request     | Placed at   | Immediate Data
+            |          |             | Remote Peer | has been
+            |          |             |             | Completed
+   ---------+----------+-------------+-------------+------------------
+   Immediate| Immediate| No Placement| Not         | Completed in
+   Data or  | Data     | Guarantee   | Applicable  | Order
+   Send     |          |             |             |
+   ---------+----------+-------------+-------------+------------------
+   RDMA     | Immediate| No Placement| Not         | Immediate Data
+   Write    | Data     | Guarantee   | Applicable  | is Completed
+            |          |             |             | after RDMA
+            |          |             |             | Write is Placed
+            |          |             |             | and Delivered
+   ---------+----------+-------------+-------------+------------------
+   RDMA Read| Immediate| No Placement| Immediate   | Not Applicable
+            | Data     | Guarantee   | Data MAY be |
+            |          | between     | Placed      |
+            |          | Immediate   | before      |
+            |          | Data and    | RDMA Read   |
+            |          | RDMA Read   | Response is |
+            |          | Request     | generated   |
+   ---------+----------+-------------+-------------+------------------
+   Atomic   | Immediate| No Placement| Immediate   | Not Applicable
+            | Data     | Guarantee   | Data MAY be |
+            |          | between     | Placed      |
+            |          | Immediate   | before      |
+            |          | Data and    | Atomic      |
+            |          | Atomic      | Response is |
+            |          | Request     | generated   |
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 23]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   ---------+----------+-------------+-------------+------------------
+   Atomic   | Send     | No Placement| Send Payload| Not Applicable
+            |          | Guarantee   | MAY be      |
+            |          | between Send| Placed      |
+            |          | Payload and | before      |
+            |          | Atomic      | Atomic      |
+            |          | Request     | Response is |
+            |          |             | generated   |
+   ---------+----------+-------------+-------------+------------------
+   Atomic   | RDMA     | No Placement| RDMA Write  | Not
+            | Write    | Guarantee   | Payload MAY | Applicable
+            |          | between RDMA| be Placed   |
+            |          | Write       | before      |
+            |          | Payload and | Atomic      |
+            |          | Atomic      | Response is |
+            |          | Request     | generated   |
+   ---------+----------+-------------+-------------+------------------
+   Atomic   | RDMA     | No Placement| No Placement| RDMA Read
+            | Read     | Guarantee   | Guarantee   | Response
+            |          | between     | between     | Message will
+            |          | Atomic      | Atomic      | not be
+            |          | Request and | Response    | generated
+            |          | RDMA Read   | and RDMA    | until Atomic
+            |          | Request     | Read        | Response Message
+            |          |             | Response    | has been
+            |          |             |             | generated
+   ---------+----------+-------------+-------------+------------------
+   Atomic   | Atomic   | Placed in   | No Placement| Second Atomic
+            |          | order       | Guarantee   | Request
+            |          |             | between two | Message will
+            |          |             | Atomic      | not be
+            |          |             | Responses   | processed
+            |          |             |             | until first
+            |          |             |             | Atomic Response
+            |          |             |             | has been
+            |          |             |             | generated
+   ---------+----------+-------------+-------------+------------------
+   Send     | Atomic   | No Placement| Atomic      | Atomic Response
+            |          | Guarantee   | Response    | Message will not
+            |          | between Send| will not be | be generated
+            |          | Payload and | Placed at   | until Send has
+            |          | Atomic      | the Local   | been Completed
+            |          | Request     | Peer until  |
+            |          |             | Send Payload|
+            |          |             | is Placed   |
+            |          |             | at the      |
+            |          |             | Remote Peer |
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 24]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   ---------+----------+-------------+-------------+------------------
+   RDMA     | Atomic   | No Placement| Atomic      | Not
+   Write    |          | Guarantee   | Response    | Applicable
+            |          | between RDMA| will not be |
+            |          | Write       | Placed at   |
+            |          | Payload and | the Local   |
+            |          | Atomic      | Peer until  |
+            |          | Request     | RDMA Write  |
+            |          |             | Payload     |
+            |          |             | is Placed   |
+            |          |             | at the      |
+            |          |             | Remote Peer |
+   ---------+----------+-------------+-------------+------------------
+   RDMA     | Atomic   | No Placement| No Placement| Atomic Response
+   Read     |          | Guarantee   | Guarantee   | Message will
+            |          | between     | between     | not be generated
+            |          | Atomic      | Atomic      | until RDMA
+            |          | Request and | Response    | Read Response
+            |          | RDMA Read   | and RDMA    | has been
+            |          | Request     | Read        | generated
+            |          |             | Response    |
+   ---------+----------+-------------+-------------+------------------
+
+8.  Error Processing
+
+   In addition to the error processing described in Section 7 of RFC
+   5040, the following rules apply for the new RDMA Messages defined in
+   this specification.
+
+8.1.  Errors Detected at the Local Peer
+
+   The Local Peer MUST send a Terminate Message for each of the
+   following cases:
+
+   1. For errors detected while creating an Atomic Request, Atomic
+      Response, Immediate Data, or Immediate Data with SE Message, or
+      other reasons not directly associated with an incoming Message,
+      the Terminate Message and Error code are sent instead of the
+      Message.  In this case, the Error Type and Error Code fields are
+      included in the Terminate Message, but the Terminated DDP Header
+      and Terminated RDMA Header fields are set to zero.
+
+   2. For errors detected on an incoming Atomic Request, Atomic
+      Response, Immediate Data, or Immediate Data with SE (after the
+      Message has been Delivered by DDP), the Terminate Message is sent
+      at the earliest possible opportunity, preferably in the next
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 25]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+      outgoing RDMA Message.  In this case, the Error Type, Error Code,
+      and Terminated DDP Header fields are included in the Terminate
+      Message, but the Terminated RDMA Header field is set to zero.
+
+8.2.  Errors Detected at the Remote Peer
+
+   On incoming Atomic Requests, Atomic Responses, Immediate Data, and
+   Immediate Data with Solicited Event, the following MUST be validated:
+
+   o  The DDP layer MUST validate all DDP Segment fields.
+
+   o  The RDMA OpCode MUST be valid.
+
+   o  The RDMA Version MUST be valid.
+
+   On incoming Atomic requests the following additional validation MUST
+   be performed:
+
+   o  The RDMAP layer MUST validate that the Remote Peer's Tagged ULP
+      Buffer address references a ULP Buffer address that is 64-bit
+      aligned.  In the case of an error, the RDMAP layer MUST generate a
+      Terminate Message indicating RDMA Layer Remote Operation Error
+      with Error Code Name "Catastrophic error, localized to RDMAP
+      Stream" as described in Section 4.8 of RFC 5040.  Implementation
+      Note: A ULP implementation can avoid this error by having the
+      target ULP Buffer of an Atomic Operation 64-bit aligned.
+
+9.  Security Considerations
+
+   This document specifies extensions to the RDMA Protocol specification
+   in RFC 5040, and as such the Security Considerations discussed in
+   Section 8 of RFC 5040 apply.  In particular, Atomic Operations use
+   ULP Buffer addresses for the Remote Peer Buffer addressing used in
+   RFC 5040 as required by the security model described in RFC 5042
+   [RFC5042].
+
+   RDMAP and related protocols may be used by applications that exhibit
+   distinctive traffic characteristics such as message timing, source,
+   destination, and size patterns.  Examples include structured high-
+   performance computing applications based on the MPI interface.  For
+   such applications, analysis of encrypted traffic could reveal
+   sensitive information, e.g., the nature of the application, size of
+   data set being used, and information about the application's rate of
+   progress.  Such information can be hidden from passive observation
+   via use of Encapsulating Security Payload version 3 (ESPv3) Traffic
+   Flow Confidentiality [RFC4303] to obfuscate the encrypted traffic's
+   characteristics.  ESPv3 implementation requirements for RDMAP are
+   specified in [RFC7146].
+
+
+
+Shah, et al.                 Standards Track                   [Page 26]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+10.  IANA Considerations
+
+   IANA has added the following entries to the "RDMAP Message Operation
+   Codes" registry of "Remote Direct Data Placement (RDDP)" registry:
+
+   0x8, Immediate Data, this specification
+
+   0x9, Immediate Data with Solicited Event, this specification
+
+   0xA, Atomic Request, this specification
+
+   0xB, Atomic Response, this specification
+
+   In addition, the following registry has been added to the "Remote
+   Direct Data Placement (RDDP)" registry.  The following section
+   specifies the registry, its initial contents, and the administration
+   policy in more detail.
+
+10.1.  RDMAP Message Atomic Operation Subcodes
+
+   Name of the registry: "RDMAP Message Atomic Operation Subcodes"
+
+   Namespace details: RDMAP Message Atomic Operation Subcodes are 4-bit
+   values.
+
+   Information that must be provided to assign a new value: An IESG-
+   approved Standards Track specification defining the semantics and
+   interoperability requirements of the proposed new value and the
+   fields to be recorded in the registry.
+
+   Fields to record in the registry: RDMAP Message Atomic Operation
+   Subcode, Atomic Operation, RFC Reference.
+
+   Initial registry contents:
+
+   0x0, FetchAdd, this specification
+
+   0x1, Reserved, this specification
+
+   0x2, CmpSwap, this specification
+
+   Note: An experimental RDMAP Message Operation Code has already been
+   allocated; hence, there is no need for an experimental RDMAP Message
+   Atomic Operation Subcode.
+
+
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 27]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   All other values are Unassigned and available to IANA for assignment.
+   New RDMAP Message Atomic Operation Subcodes should be assigned
+   sequentially in order to better support implementations that process
+   RDMAP Message Atomic Operations in hardware.
+
+   Allocation Policy: Standards Action [RFC5226]
+
+10.2.  RDMAP Queue Numbers
+
+   Name of the registry: "RDMAP DDP Untagged Queue Numbers"
+
+   Namespace details: RDMAP DDP Untagged Queue numbers are 32-bit
+   values.
+
+   Information that must be provided to assign a new value: An IESG-
+   approved Standards Track specification defining the semantics and
+   interoperability requirements of the proposed new value and the
+   fields to be recorded in the registry.
+
+   Fields to record in the registry: RDMAP DDP Untagged Queue Numbers,
+   Queue Usage Description, RFC Reference.
+
+   Initial registry contents:
+
+   0x00000000, Queue 0 (Send operation Variants), [RFC5040]
+
+   0x00000001, Queue 1 (RDMA Read Request operations), [RFC5040]
+
+   0x00000002, Queue 2 (Terminate operations), [RFC5040]
+
+   0x00000003, Queue 3 (Atomic Response operations), this specification
+
+   Note: An experimental RDMAP Message Operation Code has already been
+   allocated; hence, there is no need for an experimental RDMAP DDP
+   Untagged Queue Number.
+
+   All other values are Unassigned and available to IANA for assignment.
+   New RDMAP queue numbers should be assigned sequentially in order to
+   better support implementations that perform RDMAP queue selection in
+   hardware.
+
+   Allocation Policy: Standards Action [RFC5226]
+
+
+
+
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 28]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+11.  References
+
+11.1.  Normative References
+
+   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
+              Requirement Levels", BCP 14, RFC 2119, March 1997.
+
+   [RFC4303]  Kent, S., "IP Encapsulating Security Payload (ESP)", RFC
+              4303, December 2005.
+
+   [RFC5040]  Recio, R., Metzler, B., Culley, P., Hilland, J., and D.
+              Garcia, "A Remote Direct Memory Access Protocol
+              Specification", RFC 5040, October 2007.
+
+   [RFC5041]  Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
+              Data Placement over Reliable Transports", RFC 5041,
+              October 2007.
+
+   [RFC5042]  Pinkerton, J. and E. Deleganes, "Direct Data Placement
+              Protocol (DDP) / Remote Direct Memory Access Protocol
+              (RDMAP) Security", RFC 5042, October 2007.
+
+   [RFC5226]  Narten, T. and H. Alvestrand, "Guidelines for Writing an
+              IANA Considerations Section in RFCs", BCP 26, RFC 5226,
+              May 2008.
+
+   [RFC7146]  Black, D. and P. Koning, "Securing Block Storage Protocols
+              over IP: RFC 3723 Requirements Update for IPsec v3", RFC
+              7146, April 2014.
+
+11.2.  Informative References
+
+   [DAT_ATOMICS]
+              DAT Collaborative, "IB Transport Specific Extensions for
+              DAT 2.0", User Direct Access Programming Library,
+              <http://www.datcollaborative.org/DAT_IB_Extensions.pdf>.
+
+   [IB]       InfiniBand Trade Association, "InfiniBand Architecture
+              Specification Volumes 1 and 2", Release 1.1, November
+              2002, <http://www.infinibandta.org/specs>.
+
+   [MPI]      Message Passing Interface Forum, "MPI: A Message-Passing
+              Interface Standard, Version 3.0", September 2012,
+              <http://www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf>.
+
+
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 29]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+   [OFAVERBS] Rosenstock, H., "Subject: Re: [PATCH 0/2] Add support for
+              enhanced atomic operations", message to the linux-rdma
+              mailing list,
+              <http://www.spinics.net/lists/linux-rdma/msg02405.html>.
+
+   [RFC5044]  Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
+              Carrier, "Marker PDU Aligned Framing for TCP
+              Specification", RFC 5044, October 2007.
+
+   [RFC5045]  Bestler, C., Ed., and L. Coene, "Applicability of Remote
+              Direct Memory Access Protocol (RDMA) and Direct Data
+              Placement (DDP)", RFC 5045, October 2007.
+
+   [RFC6581]  Kanevsky, A., Ed., Bestler, C., Ed., Sharp, R., and S.
+              Wise, "Enhanced Remote Direct Memory Access (RDMA)
+              Connection Establishment", RFC 6581, April 2012.
+
+   [RSOCKETS] Hefty, S., "RDMA CM - RDMA enabled Sockets library for
+              Open Fabrics", <http://git.openfabrics.org/?p=~shefty/
+              librdmacm.git;a=summary>.
+
+12.  Acknowledgments
+
+   The authors would like to acknowledge the following individuals who
+   provided valuable comments and suggestions.
+
+   o  David Black
+
+   o  Arkady Kanevsky
+
+   o  Bernard Metzler
+
+   o  Jim Pinkerton
+
+   o  Tom Talpey
+
+   o  Steve Wise
+
+   o  Don Wood
+
+
+
+
+
+
+
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 30]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+Appendix A.  DDP Segment Formats for RDMA Messages
+
+   This appendix is for information only and is NOT part of the
+   standard.  It simply depicts the DDP Segment format for the various
+   RDMA Messages.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 31]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+A.1.  DDP Segment for Atomic Operation Request
+
+   The following figure depicts an Atomic Operation Request, DDP
+   Segment:
+
+    0                   1                   2                   3
+    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+                                   |  DDP Control  | RDMA Control  |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                      Reserved (Not Used)                      |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |              DDP (Atomic Operation Request) Queue Number      |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |        DDP (Atomic Operation Request) Message Sequence Number |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |             DDP (Atomic Operation Request) Message Offset     |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                      Reserved (Not Used)              |AOpCode|
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                       Request Identifier                      |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                          Remote STag                          |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                      Remote Tagged Offset                     |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                       Add or Swap Data                        |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                       Add or Swap Mask                        |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                         Compare Data                          |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                         Compare Mask                          |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 32]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+A.2.  DDP Segment for Atomic Response
+
+   The following figure depicts an Atomic Operation Response, DDP
+   Segment:
+
+    0                   1                   2                   3
+    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+                                   |  DDP Control  | RDMA Control  |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                      Reserved (Not Used)                      |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |              DDP (Atomic Operation Request) Queue Number      |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |        DDP (Atomic Operation Request) Message Sequence Number |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |             DDP (Atomic Operation Request) Message Offset     |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                    Original Request Identifier                |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                       Original Remote Value                   |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+A.3.  DDP Segment for Immediate Data and Immediate Data with SE
+
+   The following figure depicts an Immediate Data or Immediate Data with
+   SE, DDP Segment:
+
+    0                   1                   2                   3
+    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+                                   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+                                   |  DDP Control  | RDMA Control  |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                      Reserved (Not Used)                      |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                    DDP (Send) Queue Number                    |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                DDP (Send) Message Sequence Number             |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                       DDP Message Offset                      |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+   |                        Immediate Data                         |
+   +                                                               +
+   |                                                               |
+   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 33]
+
+RFC 7306                RDMA Protocol Extensions               June 2014
+
+
+Authors' Addresses
+
+   Hemal Shah
+   Broadcom Corporation
+   5300 California Avenue
+   Irvine, CA 92617
+   US
+   Phone: 1-949-926-6941
+   EMail: hemal@broadcom.com
+
+
+   Felix Marti
+   Chelsio Communications, Inc.
+   370 San Aleso Ave.
+   Sunnyvale, CA 94085
+   US
+   Phone: 1-408-962-3600
+   EMail: felix@chelsio.com
+
+
+   Asgeir Eiriksson
+   Chelsio Communications, Inc.
+   370 San Aleso Ave.
+   Sunnyvale, CA 94085
+   US
+   Phone: 1-408-962-3600
+   EMail: asgeir@chelsio.com
+
+
+   Wael Noureddine
+   Chelsio Communications, Inc.
+   370 San Aleso Ave.
+   Sunnyvale, CA 94085
+   US
+   Phone: 1-408-962-3600
+   EMail: wael@chelsio.com
+
+
+   Robert Sharp
+   Intel Corporation
+   1300 South Mopac Expy, Mailstop: AN4-4B
+   Austin, TX  78746
+   US
+   Phone: 1-512-362-1407
+   EMail: robert.o.sharp@intel.com
+
+
+
+
+
+
+Shah, et al.                 Standards Track                   [Page 34]
+