doc: Add RFC documents

1 files changed, 4763 insertions, 0 deletions

diff --git a/doc/rfc/rfc5046.txt b/doc/rfc/rfc5046.txt
new file mode 100644
index 0000000..07af160
--- /dev/null
+++ b/doc/rfc/rfc5046.txt
@@ -0,0 +1,4763 @@
+
+
+
+
+
+
+Network Working Group                                              M. Ko
+Request for Comments: 5046                               IBM Corporation
+Category: Standards Track                                 M. Chadalapaka
+                                                 Hewlett-Packard Company
+                                                              J. Hufferd
+                                                           Brocade, Inc.
+                                                                U. Elzur
+                                                                 H. Shah
+                                                               P. Thaler
+                                                    Broadcom Corporation
+                                                            October 2007
+
+
+      Internet Small Computer System Interface (iSCSI) Extensions
+                 for Remote Direct Memory Access (RDMA)
+
+Status of This Memo
+
+   This document specifies an Internet standards track protocol for the
+   Internet community, and requests discussion and suggestions for
+   improvements.  Please refer to the current edition of the "Internet
+   Official Protocol Standards" (STD 1) for the standardization state
+   and status of this protocol.  Distribution of this memo is unlimited.
+
+Abstract
+
+   Internet Small Computer System Interface (iSCSI) Extensions for
+   Remote Direct Memory Access (RDMA) provides the RDMA data transfer
+   capability to iSCSI by layering iSCSI on top of an RDMA-Capable
+   Protocol, such as the iWARP protocol suite.  An RDMA-Capable Protocol
+   provides RDMA Read and Write services, which enable data to be
+   transferred directly into SCSI I/O Buffers without intermediate data
+   copies.  This document describes the extensions to the iSCSI protocol
+   to support RDMA services as provided by an RDMA-Capable Protocol,
+   such as the iWARP protocol suite.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                     [Page 1]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+Table of Contents
+
+   1. Introduction ....................................................5
+      1.1. Motivation .................................................5
+      1.2. Architectural Goals ........................................6
+      1.3. Protocol Overview ..........................................7
+      1.4. RDMA Services and iSER .....................................8
+           1.4.1. STag ................................................8
+           1.4.2. Send ................................................9
+           1.4.3. RDMA Write ..........................................9
+           1.4.4. RDMA Read ...........................................9
+      1.5. SCSI Read Overview ........................................10
+      1.6. SCSI Write Overview .......................................10
+      1.7. iSCSI/iSER Layering .......................................10
+   2. Definitions and Acronyms .......................................11
+      2.1. Definitions ...............................................11
+      2.2. Acronyms ..................................................17
+      2.3. Conventions ...............................................19
+   3. Upper Layer Interface Requirements .............................19
+      3.1. Operational Primitives Offered by iSER ....................20
+           3.1.1. Send_Control .......................................20
+           3.1.2. Put_Data ...........................................20
+           3.1.3. Get_Data ...........................................21
+           3.1.4. Allocate_Connection_Resources ......................21
+           3.1.5. Deallocate_Connection_Resources ....................22
+           3.1.6. Enable_Datamover ...................................22
+           3.1.7. Connection_Terminate ...............................22
+           3.1.8. Notice_Key_Values ..................................23
+           3.1.9. Deallocate_Task_Resources ..........................23
+      3.2. Operational Primitives Used by iSER .......................23
+           3.2.1. Control_Notify .....................................24
+           3.2.2. Data_Completion_Notify .............................24
+           3.2.3. Data_ACK_Notify ....................................24
+           3.2.4. Connection_Terminate_Notify ........................25
+      3.3. iSCSI Protocol Usage Requirements .........................25
+   4. Lower Layer Interface Requirements .............................26
+      4.1. Interactions with the RCaP Layer ..........................26
+      4.2. Interactions with the Transport Layer .....................27
+   5. Connection Setup and Termination ...............................27
+      5.1. iSCSI/iSER Connection Setup ...............................27
+           5.1.1. Initiator Behavior .................................29
+           5.1.2. Target Behavior ....................................30
+           5.1.3. iSER Hello Exchange ................................32
+      5.2. iSCSI/iSER Connection Termination .........................33
+           5.2.1. Normal Connection Termination at the Initiator .....33
+           5.2.2. Normal Connection Termination at the Target ........34
+           5.2.3. Termination without Logout Request/Response PDUs ...34
+
+
+
+
+Ko, et al.                  Standards Track                     [Page 2]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   6. Login/Text Operational Keys ....................................35
+      6.1. HeaderDigest and DataDigest ...............................35
+      6.2. MaxRecvDataSegmentLength ..................................36
+      6.3. RDMAExtensions ............................................36
+      6.4. TargetRecvDataSegmentLength ...............................37
+      6.5. InitiatorRecvDataSegmentLength ............................38
+      6.6. OFMarker and IFMarker .....................................38
+      6.7. MaxOutstandingUnexpectedPDUs ..............................38
+   7. iSCSI PDU Considerations .......................................39
+      7.1. iSCSI Data-Type PDU .......................................39
+      7.2. iSCSI Control-Type PDU ....................................40
+      7.3. iSCSI PDUs ................................................40
+           7.3.1. SCSI Command .......................................40
+           7.3.2. SCSI Response ......................................42
+           7.3.3. Task Management Function Request/Response ..........44
+           7.3.4. SCSI Data-Out ......................................45
+           7.3.5. SCSI Data-In .......................................46
+           7.3.6. Ready to Transfer (R2T) ............................48
+           7.3.7. Asynchronous Message ...............................50
+           7.3.8. Text Request and Text Response .....................50
+           7.3.9. Login Request and Login Response ...................50
+           7.3.10. Logout Request and Logout Response ................51
+           7.3.11. SNACK Request .....................................51
+           7.3.12. Reject ............................................51
+           7.3.13. NOP-Out and NOP-In ................................51
+   8. Flow Control and STag Management ...............................52
+      8.1. Flow Control for RDMA Send Message Types ..................52
+           8.1.1. Flow Control for Control-Type PDUs from the
+                  Initiator ..........................................52
+           8.1.2. Flow Control for Control-Type PDUs from the
+                  Target .............................................55
+      8.2. Flow Control for RDMA Read Resources ......................56
+      8.3. STag Management ...........................................56
+           8.3.1. Allocation of STags ................................57
+           8.3.2. Invalidation of STags ..............................57
+   9. iSER Control and Data Transfer .................................58
+      9.1. iSER Header Format ........................................58
+      9.2. iSER Header Format for the iSCSI Control-Type PDU .........59
+      9.3. iSER Header Format for the iSER Hello Message .............60
+      9.4. iSER Header Format for the iSER HelloReply Message ........61
+      9.5. SCSI Data Transfer Operations .............................62
+           9.5.1. SCSI Write Operation ...............................62
+           9.5.2. SCSI Read Operation ................................63
+           9.5.3. Bidirectional Operation ............................64
+   10. iSER Error Handling and Recovery ..............................64
+      10.1. Error Handling ...........................................64
+           10.1.1. Errors in the Transport Layer .....................64
+           10.1.2. Errors in the RCaP Layer ..........................65
+
+
+
+Ko, et al.                  Standards Track                     [Page 3]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+           10.1.3. Errors in the iSER Layer ..........................66
+           10.1.4. Errors in the iSCSI Layer .........................67
+      10.2. Error Recovery ...........................................69
+           10.2.1. PDU Recovery ......................................69
+           10.2.2. Connection Recovery ...............................70
+   11. Security Considerations .......................................71
+   12. References ....................................................71
+      12.1. Normative References .....................................71
+      12.2. Informative References ...................................72
+   Appendix A. iWARP Message Format for iSER .........................73
+      A.1. iWARP Message Format for iSER Hello Message ...............73
+      A.2. iWARP Message Format for iSER HelloReply Message ..........74
+      A.3. iWARP Message Format for SCSI Read Command PDU ............75
+      A.4. iWARP Message Format for SCSI Read Data ...................76
+      A.5. iWARP Message Format for SCSI Write Command PDU ...........77
+      A.6. iWARP Message Format for RDMA Read Request ................78
+      A.7. iWARP Message Format for Solicited SCSI Write Data ........79
+      A.8. iWARP Message Format for SCSI Response PDU ................80
+   Appendix B. Architectural Discussion of iSER over InfiniBand ......81
+      B.1. The Host Side of the iSCSI and iSER Connections
+           in InfiniBand .............................................81
+      B.2. The Storage Side of the iSCSI and iSER Mixed
+           Network Environment .......................................82
+      B.3. Discovery Processes for an InfiniBand Host ................82
+      B.4. IBTA Connection Specifications ............................83
+   Acknowledgments ...................................................83
+
+Table of Figures
+
+   Figure 1. Example of iSCSI/iSER Layering in Full Feature Phase ....11
+   Figure 2. iSER Header Format ......................................58
+   Figure 3. iSER Header Format for iSCSI Control-Type PDU ...........59
+   Figure 4. iSER Header Format for iSER Hello Message ...............60
+   Figure 5. iSER Header Format for iSER HelloReply Message ..........61
+   Figure 6. SendSE Message containing an iSER Hello Message .........72
+   Figure 7. SendSE Message containing an iSER HelloReply Message ....74
+   Figure 8. SendSE Message containing a SCSI Read Command PDU .......75
+   Figure 9. RDMA Write Message containing SCSI Read Data ............76
+   Figure 10. SendSE Message containing a SCSI Write Command PDU .....77
+   Figure 11. RDMA Read Request Message ..............................78
+   Figure 12. RDMA Read Response Message containing SCSI Write Data ..79
+   Figure 13. SendInvSE Message containing SCSI Response PDU .........80
+   Figure 14. iSCSI and iSER on IB ...................................81
+   Figure 15. Storage Controller with TCP, iWARP, and IB Connections .82
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                     [Page 4]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+1.  Introduction
+
+1.1.  Motivation
+
+   The iSCSI protocol [RFC3720] is a mapping of the SCSI Architecture
+   Model (see [SAM2]) over the TCP protocol.  SCSI commands are carried
+   by iSCSI requests, and SCSI responses and status are carried by iSCSI
+   responses.  Other iSCSI protocol exchanges and SCSI data are also
+   transported in iSCSI Protocol Data Units (PDUs).
+
+   Out-of-order TCP segments in the Traditional iSCSI model have to be
+   stored and reassembled before the iSCSI protocol layer within an end
+   node can place the data in the iSCSI buffers.  This reassembly is
+   required because not every TCP segment is likely to contain an iSCSI
+   header to enable its placement, and TCP itself does not have a
+   built-in mechanism for signaling Upper Level Protocol (ULP) message
+   boundaries to aid placement of out-of-order segments.  This TCP
+   reassembly at high network speeds is quite counter-productive for the
+   following reasons: wasted memory bandwidth in data copying, the need
+   for reassembly memory, wasted CPU cycles in data copying, and the
+   general store-and-forward latency from an application perspective.
+   TCP reassembly was recognized as a serious issue in [RFC3720], and
+   the notion of a "sync and steering layer" was introduced that is
+   optional to implement and use.  One specific sync and steering
+   mechanism, called "markers", was defined in [RFC3720], which provides
+   an application-level way of framing iSCSI Protocol Data Units (PDUs)
+   within the TCP data stream even when the TCP segments are not yet
+   reassembled to be in-order.
+
+   With these defined techniques in [RFC3720], a Network Interface
+   Controller customized for iSCSI (SNIC) could offload the TCP/IP
+   processing and support direct data placement, but most iSCSI
+   implementations do not support iSCSI "markers", making SNIC marker-
+   based direct data placement unusable in practice.
+
+   The iWARP protocol stack provides direct data placement functionality
+   that is usable in practice.  In addition, there is interest in using
+   iSCSI with other Remote Direct Memory Access (RDMA) protocol stacks
+   that support direct data placement, such as the one provided by
+   InfiniBand.  The generic term RDMA-Capable Protocol (RCaP) is used to
+   refer to the RDMA functionality provided by such protocol stacks.
+
+   With the availability of RDMA-Capable Controllers within a host
+   system, which does not have SNICs, it is appropriate for iSCSI to be
+   able to exploit the direct data placement function of the RDMA-
+   Capable Controller like other applications.
+
+
+
+
+
+Ko, et al.                  Standards Track                     [Page 5]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   iSCSI Extensions for RDMA (iSER) is designed precisely to take
+   advantage of generic RDMA technologies -- iSER's goal is to permit
+   iSCSI to employ direct data placement and RDMA capabilities using a
+   generic RDMA-Capable Controller.  In summary, the iSCSI/iSER protocol
+   stack is designed to enable scaling to high speeds by relying on a
+   generic data placement process and RDMA technologies and products,
+   which enable direct data placement of both in-order and out-of-order
+   data.
+
+   This document describes iSER as a protocol extension to iSCSI, both
+   for convenience of description and because it is true in a very
+   strict protocol sense.  However, note that iSER is in reality
+   extending the connectivity of the iSCSI protocol defined in
+   [RFC3720], and the name iSER reflects this reality.
+
+   When the iSCSI protocol as defined in [RFC3720] (i.e., without the
+   iSER enhancements) is intended in the rest of the document, the term
+   "Traditional iSCSI" is used to make the intention clear.
+
+1.2.  Architectural Goals
+
+   This section summarizes the architectural goals that guided the
+   design of iSER.
+
+   1. Provide an RDMA data transfer model for iSCSI that enables direct
+      in-order or out-of-order data placement of SCSI data into pre-
+      allocated SCSI buffers while maintaining in-order data delivery.
+
+   2. Not require any major changes to the SCSI Architecture Model
+      [SAM2] and SCSI command set standards.
+
+   3. Utilize existing iSCSI infrastructure (sometimes referred to as
+      "iSCSI ecosystem") including but not limited to MIB,
+      bootstrapping, negotiation, naming and discovery, and security.
+
+   4. Require a session to operate in the Traditional iSCSI data
+      transfer mode if iSER is not supported by either the initiator or
+      the target (i.e., not require iSCSI Full Feature Phase
+      interoperability between an end node operating in Traditional
+      iSCSI mode, and an end node operating in iSER-assisted mode).
+
+   5. Allow initiator and target implementations to utilize generic
+      RDMA-Capable Controllers such as RDMA-enabled Network Interface
+      Controllers (RNICs), or to implement iSCSI and iSER in software
+      (not require iSCSI- or iSER-specific assists in the RCaP
+      implementation or RDMA-Capable Controller).
+
+
+
+
+
+Ko, et al.                  Standards Track                     [Page 6]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   6. Require full and only generic RCaP functionality at both the
+      initiator and the target.
+
+   7. Implement a lightweight Datamover protocol for iSCSI with minimal
+      state maintenance.
+
+1.3.  Protocol Overview
+
+   Consistent with the architectural goals stated in Section 2.2, the
+   iSER protocol does not require changes in the iSCSI ecosystem or any
+   related SCSI specifications.  The iSER protocol defines the mapping
+   of iSCSI PDUs to RCaP Messages in such a way that it is entirely
+   feasible to realize iSCSI/iSER implementations that are based on
+   generic RDMA-Capable Controllers.  The iSER protocol layer requires
+   minimal state maintenance to assist an iSCSI Full Feature Phase
+   connection, besides being oblivious to the notion of an iSCSI
+   session.  The crucial protocol aspects of iSER may be summarized
+   thus:
+
+   1. iSER-assisted mode is negotiated during the iSCSI login for each
+      session, and an entire iSCSI session can only operate in one mode
+      (i.e., a connection in a session cannot operate in iSER-assisted
+      mode if a different connection of the same session is already in
+      Full Feature Phase in the Traditional iSCSI mode).
+
+   2. Once in iSER-assisted mode, all iSCSI interactions on that
+      connection use RCaP Messages.
+
+   3. A Send Message Type is used for carrying an iSCSI control-type PDU
+      preceded by an iSER header.  See Section 7.2 for more details on
+      iSCSI control-type PDUs.
+
+   4. RDMA Write, RDMA Read Request, and RDMA Read Response Messages are
+      used for carrying control and all data information associated with
+      the iSCSI data-type PDUs.  See Section 7.1 for more details on
+      iSCSI data-type PDUs.
+
+   5. Target drives all data transfer (with the exception of iSCSI
+      unsolicited data) for SCSI writes and SCSI reads, by issuing RDMA
+      Read Requests and RDMA Writes, respectively.
+
+   6. RCaP is responsible for ensuring data integrity.  (For example,
+      iWARP includes a CRC-enhanced framing layer called Marker PDU
+      Aligned Framing for TCP (MPA) on top of TCP; and for InfiniBand,
+      the CRCs are included in the Reliable Connection mode).  For this
+      reason, iSCSI header and data digests are negotiated to "None" for
+      iSCSI/iSER sessions.
+
+
+
+
+Ko, et al.                  Standards Track                     [Page 7]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   7. The iSCSI error recovery hierarchy defined in [RFC3720] is fully
+      supported by iSER.  (However, see Section 7.3.11 on the handling
+      of SNACK Request PDUs.)
+
+   8. iSER requires no changes to iSCSI authentication, security, and
+      text mode negotiation mechanisms.
+
+   Note that Traditional iSCSI implementations may have to be adapted to
+   employ iSER.  It is expected that the adaptation when required is
+   likely to be centered around the upper layer interface requirements
+   of iSER (Section 3).
+
+1.4.  RDMA Services and iSER
+
+   iSER is designed to work with software and/or hardware protocol
+   stacks providing the protocol services defined in RCaP documents such
+   as [RDMAP], [IB], etc.  The following subsections describe the key
+   protocol elements of RCaP services that iSER relies on.
+
+1.4.1.  STag
+
+   A Steering Tag (STag) is the identifier of an I/O Buffer unique to an
+   RDMA-Capable Controller that the iSER layer Advertises to the remote
+   iSCSI/iSER node in order to complete a SCSI I/O.
+
+   In iSER, Advertisement is the act of informing the target by the
+   initiator that an I/O Buffer is available at the initiator for RDMA
+   Read or RDMA Write access by the target.  The initiator Advertises
+   the I/O Buffer by including the STag in the header of an iSER Message
+   containing the SCSI Command PDU to the target.  The base Tagged
+   Offset is not explicitly specified, but the target must always assume
+   it as zero.  The buffer length is as specified in the SCSI Command
+   PDU.
+
+   The iSER layer at the initiator Advertises the STag for the I/O
+   Buffer of each SCSI I/O to the iSER layer at the target in the iSER
+   header of the Send with Solicited Event (SendSE) Message containing
+   the SCSI Command PDU, unless the I/O can be completely satisfied by
+   unsolicited data alone.
+
+   The iSER layer at the target provides the STag for the I/O Buffer
+   that is the Data Sink of an RDMA Read Operation (Section 2.4.4) to
+   the RCaP layer on the initiator node -- i.e., this is completely
+   transparent to the iSER layer at the initiator.
+
+   The iSER protocol is defined so that the Advertised STag is
+   automatically invalidated upon a normal completion of the associated
+   task.  This automatic invalidation is realized via the Send with
+
+
+
+Ko, et al.                  Standards Track                     [Page 8]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   Solicited Event and Invalidate (SendInvSE) Message carrying the SCSI
+   Response PDU.  There are two exceptions to this automatic
+   invalidation -- bidirectional commands, and abnormal completion of a
+   command.  The iSER layer at the initiator is required to explicitly
+   invalidate the STag in these cases, in addition to sanity checking
+   the automatic invalidation even when that does happen.
+
+1.4.2.  Send
+
+   Send is the RDMA Operation that is not addressed to an Advertised
+   buffer by the sending side, and thus uses Untagged buffers on the
+   receiving side.
+
+   The iSER layer at the initiator uses the Send Operation to transmit
+   any iSCSI control-type PDU to the target.  As an example, the
+   initiator uses Send Operations to transfer iSER Messages containing
+   SCSI Command PDUs to the iSER layer at the target.
+
+   An iSER layer at the target uses the Send Operation to transmit any
+   iSCSI control-type PDU to the initiator.  As an example, the target
+   uses Send Operations to transfer iSER Messages containing SCSI
+   Response PDUs to the iSER layer at the initiator.
+
+1.4.3.  RDMA Write
+
+   RDMA Write is the RDMA Operation that is used to place data into an
+   Advertised buffer on the receiving side.  The sending side addresses
+   the Message using an STag and a Tagged Offset that are valid on the
+   Data Sink.
+
+   The iSER layer at the target uses the RDMA Write Operation to
+   transfer the contents of a local I/O Buffer to an Advertised I/O
+   Buffer at the initiator.  The iSER layer at the target uses the RDMA
+   Write to transfer whole or part of the data required to complete a
+   SCSI read command.
+
+   The iSER layer at the initiator does not employ RDMA Writes.
+
+1.4.4.  RDMA Read
+
+   RDMA Read is the RDMA Operation that is used to retrieve data from an
+   Advertised buffer on a remote node.  The sending side of the RDMA
+   Read Request addresses the Message using an STag and a Tagged Offset
+   that are valid on the Data Source in addition to providing a valid
+   local STag and Tagged Offset that identify the Data Sink.
+
+   The iSER layer at the target uses the RDMA Read Operation to transfer
+   the contents of an Advertised I/O Buffer at the initiator to a local
+
+
+
+Ko, et al.                  Standards Track                     [Page 9]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   I/O Buffer at the target.  The iSER layer at the target uses the RDMA
+   Read to fetch whole or part of the data required to complete a SCSI
+   write command.
+
+   The iSER layer at the initiator does not employ RDMA Reads.
+
+1.5.  SCSI Read Overview
+
+   The iSER layer at the initiator receives the SCSI Command PDU from
+   the iSCSI layer.  The iSER layer at the initiator generates an STag
+   for the I/O Buffer of the SCSI Read and Advertises the buffer by
+   including the STag as part of the iSER header for the PDU.  The iSER
+   Message is transferred to the target using a SendSE Message.
+
+   The iSER layer at the target uses one or more RDMA Writes to transfer
+   the data required to complete the SCSI Read.
+
+   The iSER layer at the target uses a SendInvSE Message to transfer the
+   SCSI Response PDU back to the iSER layer at the initiator.  The iSER
+   layer at the initiator notifies the iSCSI layer of the availability
+   of the SCSI Response PDU.
+
+1.6.  SCSI Write Overview
+
+   The iSER layer at the initiator receives the SCSI Command PDU from
+   the iSCSI layer.  If solicited data transfer is involved, the iSER
+   layer at the initiator generates an STag for the I/O Buffer of the
+   SCSI Write and Advertises the buffer by including the STag as part of
+   the iSER header for the PDU.  The iSER Message is transferred to the
+   target using a SendSE Message.
+
+   The iSER layer at the initiator may optionally send one or more non-
+   immediate unsolicited data PDUs to the target using Send Message
+   Types.
+
+   If solicited data transfer is involved, the iSER layer at the target
+   uses one or more RDMA Reads to transfer the data required to complete
+   the SCSI Write.
+
+   The iSER layer at the target uses a SendInvSE Message to transfer the
+   SCSI Response PDU back to the iSER layer at the initiator.  The iSER
+   layer at the initiator notifies the iSCSI layer of the availability
+   of the SCSI Response PDU.
+
+1.7.  iSCSI/iSER Layering
+
+   iSCSI Extensions for RDMA (iSER) is layered between the iSCSI layer
+   and the RCaP layer.  Note that the RCaP layer may be composed of one
+
+
+
+Ko, et al.                  Standards Track                    [Page 10]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   or more distinct protocol layers depending on the specifics of the
+   RCaP.  Figure 1 shows an example of the relationship between SCSI,
+   iSCSI, iSER, and the different RCaP layers.  For TCP, the RCaP is
+   iWARP.  For InfiniBand, the RCaP is the Reliable Connected Transport
+   Service.  Note that the iSCSI layer as described here supports the
+   RDMA Extensions as used in iSER.
+
+                 +-------------------------------------+
+                 |              SCSI                   |
+                 +-------------------------------------+
+                 |              iSCSI                  |
+      DI ------> +-------------------------------------+
+                 |              iSER                   |
+                 +---------+--------------+------------+
+                 |  RDMAP  |              |            |
+                 +---------+  InfiniBand  |            |
+                 |   DDP   |   Reliable   |   Other    |
+                 +---------+  Connected   |   RDMA-    |
+                 |   MPA   |  Transport   |  Capable   |
+                 +---------+   Service    |  Protocol  |
+                 |   TCP   |              |            |
+                 +---------+--------------+------------+
+                 |         |  InfiniBand  |   Other    |
+                 |    IP   |   Network    |  Network   |
+                 |         |    Layer     |   Layer    |
+                 +---------+--------------+------------+
+
+   Figure 1.  Example of iSCSI/iSER Layering in Full Feature Phase
+
+2.  Definitions and Acronyms
+
+2.1.  Definitions
+
+   Advertisement (Advertised, Advertise, Advertisements, Advertises) -
+      The act of informing a remote iSER layer that a local node's
+      buffer is available to it.  A Node makes a buffer available for
+      incoming RDMA Read Request Message or incoming RDMA Write Message
+      access by informing the remote iSER layer of the Tagged Buffer
+      identifiers (STag, TO, and buffer length).  Note that this
+      Advertisement of Tagged Buffer information is the responsibility
+      of the iSER layer on either end and is not defined by the RDMA-
+      Capable Protocol.  A typical method would be for the iSER layer to
+      embed the Tagged Buffer's STag, TO, and buffer length in a Send
+      Message destined for the remote iSER layer.
+
+   Completion (Completed, Complete, Completes) - Completion is defined
+      as the process by the RDMA-Capable Protocol layer to inform the
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 11]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+      iSER layer, that a particular RDMA Operation has performed all
+      functions specified for the RDMA Operation.
+
+   Connection - A connection is a logical circuit between the initiator
+      and the target, e.g., a TCP connection.  Communication between the
+      initiator and the target occurs over one or more connections.  The
+      connections carry control messages, SCSI commands, parameters, and
+      data within iSCSI Protocol Data Units (iSCSI PDUs).
+
+   Connection Handle - An information element that identifies the
+      particular iSCSI connection and is unique for a given iSCSI-iSER
+      pair.  Every invocation of an Operational Primitive is qualified
+      with the Connection Handle.
+
+   Data Sink - The peer receiving a data payload.  Note that the Data
+      Sink can be required to both send and receive RCaP Messages to
+      transfer a data payload.
+
+   Data Source - The peer sending a data payload.  Note that the Data
+      Source can be required to both send and receive RCaP Messages to
+      transfer a data payload.
+
+   Datamover Interface (DI) - The interface between the iSCSI layer and
+      the Datamover layer as described in [DA].
+
+   Datamover Layer - A layer that is directly below the iSCSI layer and
+      above the underlying transport layers.  This layer exposes and
+      uses a set of transport independent Operational Primitives for the
+      communication between the iSCSI layer and itself.  The Datamover
+      layer, operating in conjunction with the transport layers, moves
+      the control and data information on the iSCSI connection.  In this
+      specification, the iSER layer is the Datamover layer.
+
+   Datamover Protocol - A Datamover protocol is the wire-protocol that
+      is defined to realize the Datamover layer functionality.  In this
+      specification, the iSER protocol is the Datamover protocol.
+
+   Event - An indication provided by the RDMA-Capable Protocol layer to
+      the iSER layer to indicate a Completion or other condition
+      requiring immediate attention.
+
+   Inbound RDMA Read Queue Depth (IRD) - The maximum number of incoming
+      outstanding RDMA Read Requests that the RDMA-Capable Controller
+      can handle on a particular RCaP Stream at the Data Source.  For
+      some RDMA-Capable Protocol layers, the term "IRD" may be known by
+      a different name.  For example, for InfiniBand, the equivalent for
+      IRD is the Responder Resources.
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 12]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   Invalidate STag - A mechanism used to prevent the Remote Peer from
+      reusing a previous explicitly Advertised STag, until the iSER
+      layer at the local node makes it available through a subsequent
+      explicit Advertisement.
+
+   I/O Buffer - A buffer that is used in a SCSI Read or Write operation
+      so SCSI data may be sent from or received into that buffer.
+
+   iSCSI - The iSCSI protocol as defined in [RFC3720] is a mapping of
+      the SCSI Architecture Model of SAM-2 over TCP.
+
+   iSCSI control-type PDU - Any iSCSI PDU that is not an iSCSI data-
+      type PDU and also not a SCSI Data-out PDU carrying solicited data
+      is defined as an iSCSI control-type PDU.  Specifically, it is to
+      be noted that SCSI Data-out PDUs for unsolicited data are defined
+      as iSCSI control-type PDUs.
+
+   iSCSI data-type PDU - An iSCSI data-type PDU is defined as an iSCSI
+      PDU that causes data transfer, transparent to the remote iSCSI
+      layer, to take place between the peer iSCSI nodes on a Full
+      Feature Phase iSCSI connection.  An iSCSI data-type PDU, when
+      requested for transmission by the sender iSCSI layer, results in
+      the associated data transfer without the participation of the
+      remote iSCSI layer, i.e. the PDU itself is not delivered as-is to
+      the remote iSCSI layer.  The following iSCSI PDUs constitute the
+      set of iSCSI data-type PDUs - SCSI Data-In PDU and R2T PDU.
+
+   iSCSI Layer - A layer in the protocol stack implementation within an
+      end node that implements the iSCSI protocol and interfaces with
+      the iSER layer via the Datamover Interface.
+
+   iSCSI PDU (iSCSI Protocol Data Unit) - The iSCSI layer at the
+      initiator and the iSCSI layer at the target divide their
+      communications into messages.  The term "iSCSI protocol data unit"
+      (iSCSI PDU) is used for these messages.
+
+   iSCSI/iSER Connection - An iSER-assisted iSCSI connection.
+
+   iSCSI/iSER Session - An iSER-assisted iSCSI session.
+
+   iSCSI-iSER Pair - The iSCSI layer and the underlying iSER layer.
+
+   iSER - iSCSI Extensions for RDMA, the protocol defined in this
+      document.
+
+   iSER-assisted - A term generally used to describe the operation of
+      iSCSI when the iSER functionality is also enabled below the iSCSI
+      layer for the specific iSCSI/iSER connection in question.
+
+
+
+Ko, et al.                  Standards Track                    [Page 13]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   iSER-IRD - This variable represents the maximum number of incoming
+      outstanding RDMA Read Requests that the iSER layer at the
+      initiator declares on a particular RCaP Stream.
+
+   iSER-ORD - This variable represents the maximum number of outstanding
+      RDMA Read Requests that the iSER layer can initiate on a
+      particular RCaP Stream.  This variable is maintained only by the
+      iSER layer at the target.
+
+   iSER Layer - The layer that implements the iSCSI Extensions for RDMA
+      (iSER) protocol.
+
+   iWARP - A suite of wire protocols comprising of [RDMAP], [DDP], and
+      [MPA] when layered above [TCP].  [RDMAP] and [DDP] may be layered
+      above SCTP or other transport protocols.
+
+   Local Mapping - A task state record maintained by the iSER layer that
+      associates the Initiator Task Tag to the local STag(s).  The
+      specifics of the record structure are implementation dependent.
+
+   Local Peer - The implementation of the RDMA-Capable Protocol on the
+      local end of the connection.  Used to refer to the local entity
+      when describing protocol exchanges or other interactions between
+      two Nodes.
+
+   Node - A computing device attached to one or more links of a network.
+      A Node in this context does not refer to a specific application or
+      protocol instantiation running on the computer.  A Node may
+      consist of one or more RDMA-Capable Controllers installed in a
+      host computer.
+
+   Operational Primitive - An Operational Primitive is an abstract
+      functional interface procedure that requests that another layer
+      perform a specific action on the requestor's behalf or notifies
+      the other layer of some event.  The Datamover Interface between an
+      iSCSI layer and a Datamover layer within an iSCSI end node uses a
+      set of Operational Primitives to define the functional interface
+      between the two layers.  Note that not every invocation of an
+      Operational Primitive may elicit a response from the requested
+      layer.  A full discussion of the Operational Primitive types and
+      request-response semantics available to iSCSI and iSER can be
+      found in [DA].
+
+   Outbound RDMA Read Queue Depth (ORD) - The maximum number of
+      outstanding RDMA Read Requests that the RDMA-Capable Controller
+      can initiate on a particular RCaP Stream at the Data Sink.  For
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 14]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+      some RDMA-Capable Protocol layer, the term "ORD" may be known by a
+      different name.  For example, for InfiniBand, the equivalent for
+      ORD is the Initiator Depth.
+
+   Phase-Collapse - Refers to the optimization in iSCSI where the SCSI
+      status is transferred along with the final SCSI Data-in PDU from a
+      target.  See Section 3.2 in [RFC3720].
+
+   RCaP Message - One or more packets of the network layer comprising a
+      single RDMA Operation or a part of an RDMA Read Operation of the
+      RDMA-Capable Protocol.  For iWARP, an RCaP Message is known as an
+      RDMAP Message.
+
+   RCaP Stream - A single bidirectional association between the peer
+      RDMA-Capable Protocol layers on two Nodes over a single
+      transport-level stream.  For iWARP, an RCaP Stream is known as an
+      RDMAP Stream, and the association is created when the connection
+      transitions to iSER-assisted mode following a successful Login
+      Phase during which iSER support is negotiated.
+
+   RDMA-Capable Protocol (RCaP) - The protocol or protocol suite that
+      provides a reliable RDMA transport functionality, e.g., iWARP,
+      InfiniBand, etc.
+
+   RDMA-Capable Controller - A network I/O adapter or embedded
+      controller with RDMA functionality.  For example, for iWARP, this
+      could be an RNIC, and for InfiniBand, this could be a HCA (Host
+      Channel Adapter) or TCA (Target Channel Adapter).
+
+   RDMA-enabled Network Interface Controller (RNIC) - A network I/O
+      adapter or embedded controller with iWARP functionality.
+
+   RDMA Operation - A sequence of RCaP Messages, including control
+      Messages, to transfer data from a Data Source to a Data Sink.  The
+      following RDMA Operations are defined - RDMA Write Operation, RDMA
+      Read Operation, Send Operation, Send with Invalidate Operation,
+      Send with Solicited Event Operation, Send with Solicited Event and
+      Invalidate Operation, and Terminate Operation.
+
+   RDMA Protocol (RDMAP) - A wire protocol that supports RDMA Operations
+      to transfer ULP data between a Local Peer and the Remote Peer as
+      described in [RDMAP].
+
+   RDMA Read Operation - An RDMA Operation used by the Data Sink to
+      transfer the contents of a Data Source buffer from the Remote Peer
+      to a Data Sink buffer at the Local Peer.  An RDMA Read operation
+      consists of a single RDMA Read Request Message and a single RDMA
+      Read Response Message.
+
+
+
+Ko, et al.                  Standards Track                    [Page 15]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   RDMA Read Request - An RCaP Message used by the Data Sink to request
+      that the Data Source transfer the contents of a buffer.  The RDMA
+      Read Request Message describes both the Data Source and the Data
+      Sink buffers.
+
+   RDMA Read Response - An RCaP Message used by the Data Source to
+      transfer the contents of a buffer to the Data Sink, in response to
+      an RDMA Read Request.  The RDMA Read Response Message only
+      describes the Data Sink buffer.
+
+   RDMA Write Operation - An RDMA Operation used by the Data Source to
+      transfer the contents of a Data Source buffer from the Local Peer
+      to a Data Sink buffer at the Remote Peer.  The RDMA Write Message
+      only describes the Data Sink buffer.
+
+   Remote Direct Memory Access (RDMA) - A method of accessing memory on
+      a remote system in which the local system specifies the remote
+      location of the data to be transferred.  Employing an RDMA-
+      Capable Controller in the remote system allows the access to take
+      place without interrupting the processing of the CPU(s) on the
+      system.
+
+   Remote Mapping - A task state record maintained by the iSER layer
+      that associates the Initiator Task Tag to the Advertised STag(s).
+      The specifics of the record structure are implementation
+      dependent.
+
+   Remote Peer - The implementation of the RDMA-Capable Protocol on the
+      opposite end of the connection.  Used to refer to the remote
+      entity when describing protocol exchanges or other interactions
+      between two Nodes.
+
+   SCSI Layer - This layer builds/receives SCSI CDBs (Command Descriptor
+      Blocks) and sends/receives them with the remaining command execute
+      [SAM2] parameters to/from the iSCSI layer.
+
+   Send - An RDMA Operation that transfers the contents of a Buffer from
+      the Local Peer to a Buffer at the Remote Peer.
+
+   Send Message Type - A Send Message, Send with Invalidate Message,
+      Send with Solicited Event Message, or Send with Solicited Event
+      and Invalidate Message.
+
+   SendInvSE Message - A Send with Solicited Event and Invalidate
+      Message.
+
+   SendSE Message - A Send with Solicited Event Message.
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 16]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   Sequence Number (SN) - DataSN for a SCSI Data-in PDU and R2TSN for an
+      R2T PDU.  The semantics for both types of sequence numbers are as
+      defined in [RFC3720].
+
+   Session, iSCSI Session - The group of connections that link an
+      initiator SCSI port with a target SCSI port form an iSCSI session
+      (equivalent to a SCSI I-T nexus).  Connections can be added to and
+      removed from a session even while the I-T nexus is intact.  Across
+      all connections within a session, an initiator sees one and the
+      same target.
+
+   Solicited Event (SE) - A facility by which an RDMA Operation sender
+      may cause an Event to be generated at the recipient, if the
+      recipient is configured to generate such an Event, when a Send
+      with Solicited Event or Send with Solicited Event and Invalidate
+      Message is received.
+
+   Steering Tag (STag) - An identifier of a Tagged Buffer on a Node
+      (Local or Remote) as defined in [RDMAP] and [DDP].  For other
+      RDMA-Capable Protocols, the Steering Tag may be known by different
+      names but will be herein referred to as STags.  For example, for
+      InfiniBand, a Remote STag is known as an R-Key, and a local STag
+      is known as an L-Key, and both will be considered STags.
+
+   Tagged Buffer - A buffer that is explicitly Advertised to the iSER
+      layer at the remote node through the exchange of an STag, Tagged
+      Offset, and length.
+
+   Tagged Offset (TO) - The offset within a Tagged Buffer.
+
+   Traditional iSCSI - Refers to the iSCSI protocol as defined in
+      [RFC3720] (i.e. without the iSER enhancements).
+
+   Untagged Buffer - A buffer that is not explicitly Advertised to the
+      iSER layer at the remode node.
+
+2.2.  Acronyms
+
+      Acronym        Definition
+      --------------------------------------------------------------
+
+      AHS            Additional Header Segment
+
+      BHS            Basic Header Segment
+
+      CO             Connection Only
+
+      CRC            Cyclic Redundancy Check
+
+
+
+Ko, et al.                  Standards Track                    [Page 17]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+      DDP            Direct Data Placement Protocol
+
+      DI             Datamover Interface
+
+      HCA            Host Channel Adapter
+
+      IANA           Internet Assigned Numbers Authority
+
+      IB             InfiniBand
+
+      IETF           Internet Engineering Task Force
+
+      I/O            Input - Output
+
+      IO             Initialize Only
+
+      IP             Internet Protocol
+
+      IPoIB          IP over InfiniBand
+
+      IPsec          Internet Protocol Security
+
+      iSER           iSCSI Extensions for RDMA
+
+      ITT            Initiator Task Tag
+
+      LO             Leading Only
+
+      MPA            Marker PDU Aligned Framing for TCP
+
+      NOP            No Operation
+
+      NSG            Next Stage (during the iSCSI Login Phase)
+
+      OS             Operating System
+
+      PDU            Protocol Data Unit
+
+      R2T            Ready To Transfer
+
+      R2TSN          Ready To Transfer Sequence Number
+
+      RDMA           Remote Direct Memory Access
+
+      RDMAP          Remote Direct Memory Access Protocol
+
+      RFC            Request For Comments
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 18]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+      RNIC           RDMA-enabled Network Interface Controller
+
+      SAM2           SCSI Architecture Model - 2
+
+      SCSI           Small Computer Systems Interface
+
+      SNACK          Selective Negative Acknowledgment - also
+                     Sequence Number Acknowledgement for data
+
+      STag           Steering Tag
+
+      SW             Session Wide
+
+      TCA            Target Channel Adapter
+
+      TCP            Transmission Control Protocol
+
+      TMF            Task Management Function
+
+      TTT            Target Transfer Tag
+
+      TO             Tagged Offset
+
+      ULP            Upper Level Protocol
+
+2.3.  Conventions
+
+   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
+   document are to be interpreted as described in [RFC2119].
+
+3.  Upper Layer Interface Requirements
+
+   This section discusses the upper layer interface requirements in the
+   form of an abstract model of the required interactions between the
+   iSCSI layer and the iSER layer.  The abstract model used here is
+   derived from the architectural model described in [DA].  [DA] also
+   provides a functional overview of the interactions between the iSCSI
+   layer and the Datamover layer as intended by the Datamover
+   Architecture.
+
+   The interface requirements are specified by Operational Primitives.
+   An Operational Primitive is an abstract functional interface
+   procedure between the iSCSI layer and the iSER layer that requests
+   one layer to perform a specific action on behalf of the other layer
+   or notifies the other layer of some event.  Whenever an Operational
+   Primitive in invoked, the Connection_Handle qualifier is used to
+   identify a particular iSCSI connection.  For some Operational
+
+
+
+Ko, et al.                  Standards Track                    [Page 19]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   Primitives, a Data_Descriptor is used to identify the iSCSI/SCSI data
+   buffer associated with the requested or completed operation.
+
+   The abstract model and the Operational Primitives defined in this
+   section facilitate the description of the iSER protocol.  In the rest
+   of the iSER specification, the compliance statements related to the
+   use of these Operational Primitives are only for the purpose of the
+   required interactions between the iSCSI layer and the iSER layer.
+   Note that the compliance statements related to the Operational
+   Primitives in the rest of this specification only mandate functional
+   equivalence on implementations, but do not put any requirements on
+   the implementation specifics of the interface between the iSCSI layer
+   and the iSER layer.
+
+   Each Operational Primitive is invoked with a set of qualifiers that
+   specify the information context for performing the specific action
+   being requested of the Operational Primitive.  While the qualifiers
+   are required, the method of realizing the qualifiers (e.g., by
+   passing synchronously with invocation, or by retrieving from task
+   context, or by retrieving from shared memory, etc.) is implementation
+   dependent.
+
+3.1.  Operational Primitives Offered by iSER
+
+   The iSER protocol layer MUST support the following Operational
+   Primitives to be used by the iSCSI protocol layer.
+
+3.1.1.  Send_Control
+
+      Input qualifiers:  Connection_Handle, BHS and AHS (if any) of the
+      iSCSI PDU, PDU-specific qualifiers
+
+      Return results:  Not specified
+
+   This is used by the iSCSI layers at the initiator and the target to
+   request the outbound transfer of an iSCSI control-type PDU (see
+   Section 7.2).  Qualifiers that only apply for a particular control-
+   type PDU are known as PDU-specific qualifiers, e.g.,
+   ImmediateDataSize for a SCSI write command.  For details on PDU-
+   specific qualifiers, see Section 7.3.  The iSCSI layer can only
+   invoke the Send_Control Operational Primitive when the connection is
+   in iSER-assisted mode.
+
+3.1.2.  Put_Data
+
+      Input qualifiers:  Connection_Handle, content of a SCSI Data-in
+      PDU header, Data_Descriptor, Notify_Enable
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 20]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+      Return results:  Not specified
+
+   This is used by the iSCSI layer at the target to request the outbound
+   transfer of data for a SCSI Data-in PDU from the buffer identified by
+   the Data_Descriptor qualifier.  The iSCSI layer can only invoke the
+   Put_Data Operational Primitive when the connection is in iSER-
+   assisted mode.
+
+   The Notify_Enable qualifier is used to indicate to the iSER layer
+   whether or not it should generate an eventual local completion
+   notification to the iSCSI layer.  See Section 3.2.2 on
+   Data_Completion_Notify for details.
+
+3.1.3.  Get_Data
+
+      Input qualifiers:  Connection_Handle, content of an R2T PDU,
+      Data_Descriptor, Notify_Enable
+
+      Return results:  Not specified
+
+   This is used by the iSCSI layer at the target to request the inbound
+   transfer of solicited data requested by an R2T PDU into the buffer
+   identified by the Data_Descriptor qualifier.  The iSCSI layer can
+   only invoke the Get_Data Operational Primitive when the connection is
+   in iSER-assisted mode.
+
+   The Notify_Enable qualifier is used to indicate to the iSER layer
+   whether or not it should generate the eventual local completion
+   notification to the iSCSI layer.  See Section 3.2.2 on
+   Data_Completion_Notify for details.
+
+3.1.4.  Allocate_Connection_Resources
+
+      Input qualifiers:  Connection_Handle, Resource_Descriptor
+      (optional)
+
+      Return results:  Status
+
+   This is used by the iSCSI layers at the initiator and the target to
+   request the allocation of all connection resources necessary to
+   support RCaP for an operational iSCSI/iSER connection.  The iSCSI
+   layer may optionally specify the implementation-specific resource
+   requirements for the iSCSI connection using the Resource_Descriptor
+   qualifier.
+
+   A return result of Status=success means that the invocation
+   succeeded, and a return result of Status=failure means that the
+   invocation failed.  If the invocation is for a Connection_Handle for
+
+
+
+Ko, et al.                  Standards Track                    [Page 21]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   which an earlier invocation succeeded, the request will be ignored by
+   the iSER layer and the result of Status=success will be returned.
+   Only one Allocate_Connection_Resources Operational Primitive
+   invocation can be outstanding for a given Connection_Handle at any
+   time.
+
+3.1.5.  Deallocate_Connection_Resources
+
+      Input qualifiers:  Connection_Handle
+
+      Return results:  Not specified
+
+   This is used by the iSCSI layers at the initiator and the target to
+   request the deallocation of all connection resources that were
+   allocated earlier as a result of a successful invocation of the
+   Allocate_Connection_Resources Operational Primitive.
+
+3.1.6.  Enable_Datamover
+
+      Input qualifiers:  Connection_Handle,
+      Transport_Connection_Descriptor, Final Login_Response_PDU
+      (optional)
+
+      Return results:  Not specified
+
+   This is used by the iSCSI layers at the initiator and the target to
+   request that a specified iSCSI connection be transitioned to iSER-
+   assisted mode.  The Transport_Connection_Descriptor qualifier is used
+   to identify the specific connection associated with the
+   Connection_Handle.  The iSCSI layer can only invoke the
+   Enable_Datamover Operational Primitive when there is a corresponding
+   prior resource allocation.
+
+   The Final_Login_Response_PDU input qualifier is applicable only for a
+   target, and contains the final Login Response PDU that concludes the
+   iSCSI Login Phase.  If the underlying transport is TCP, the final
+   Login Response PDU must be sent as a byte stream as expected by the
+   iSCSI layer at the initiator.  When this qualifier is used, the iSER
+   layer at the target MUST transmit this final Login Response PDU
+   before transitioning to iSER-assisted mode.
+
+3.1.7.  Connection_Terminate
+
+      Input qualifiers:  Connection_Handle
+
+      Return results:  Not specified
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 22]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   This is used by the iSCSI layers at the initiator and the target to
+   request that a specified iSCSI/iSER connection be terminated and all
+   associated connection and task resources be freed.  When this
+   Operational Primitive invocation returns to the iSCSI layer, the
+   iSCSI layer may assume full ownership of all iSCSI-level resources,
+   e.g., I/O Buffers, associated with the connection.
+
+3.1.8.  Notice_Key_Values
+
+      Input qualifiers:  Connection_Handle, number of keys, list of
+      Key-Value pairs
+
+      Return results:  Not specified
+
+   This is used by the iSCSI layers at the initiator and the target to
+   request that the iSER layer take note of the specified Key-Value
+   pairs that were negotiated by the iSCSI peers for the connection.
+
+3.1.9.  Deallocate_Task_Resources
+
+      Input qualifiers:  Connection_Handle, ITT
+
+      Return results:  Not specified
+
+   This is used by the iSCSI layers at the initiator and the target to
+   request the deallocation of all RCaP-specific resources allocated by
+   the iSER layer for the task identified by the ITT qualifier.  The
+   iSER layer may require a certain number of RCaP-specific resources
+   associated with the ITT for each new iSCSI task.  In the normal
+   course of execution, these task-level resources in the iSER layer are
+   assumed to be transparently allocated on each task initiation and
+   deallocated on the conclusion of each task as appropriate.  In
+   exception scenarios where the task does not conclude with a SCSI
+   Response PDU, the iSER layer needs to be notified of the individual
+   task terminations to aid its task-level resource management.  This
+   Operational Primitive is used for this purpose, and is not needed
+   when a SCSI Response PDU normally concludes a task.  Note that RCaP-
+   specific task resources are deallocated by the iSER layer when a SCSI
+   Response PDU normally concludes a task, even if the SCSI status was
+   not success.
+
+3.2.  Operational Primitives Used by iSER
+
+   The iSER layer MUST use the following Operational Primitives offered
+   by the iSCSI protocol layer when the connection is in iSER-assisted
+   mode.
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 23]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+3.2.1.  Control_Notify
+
+      Input qualifiers:  Connection_Handle, an iSCSI control-type PDU
+
+      Return results:  Not specified
+
+   This is used by the iSER layers at the initiator and the target to
+   notify the iSCSI layer of the availability of an inbound iSCSI
+   control-type PDU.  A PDU is described as "available" to the iSCSI
+   layer when the iSER layer notifies the iSCSI layer of the reception
+   of that inbound PDU, along with an implementation-specific indication
+   as to where the received PDU is.
+
+3.2.2.  Data_Completion_Notify
+
+      Input qualifiers:  Connection_Handle, ITT, SN
+
+      Return results:  Not specified
+
+   This is used by the iSER layer to notify the iSCSI layer of the
+   completion of outbound data transfer that was requested by the iSCSI
+   layer only if the invocation of the Put_Data Operational Primitive
+   (see Section 3.1.2) was qualified with Notify_Enable set.  SN refers
+   to the DataSN associated with the SCSI Data-in PDU.
+
+   This is used by the iSER layer to notify the iSCSI layer of the
+   completion of inbound data transfer that was requested by the iSCSI
+   layer only if the invocation of the Get_Data Operational Primitive
+   (see Section 3.1.3) was qualified with Notify_Enable set.  SN refers
+   to the R2TSN associated with the R2T PDU.
+
+3.2.3.  Data_ACK_Notify
+
+      Input qualifier:  Connection_Handle, ITT, DataSN
+
+      Return results:  Not specified
+
+   This is used by the iSER layer at the target to notify the iSCSI
+   layer of the arrival of the data acknowledgement (as defined in
+   [RFC3720]) requested earlier by the iSCSI layer for the outbound data
+   transfer via an invocation of the Put_Data Operational Primitive
+   where the A-bit in the SCSI Data-in PDU is set to 1.  See Section
+   7.3.5.  DataSN refers to the expected DataSN of the next SCSI Data-in
+   PDU, which immediately follows the SCSI Data-in PDU with the A-bit
+   set to which this notification corresponds, with semantics as defined
+   in [RFC3720].
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 24]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+3.2.4.  Connection_Terminate_Notify
+
+      Input qualifiers:  Connection_Handle
+
+      Return results:  Not specified
+
+   This is used by the iSER layers at the initiator and the target to
+   notify the iSCSI layer of the unsolicited termination or failure of
+   an iSCSI/iSER connection.  The iSER layer MUST deallocate the
+   connection and task resources associated with the terminated
+   connection before the invocation of this Operational Primitive.  Note
+   that the Connection_Terminate_Notify Operational Primitive is not
+   invoked when the termination of the connection is earlier requested
+   by the local iSCSI layer.
+
+3.3.  iSCSI Protocol Usage Requirements
+
+   To operate in an iSER-assisted mode, the iSCSI layers at both the
+   initiator and the target MUST negotiate the RDMAExtensions key (see
+   Section 6.3) to "Yes" on the leading connection.  If the
+   RDMAExtensions key is not negotiated to "Yes", then iSER-assisted
+   mode MUST NOT be used.  If the RDMAExtensions key is negotiated to
+   "Yes" but the invocation of the Allocate_Connection_Resources
+   Operational Primitive to the iSER layer fails, the iSCSI layer MUST
+   fail the iSCSI Login process or terminate the connection as
+   appropriate.  See Section 10.1.3.1 for details.
+
+   If the RDMAExtensions key is negotiated to "Yes", the iSCSI layer
+   MUST satisfy the following protocol usage requirements from the iSER
+   protocol:
+
+   1.  The iSCSI layer at the initiator MUST set ExpDataSN to 0 in Task
+       Management Function Requests for Task Allegiance Reassignment for
+       read/bidirectional commands, so as to cause the target to send
+       all unacknowledged read data.
+
+   2.  The iSCSI layer at the target MUST always return the SCSI status
+       in a separate SCSI Response PDU for read commands, i.e., there
+       MUST NOT be a "phase collapse" in concluding a SCSI read command.
+
+   3.  The iSCSI layers at both the initiator and the target MUST
+       support the keys as defined in Section 6 on Login/Text
+       Operational Keys.  If used as specified, these keys MUST NOT be
+       answered with NotUnderstood, and the semantics as defined MUST be
+       followed for each iSER-assisted connection.
+
+   4.  The iSCSI layer at the initiator MUST NOT issue SNACKs for PDUs.
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 25]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+4.  Lower Layer Interface Requirements
+
+4.1.  Interactions with the RCaP Layer
+
+   The iSER protocol layer is layered on top of an RCaP layer (see
+   Figure 1) and the following are the key features that are assumed to
+   be supported by any RCaP layer:
+
+   *  The RCaP layer supports all basic RDMA operations, including RDMA
+      Write Operation, RDMA Read Operation, Send Operation, Send with
+      Invalidate Operation, Send with Solicited Event Operation, Send
+      with Solicited Event and Invalidate Operation, and Terminate
+      Operation.
+
+   *  The RCaP layer provides reliable, in-order message delivery and
+      direct data placement.
+
+   *  When the iSER layer initiates an RDMA Read Operation following an
+      RDMA Write Operation on one RCaP Stream, the RDMA Read Response
+      Message processing on the remote node will be started only after
+      the preceding RDMA Write Message payload is placed in the memory
+      of the remote node.
+
+   *  The RCaP layer encapsulates a single iSER Message into a single
+      RCaP Message on the Data Source side.  The RCaP layer decapsulates
+      the iSER Message before delivering it to the iSER layer on the
+      Data Sink side.
+
+   *  When the iSER layer provides the STag to be remotely invalidated
+      to the RCaP layer for a SendInvSE Message, the RCaP layer uses
+      this STag as the STag to be invalidated in the SendInvSE Message.
+
+   *  The RCaP layer uses the STag and Tagged Offset provided by the
+      iSER layer for the RDMA Write and RDMA Read Request Messages.
+
+   *  When the RCaP layer delivers the content of an RDMA Send Message
+      Type to the iSER layer, the RCaP layer provides the length of the
+      RDMA Send message.  This ensures that the iSER layer does not have
+      to carry a length field in the iSER header.
+
+   *  When the RCaP layer delivers the SendSE or SendInvSE Message to
+      the iSER layer, it notifies the iSER layer with the mechanism
+      provided on that interface.
+
+   *  When the RCaP layer delivers a SendInvSE Message to the iSER
+      layer, it passes the value of the STag that was invalidated.
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 26]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   *  The RCaP layer propagates all status and error indications to the
+      iSER layer.
+
+   *  For a transport layer that operates in byte stream mode such as
+      TCP, the RCaP implementation supports the enabling of the RDMA
+      mode after connection establishment and the exchange of Login
+      parameters in byte stream mode.  For a transport layer that
+      provides message delivery capability such as [IB], the RCaP
+      implementation supports the use of the messaging capability by the
+      iSCSI layer directly for the Login Phase after connection
+      establishment before enabling iSER-assisted mode.
+
+   *  Whenever the iSER layer terminates the RCaP Stream, the RCaP layer
+      terminates the associated connection.
+
+4.2.  Interactions with the Transport Layer
+
+   The iSER layer does not directly setup the transport layer connection
+   (e.g., TCP, or [IB]).  During connection setup, the iSCSI layer is
+   responsible for setting up the connection.  If the login is
+   successful, the iSCSI layer invokes the Enable_Datamover Operational
+   Primitive to request the iSER layer to transition to the iSER-
+   assisted mode for that iSCSI connection.  See Section 5.1 on
+   iSCSI/iSER connection setup.  After transitioning to iSER-assisted
+   mode, the RCaP layer and the underlying transport layer are
+   responsible for maintaining the connection and reporting to the iSER
+   layer any connection failures.
+
+5.  Connection Setup and Termination
+
+5.1.  iSCSI/iSER Connection Setup
+
+   During connection setup, the iSCSI layer at the initiator is
+   responsible for establishing a connection with the target.  After the
+   connection is established, the iSCSI layers at the initiator and the
+   target enter the Login Phase using the same rules as outlined in
+   [RFC3720].  Transition to iSER-assisted mode occurs when the
+   connection transitions into the iSCSI Full Feature Phase following a
+   successful login negotiation between the initiator and the target in
+   which iSER-assisted mode is negotiated and the connection resources
+   necessary to support RCaP have been allocated at both the initiator
+   and the target.  The same connection MUST be used for both the iSCSI
+   Login Phase and the subsequent iSER-assisted Full Feature Phase.
+
+   iSER-assisted mode MUST be enabled only if it is negotiated on the
+   leading connection during the LoginOperationalNegotiation stage of
+   the iSCSI Login Phase.  iSER-assisted mode is negotiated using the
+   RDMAExtensions=<boolean-value> key.  Both the initiator and the
+
+
+
+Ko, et al.                  Standards Track                    [Page 27]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   target MUST exchange the RDMAExtensions key with the value set to
+   "Yes" to enable iSER-assisted mode.  If both the initiator and the
+   target fail to negotiate the RDMAExtensions key set to "Yes", then
+   the connection MUST continue with the login semantics as defined in
+   [RFC3720].  If the RDMAExtensions key is not negotiated to Yes, then
+   for some RCaP implementation (such as [IB]), the connection may need
+   to be re-established in TCP capable mode.  (For InfiniBand this will
+   require an [IPoIB] type connection.)
+
+   iSER-assisted mode is defined for a Normal session only and the
+   RDMAExtensions key MUST NOT be negotiated for a Discovery session.
+   Discovery sessions are always conducted using the transport layer as
+   described in [RFC3720].
+
+   An iSER enabled node is not required to initiate the RDMAExtensions
+   key exchange if its preference is for the Traditional iSCSI mode.
+   The RDMAExtensions key, if offered, MUST be sent in the first
+   available Login Response or Login Request PDU in the
+   LoginOperationalNegotiation stage.  This is due to the fact that the
+   value of some login parameters might depend on whether iSER-assisted
+   mode is enabled.
+
+   iSER-assisted mode is a session-wide attribute.  If both the
+   initiator and the target negotiate RDMAExtensions="Yes" on the
+   leading connection of a session, then all subsequent connections of
+   the same session MUST enable iSER-assisted mode without having to
+   exchange an RDMAExtensions key during the iSCSI Login Phase.
+
+   Conversely, if both the initiator and the target fail to negotiate
+   RDMAExtensions to "Yes" on the leading connection of a session, then
+   the RDMAExtensions key MUST NOT be negotiated further on any
+   additional subsequent connection of the session.
+
+   When the RDMAExtensions key is negotiated to "Yes", the HeaderDigest
+   and the DataDigest keys MUST be negotiated to "None" on all
+   iSCSI/iSER connections participating in that iSCSI session.  This is
+   because, for an iSCSI/iSER connection, RCaP is responsible for
+   providing error detection that is at least as good as a 32-bit CRC
+   for all iSER Messages.  Furthermore, all SCSI Read data are sent
+   using RDMA Write Messages instead of the SCSI Data-in PDUs, and all
+   solicited SCSI write data are sent using RDMA Read Response Messages
+   instead of the SCSI Data-out PDUs.  HeaderDigest and DataDigest that
+   apply to iSCSI PDUs, would not be appropriate for RDMA Read and RDMA
+   Write operations used with iSER.
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 28]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+5.1.1.  Initiator Behavior
+
+   If the outcome of the iSCSI negotiation is to enable iSER-assisted
+   mode, then on the initiator side, prior to sending the Login Request
+   with the T (Transit) bit set to 1 and the NSG (Next Stage) field set
+   to FullFeaturePhase, the iSCSI layer MUST request that the iSER layer
+   allocate the connection resources necessary to support RCaP by
+   invoking the Allocate_Connection_Resources Operational Primitive.
+   The connection resources required are defined by implementation and
+   are outside the scope of this specification.  The iSCSI layer may
+   invoke the Notice_Key_Values Operational Primitive before invoking
+   the Allocate_Connection_Resources Operational Primitive to request
+   that the iSER layer take note of the negotiated values of the iSCSI
+   keys for the connection.  The specific keys to be passed as input
+   qualifiers are implementation dependent.  These may include, but are
+   not limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc.
+
+   To minimize the potential for a denial-of service attack, the iSCSI
+   layer MUST NOT request that the iSER layer allocate the connection
+   resources necessary to support RCaP until the iSCSI layer is
+   sufficiently far along in the iSCSI Login Phase that it is reasonably
+   certain that the peer side is not an attacker.  In particular, if the
+   Login Phase includes a SecurityNegotiation stage, the iSCSI layer
+   MUST defer the connection resource allocation (i.e., invoking the
+   Allocate_Connection_Resources Operational Primitive) to the
+   LoginOperationalNegotiation stage [RFC3720] so that the resource
+   allocation occurs after the authentication phase is completed.
+
+   Among the connection resources allocated at the initiator is the
+   Inbound RDMA Read Queue Depth (IRD).  As described in Section 9.5.1,
+   R2Ts are transformed by the target into RDMA Read operations.  IRD
+   limits the maximum number of simultaneously incoming outstanding RDMA
+   Read Requests per an RCaP Stream from the target to the initiator.
+   The required value of IRD is outside the scope of the iSER
+   specification.  The iSER layer at the initiator MUST set IRD to 1 or
+   higher if R2Ts are to be used in the connection.  However, the iSER
+   layer at the initiator MAY set IRD to 0 based on implementation
+   configuration, which indicates that no R2Ts will be used on that
+   connection.  Initially, the iSER-IRD value at the initiator SHOULD be
+   set to the IRD value at the initiator and MUST NOT be more than the
+   IRD value.
+
+   On the other hand, the Outbound RDMA Read Queue Depth (ORD) MAY be
+   set to 0, since the iSER layer at the initiator does not issue RDMA
+   Read Requests to the target.
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 29]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   Failure to allocate the requested connection resources locally
+   results in a login failure and its handling is described in Section
+   10.1.3.1.
+
+   If the iSER layer at the initiator is successful in allocating the
+   connection resources necessary to support RCaP, the following events
+   MUST occur in the specified sequence:
+
+   1.  The iSER layer MUST return a success status to the iSCSI layer in
+       response to the Allocate_Connection_Resources Operational
+       Primitive.
+
+   2.  After the target returns the Login Response with the T bit set to
+       1 and the NSG field set to FullFeaturePhase, and a status class
+       of 0 (Success), the iSCSI layer MUST request that the iSER layer
+       transition to iSER-assisted mode by invoking the Enable_Datamover
+       Operational Primitive with the following qualifiers.  (See
+       Section 10.1.4.6 for the case when the status class is not
+       Success.):
+
+       a.  Connection_Handle that identifies the iSCSI connection.
+
+       b.  Transport_Connection_Descriptor that identifies the specific
+           transport connection associated with the Connection_Handle.
+
+   3.  If necessary, the iSER layer should enable RCaP and transition
+       the connection to iSER-assisted mode.  When the RCaP is iWARP,
+       then this step MUST be done.  Not all RCaPs may need it depending
+       on the RCaP Stream start-up state.
+
+   4.  The iSER layer MUST send the iSER Hello Message as the first iSER
+       Message.  See Section 5.1.3 on iSER Hello Exchange.
+
+5.1.2.  Target Behavior
+
+   If the outcome of the iSCSI negotiation is to enable iSER-assisted
+   mode, then on the target side, prior to sending the Login Response
+   with the T (Transit) bit set to 1 and the NSG (Next Stage) field set
+   to FullFeaturePhase, the iSCSI layer MUST request that the iSER layer
+   allocate the resources necessary to support RCaP by invoking the
+   Allocate_Connection_Resources Operational Primitive.  The connection
+   resources required are defined by implementation and are outside the
+   scope of this specification.  Optionally, the iSCSI layer may invoke
+   the Notice_Key_Values Operational Primitive before invoking the
+   Allocate_Connection_Resources Operational Primitive to request that
+   the iSER layer take note of the negotiated values of the iSCSI keys
+   for the connection.  The specific keys to be passed as input
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 30]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   qualifiers are implementation dependent.  These may include, but are
+   not limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc.
+
+   To minimize the potential for a denial-of-service attack, the iSCSI
+   layer MUST NOT request that the iSER layer allocate the connection
+   resources necessary to support RCaP until the iSCSI layer is
+   sufficiently far along in the iSCSI Login Phase that it is reasonably
+   certain that the peer side is not an attacker.  In particular, if the
+   Login Phase includes a SecurityNegotiation stage, the iSCSI layer
+   MUST defer the connection resource allocation (i.e., invoking the
+   Allocate_Connection_Resources Operational Primitive) to the
+   LoginOperationalNegotiation stage [RFC3720] so that the resource
+   allocation occurs after the authentication phase is completed.
+
+   Among the connection resources allocated at the target is the
+   Outbound RDMA Read Queue Depth (ORD).  As described in Section 9.5.1,
+   R2Ts are transformed by the target into RDMA Read operations.  The
+   ORD limits the maximum number of simultaneously outstanding RDMA Read
+   Requests per RCaP Stream from the target to the initiator.
+   Initially, the iSER-ORD value at the target SHOULD be set to the ORD
+   value at the target.
+
+   On the other hand, the IRD at the target MAY be set to 0 since the
+   iSER layer at the target does not expect RDMA Read Requests to be
+   issued by the initiator.
+
+   Failure to allocate the requested connection resources locally
+   results in a login failure and its handling is described in Section
+   10.1.3.1.
+
+   If the iSER layer at the target is successful in allocating the
+   connection resources necessary to support RCaP, the following events
+   MUST occur in the specified sequence:
+
+   1.  The iSER layer MUST return a success status to the iSCSI layer in
+       response to the Allocate_Connection_Resources Operational
+       Primitive.
+
+   2.  The iSCSI layer MUST request that the iSER layer transition to
+       iSER-assisted mode by invoking the Enable_Datamover Operational
+       Primitive with the following qualifiers:
+
+       a.  Connection_Handle that identifies the iSCSI connection.
+
+       b.  Transport_Connection_Descriptor that identifies the specific
+           transport connection associated with the Connection_Handle.
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 31]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+       c.  The final transport layer (e.g., TCP) message containing the
+           Login Response with the T bit set to 1 and the NSG field set
+           to FullFeaturePhase.
+
+   3.  The iSER layer MUST send the final Login Response PDU in the
+       native transport mode to conclude the iSCSI Login Phase.  If the
+       underlying transport is TCP, then the iSER layer MUST send the
+       final Login Response PDU in byte stream mode.
+
+   4.  After sending the final Login Response PDU, the iSER layer should
+       enable RCaP if necessary and transition the connection to iSER-
+       assisted mode.  When the RCaP is iWARP, then this step MUST be
+       done.  Not all RCaPs may need it depending on the RCaP Stream
+       start-up state.
+
+   5.  After receiving the iSER Hello Message from the initiator, the
+       iSER layer MUST respond with the iSER HelloReply Message to be
+       sent as the first iSER Message.  See Section 5.1.3 on iSER Hello
+       Exchange for more details.
+
+   Note: In the above sequence, the operations as described in bullets 3
+   and 4 MUST be performed atomically for iWARP connections.  Failure to
+   do this may result in race conditions.
+
+5.1.3.  iSER Hello Exchange
+
+   After the connection transitions into iSER-assisted mode, the first
+   iSER Message sent by the iSER layer at the initiator to the target
+   MUST be the iSER Hello Message.  The iSER Hello Message is used by
+   the iSER layer at the initiator to declare iSER parameters to the
+   target.  See Section 9.3 on iSER Header Format for the iSER Hello
+   Message.
+
+   In response to the iSER Hello Message, the iSER layer at the target
+   MUST return the iSER HelloReply Message as the first iSER Message
+   sent by the target.  The iSER HelloReply Message is used by the iSER
+   layer at the target to declare iSER parameters to the initiator.  See
+   Section 9.4 on iSER Header Format for the iSER HelloReply Message.
+
+   In the iSER Hello Message, the iSER layer at the initiator declares
+   the iSER-IRD value to the target.
+
+   Upon receiving the iSER Hello Message, the iSER layer at the target
+   MUST set the iSER-ORD value to the minimum of the iSER-ORD value at
+   the target and the iSER-IRD value declared by the initiator.  The
+   iSER layer at the target MAY adjust (lower) its ORD value to match
+   the iSER-ORD value if the iSER-ORD value is smaller than the ORD
+   value at the target in order to free up the unused resources.
+
+
+
+Ko, et al.                  Standards Track                    [Page 32]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   In the iSER HelloReply Message, the iSER layer at the target declares
+   the iSER-ORD value to the initiator.
+
+   Upon receiving the iSER HelloReply Message, the iSER layer at the
+   initiator MAY adjust (lower) its IRD value to match the iSER-ORD
+   value in order to free up the unused resources, if the iSER-ORD value
+   declared by the target is smaller than the iSER-IRD value declared by
+   the initiator.
+
+   It is an iSER level negotiation failure if the iSER parameters
+   declared in the iSER Hello Message by the initiator are unacceptable
+   to the target.  This includes the following:
+
+   *  The initiator-declared iSER-IRD value is greater than 0 and the
+      target-declared iSER-ORD value is 0.
+
+   *  The initiator-supported and the target-supported iSER protocol
+      versions do not overlap.
+
+   See Section 10.1.3.2 for the handling of the error situation.
+
+5.2.  iSCSI/iSER Connection Termination
+
+5.2.1.  Normal Connection Termination at the Initiator
+
+   The iSCSI layer at the initiator terminates an iSCSI/iSER connection
+   normally by invoking the Send_Control Operational Primitive qualified
+   with the Logout Request PDU.  The iSER layer at the initiator MUST
+   use a SendSE Message to send the Logout Request PDU to the target.
+   After the iSER layer at the initiator receives the SendSE Message
+   containing the Logout Response PDU from the target, it MUST notify
+   the iSCSI layer by invoking the Control_Notify Operational Primitive
+   qualified with the Logout Response PDU.
+
+   After the iSCSI logout process is complete, the iSCSI layer at the
+   target is responsible for closing the iSCSI/iSER connection as
+   described in Section 5.2.2.  After the RCaP layer at the initiator
+   reports that the connection has been closed, the iSER layer at the
+   initiator MUST deallocate all connection and task resources (if any)
+   associated with the connection, and invalidate the Local Mapping(s)
+   (if any) that associate the ITT(s) used on that connection to the
+   local STag(s) before notifying the iSCSI layer by invoking the
+   Connection_Terminate_Notify Operational Primitive.
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 33]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+5.2.2.  Normal Connection Termination at the Target
+
+   Upon receiving the SendSE Message containing the Logout Request PDU,
+   the iSER layer at the target MUST notify the iSCSI layer at the
+   target by invoking the Control_Notify Operational Primitive qualified
+   with the Logout Request PDU.  The iSCSI layer completes the logout
+   process by invoking the Send_Control Operational Primitive qualified
+   with the Logout Response PDU.  The iSER layer at the target MUST use
+   a SendSE Message to send the Logout Response PDU to the initiator.
+   After the iSCSI logout process is complete, the iSCSI layer at the
+   target MUST request that the iSER layer at the target terminate the
+   RCaP Stream by invoking the Connection_Terminate Operational
+   Primitive.
+
+   As part of the termination process, the RCaP layer MUST close the
+   connection.  When the RCaP layer notifies the iSER layer after the
+   RCaP Stream and the associated connection are terminated, the iSER
+   layer MUST deallocate all connection and task resources (if any)
+   associated with the connection, and invalidate the Local and Remote
+   Mapping(s) (if any) that associate the ITT(s) used on that connection
+   to the local STag(s) and the Advertised STag(s) respectively.
+
+5.2.3.  Termination without Logout Request/Response PDUs
+
+5.2.3.1.  Connection Termination Initiated by the iSCSI Layer
+
+   The Connection_Terminate Operational Primitive MAY be invoked by the
+   iSCSI layer to request that the iSER layer terminate the RCaP Stream
+   without having previously exchanged the Logout Request and Logout
+   Response PDUs between the two iSCSI/iSER nodes.  As part of the
+   termination process, the RCaP layer will close the connection.  When
+   the RCaP layer notifies the iSER layer after the RCaP Stream and the
+   associated connection are terminated, the iSER layer MUST perform the
+   following actions.
+
+   If the Connection_Terminate Operational Primitive is invoked by the
+   iSCSI layer at the target, then the iSER layer at the target MUST
+   deallocate all connection and task resources (if any) associated with
+   the connection, and invalidate the Local and Remote Mappings (if any)
+   that associate the ITT(s) used on the connection to the local STag(s)
+   and the Advertised STag(s), respectively.
+
+   If the Connection_Terminate Operational Primitive is invoked by the
+   iSCSI layer at the initiator, then the iSER layer at the initiator
+   MUST deallocate all connection and task resources (if any) associated
+   with the connection, and invalidate the Local Mapping(s) (if any)
+   that associate the ITT(s) used on the connection to the local
+   STag(s).
+
+
+
+Ko, et al.                  Standards Track                    [Page 34]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+5.2.3.2.  Connection Termination Notification to the iSCSI Layer
+
+   If the iSCSI/iSER connection is terminated without the invocation of
+   Connection_Terminate from the iSCSI layer, the iSER layer MUST notify
+   the iSCSI layer that the iSCSI/iSER connection has been terminated by
+   invoking the Connection_Terminate_Notify Operational Primitive.
+
+   Prior to invoking Connection_Terminate_Notify, the iSER layer at the
+   target MUST deallocate all connection and task resources (if any)
+   associated with the connection, and invalidate the Local and Remote
+   Mappings (if any) that associate the ITT(s) used on the connection to
+   the local STag(s) and the Advertised STag(s), respectively.
+
+   Prior to invoking Connection_Terminate_Notify, the iSER layer at the
+   initiator MUST deallocate all connection and task resources (if any)
+   associated with the connection, and invalidate the Local Mappings (if
+   any) that associate the ITT(s) used on the connection to the local
+   STag(s).
+
+   If the remote iSCSI/iSER node initiated the closing of the connection
+   (e.g., by sending a TCP FIN or TCP RST), the iSER layer MUST notify
+   the iSCSI layer after the RCaP layer reports that the connection is
+   closed by invoking the Connection_Terminate_Notify Operational
+   Primitive.
+
+   Another example of a connection termination without a preceding
+   logout is when the iSCSI layer at the initiator does an implicit
+   logout (connection reinstatement).
+
+6.  Login/Text Operational Keys
+
+   Certain iSCSI login/text operational keys have restricted usage in
+   iSER, and additional keys are used to support the iSER protocol
+   functionality.  All other keys defined in [RFC3720] and not discussed
+   in this section may be used on iSCSI/iSER connections with the same
+   semantics.
+
+6.1.  HeaderDigest and DataDigest
+
+   Irrelevant when: RDMAExtensions=Yes
+
+   Negotiations resulting in RDMAExtensions=Yes for a session implies
+   HeaderDigest=None and DataDigest=None for all connections in that
+   session and overrides both the default and an explicit setting.
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 35]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+6.2.  MaxRecvDataSegmentLength
+
+   For an iSCSI connection belonging to a session in which
+   RDMAExtensions=Yes was negotiated on the leading connection of the
+   session, MaxRecvDataSegmentLength need not be declared in the Login
+   Phase.  Instead, InitiatorRecvDataSegmentLength (as described in
+   Section 6.5) and TargetRecvDataSegmentLength (as described in Section
+   6.4) keys are negotiated.  The values of the local and remote
+   MaxRecvDataSegmentLength are derived from the
+   InitiatorRecvDataSegmentLength and TargetRecvDataSegmentLength keys
+   even if the MaxRecvDataSegmentLength is declared during the Login
+   Phase.
+
+   In the Full Feature Phase, the initiator MUST consider the value of
+   its local MaxRecvDataSegmentLength (that it would have declared to
+   the target) as having the value of InitiatorRecvDataSegmentLength,
+   and the value of the remote MaxRecvDataSegmentLength (that would have
+   been declared by the target) as having the value of
+   TargetRecvDataSegmentLength.  Similarly, the target MUST consider the
+   value of its local MaxRecvDataSegmentLength (that it would have
+   declared to the initiator) as having the value of
+   TargetRecvDataSegmentLength, and the value of the remote
+   MaxRecvDataSegmentLength (that would have been declared by the
+   initiator) as having the value of InitiatorRecvDataSegmentLength.
+
+   The MaxRecvDataSegmentLength key is applicable only for iSCSI
+   control-type PDUs.
+
+6.3.  RDMAExtensions
+
+   Use: LO (leading only)
+
+   Senders: Initiator and Target
+
+   Scope: SW (session-wide)
+
+   RDMAExtensions=<boolean-value>
+
+   Irrelevant when: SessionType=Discovery
+
+   Default is No
+
+   Result function is AND
+
+   This key is used by the initiator and the target to negotiate support
+   for iSER-assisted mode.  To enable the use of iSER-assisted mode,
+   both the initiator and the target MUST exchange RDMAExtensions=Yes.
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 36]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   iSER-assisted mode MUST NOT be used if either the initiator or the
+   target offers RDMAExtensions=No.
+
+   An iSER-enabled node is not required to initiate the RDMAExtensions
+   key exchange if it prefers to operate in the Traditional iSCSI mode.
+   However, if the RDMAExtensions key is to be negotiated, an initiator
+   MUST offer the key in the first Login Request PDU in the
+   LoginOperationalNegotiation stage of the leading connection, and a
+   target MUST offer the key in the first Login Response PDU with which
+   it is allowed to do so (i.e., the first Login Response PDU issued
+   after the first Login Request PDU with the C bit set to 0) in the
+   LoginOperationalNegotiation stage of the leading connection.  In
+   response to the offered key=value pair of RDMAExtensions=yes, an
+   initiator MUST respond in the next Login Request PDU with which it is
+   allowed to do so, and a target MUST respond in the next Login
+   Response PDU with which it is allowed to do so.
+
+   Negotiating the RDMAExtensions key first enables a node to negotiate
+   the optimal value for other keys.  Certain iSCSI keys such as
+   MaxBurstLength, MaxOutstandingR2T, ErrorRecoveryLevel, InitialR2T,
+   ImmediateData, etc., may be negotiated differently depending on
+   whether the connection is in Traditional iSCSI mode or iSER-assisted
+   mode.
+
+6.4.  TargetRecvDataSegmentLength
+
+   Use: IO (Initialize only)
+
+   Senders: Initiator and Target
+
+   Scope: CO (connection-only)
+
+   Irrelevant when: RDMAExtensions=No
+
+   TargetRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>
+
+   Default is 8192 bytes
+
+   Result function is minimum
+
+   This key is relevant only for the iSCSI connection of an iSCSI
+   session if RDMAExtensions=Yes is negotiated on the leading connection
+   of the session.  It is used by the initiator and target to negotiate
+   the maximum size of the data segment that an initiator may send to
+   the target in an iSCSI control-type PDU in the Full Feature Phase.
+   For SCSI Command PDUs and SCSI Data-out PDUs containing non-immediate
+   unsolicited data to be sent by the initiator, the initiator MUST send
+   all non-Final PDUs with a data segment size of exactly
+
+
+
+Ko, et al.                  Standards Track                    [Page 37]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   TargetRecvDataSegmentLength whenever the PDUs constitute a data
+   sequence whose size is larger than TargetRecvDataSegmentLength.
+
+6.5.  InitiatorRecvDataSegmentLength
+
+   Use: IO (Initialize only)
+
+   Senders: Initiator and Target
+
+   Scope: CO (connection-only)
+
+   Irrelevant when: RDMAExtensions=No
+
+   InitiatorRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>
+
+   Default is 8192 bytes
+
+   Result function is minimum
+
+   This key is relevant only for the iSCSI connection of an iSCSI
+   session if RDMAExtensions=Yes is negotiated on the leading connection
+   of the session.  It is used by the initiator and target to negotiate
+   the maximum size of the data segment that a target may send to the
+   initiator in an iSCSI control-type PDU in the Full Feature Phase.
+
+6.6.  OFMarker and IFMarker
+
+   Irrelevant when: RDMAExtensions=Yes
+
+   Negotiations resulting in RDMAExtensions=Yes for a session implies
+   OFMarker=No and IFMarker=No for all connections in that session and
+   overrides both the default and an explicit setting.
+
+6.7.  MaxOutstandingUnexpectedPDUs
+
+   Use: LO (leading only), Declarative
+
+   Senders: Initiator and Target
+
+   Scope: SW (session-wide)
+
+   Irrelevant when: RDMAExtensions=No
+
+   MaxOutstandingUnexpectedPDUs=<numerical-value-from-2-to-(2**32-1) |
+   0>
+
+   Default is 0
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 38]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   This key is used by the initiator and the target to declare the
+   maximum number of outstanding "unexpected" iSCSI control-type PDUs
+   that it can receive in the Full Feature Phase.  It is intended to
+   allow the receiving side to determine the amount of buffer resources
+   needed beyond the normal flow control mechanism available in iSCSI.
+   An initiator or target should select a value such that it would not
+   impose an unnecessary constraint on the iSCSI layer under normal
+   circumstances.  The value of 0 is defined to indicate that the
+   declarer has no limit on the maximum number of outstanding
+   "unexpected" iSCSI control-type PDUs that it can receive.  See
+   Sections 8.1.1 and 8.1.2 for the usage of this key.  Note that iSER
+   Hello and HelloReply Messages are not iSCSI control-type PDUs and are
+   not affected by this key.
+
+7.  iSCSI PDU Considerations
+
+   When a connection is in the iSER-assisted mode, two types of message
+   transfers are allowed between the iSCSI layer at the initiator and
+   the iSCSI layer at the target.  These are known as the iSCSI data-
+   type PDUs and the iSCSI control-type PDUs, and these terms are
+   described in the following sections.
+
+7.1.  iSCSI Data-Type PDU
+
+   An iSCSI data-type PDU is defined as an iSCSI PDU that causes data
+   transfer, transparent to the remote iSCSI layer, to take place
+   between the peer iSCSI nodes in the full feature phase of an
+   iSCSI/iSER connection.  An iSCSI data-type PDU, when requested for
+   transmission by the iSCSI layer in the sending node, results in the
+   data being transferred without the participation of the iSCSI layers
+   at the sending and the receiving nodes.  This is due to the fact that
+   the PDU itself is not delivered as-is to the iSCSI layer in the
+   receiving node.  Instead, the data transfer operations are
+   transformed into the appropriate RDMA operations that are handled by
+   the RDMA-Capable Controller.  The set of iSCSI data-type PDUs
+   consists of SCSI Data-in PDUs and R2T PDUs.
+
+   If the invocation of the Operational Primitive by the iSCSI layer to
+   request that the iSER layer process an iSCSI data-type PDU is
+   qualified with Notify_Enable set, then upon completing the RDMA
+   operation, the iSER layer at the target MUST notify the iSCSI layer
+   at the target by invoking the Data_Completion_Notify Operational
+   Primitive qualified with ITT and SN.  There is no data completion
+   notification at the initiator since the RDMA operations are
+   completely handled by the RDMA-Capable Controller at the initiator
+   and the iSER layer at the initiator is not involved with the data
+   transfer associated with iSCSI data-type PDUs.
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 39]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   If the invocation of the Operational Primitive by the iSCSI layer to
+   request that the iSER layer process an iSCSI data-type PDU is
+   qualified with Notify_Enable cleared, then upon completing the RDMA
+   operation, the iSER layer at the target MUST NOT notify the iSCSI
+   layer at the target and MUST NOT invoke the Data_Completion_Notify
+   Operational Primitive.
+
+   If an operation associated with an iSCSI data-type PDU fails for any
+   reason, the contents of the Data Sink buffers associated with the
+   operation are considered indeterminate.
+
+7.2.  iSCSI Control-Type PDU
+
+   Any iSCSI PDU that is not an iSCSI data-type PDU and also not a SCSI
+   Data-out PDU carrying solicited data is defined as an iSCSI control-
+   type PDU.  The iSCSI layer invokes the Send_Control Operational
+   Primitive to request that the iSER layer process an iSCSI control-
+   type PDU.  iSCSI control-type PDUs are transferred using Send Message
+   Types of RCaP.  Specifically, note that SCSI Data-out PDUs carrying
+   unsolicited data are defined as iSCSI control-type PDUs.  See Section
+   7.3.4 on the treatment of SCSI Data-out PDUs.
+
+   When the iSER layer receives an iSCSI control-type PDU, it MUST
+   notify the iSCSI layer by invoking the Control_Notify Operational
+   Primitive qualified with the iSCSI control-type PDU.
+
+7.3.  iSCSI PDUs
+
+   This section describes the handling of each of the iSCSI PDU types by
+   the iSER layer.  The iSCSI layer requests that the iSER layer process
+   the iSCSI PDU by invoking the appropriate Operational Primitive.  A
+   Connection_Handle MUST qualify each of these invocations.  In
+   addition, BHS and the optional AHS of the iSCSI PDU as defined in
+   [RFC3720] MUST qualify each of the invocations.  The qualifying
+   Connection_Handle, the BHS, and the AHS are not explicitly listed in
+   the subsequent sections.
+
+7.3.1.  SCSI Command
+
+      Type:  control-type PDU
+
+      PDU-specific qualifiers (for SCSI Write or bidirectional command):
+      ImmediateDataSize, UnsolicitedDataSize, DataDescriptorOut
+
+      PDU-specific qualifiers (for SCSI read or bidirectional command):
+      DataDescriptorIn
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 40]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   The iSER layer at the initiator MUST send the SCSI command in a
+   SendSE Message to the target.
+
+   For a SCSI Write or bidirectional command, the iSCSI layer at the
+   initiator MUST invoke the Send_Control Operational Primitive as
+   follows:
+
+   *  If there is immediate data to be transferred for the SCSI Write or
+      bidirectional command, the qualifier ImmediateDataSize MUST be
+      used to define the number of bytes of immediate unsolicited data
+      to be sent with the Write or bidirectional command, and the
+      qualifier DataDescriptorOut MUST be used to define the initiator's
+      I/O Buffer containing the SCSI Write data.
+
+   *  If there is unsolicited data to be transferred for the SCSI Write
+      or bidirectional command, the qualifier UnsolicitedDataSize MUST
+      be used to define the number of bytes of immediate and non-
+      immediate unsolicited data for the command.  The iSCSI layer will
+      issue one or more SCSI Data-out PDUs for the non-immediate
+      unsolicited data.  See Section 7.3.4 on SCSI Data-out.
+
+   *  If there is solicited data to be transferred for the SCSI write or
+      bidirectional command, as indicated by the Expected Data Transfer
+      Length in the SCSI Command PDU exceeding the value of
+      UnsolicitedDataSize, the iSER layer at the initiator MUST do the
+      following:
+
+         a.  It MUST allocate a Write STag for the I/O Buffer defined by
+             the qualifier DataDescriptorOut.  The DataDescriptorOut
+             describes the I/O buffer starting with the immediate
+             unsolicited data (if any), followed by the non-immediate
+             unsolicited data (if any) and solicited data.  This means
+             that the BufferOffset for the SCSI Data-out for this
+             command is equal to the TO.  This implies that a zero TO
+             for this STag points to the beginning of this I/O Buffer.
+
+         b.  It MUST establish a Local Mapping that associates the
+             Initiator Task Tag (ITT) to the Write STag.
+
+         c.  It MUST Advertise the Write STag to the target by sending
+             it as the Write STag in the iSER header of the iSER Message
+             (the payload of the SendSE Message of RCaP) containing the
+             SCSI write or bidirectional command PDU.  See Section 9.2
+             on iSER Header Format for the iSCSI Control-Type PDU.
+
+   For a SCSI read or bidirectional command, the iSCSI layer at the
+   initiator MUST invoke the Send_Control Operational Primitive
+   qualified with DataDescriptorIn, which defines the initiator's I/O
+
+
+
+Ko, et al.                  Standards Track                    [Page 41]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   Buffer for receiving the SCSI Read data.  The iSER layer at the
+   initiator MUST do the following:
+
+         a.  It MUST allocate a Read STag for the I/O Buffer.
+
+         b.  It MUST establish a Local Mapping that associates the
+             Initiator Task Tag (ITT) to the Read STag.
+
+         c.  It MUST Advertise the Read STag to the target by sending it
+             as the Read STag in the iSER header of the iSER Message
+             (the payload of the SendSE Message of RCaP) containing the
+             SCSI read or bidirectional command PDU.  See Section 9.2 on
+             iSER Header Format for the iSCSI Control-Type PDU.
+
+   If the amount of unsolicited data to be transferred in a SCSI command
+   exceeds TargetRecvDataSegmentLength, then the iSCSI layer at the
+   initiator MUST segment the data into multiple iSCSI control-type
+   PDUs, with the data segment length in all PDUs generated except the
+   last one having exactly the size TargetRecvDataSegmentLength.  The
+   data segment length of the last iSCSI control-type PDU carrying the
+   unsolicited data can be up to TargetRecvDataSegmentLength.
+
+   When the iSER layer at the target receives the SCSI command, it MUST
+   establish a Remote Mapping that associates the ITT to the Advertised
+   Write STag and the Read STag if present in the iSER header.  The
+   Write STag is used by the iSER layer at the target in handling the
+   data transfer associated with the R2T PDU(s) as described in Section
+   7.3.6.  The Read STag is used in handling the SCSI Data-in PDU(s)
+   from the iSCSI layer at the target as described in Section 7.3.5.
+
+7.3.2.  SCSI Response
+
+      Type:  control-type PDU
+
+      PDU-specific qualifiers:  DataDescriptorStatus
+
+   The iSCSI layer at the target MUST invoke the Send_Control
+   Operational Primitive qualified with DataDescriptorStatus, which
+   defines the buffer containing the sense and response information.
+   The iSCSI layer at the target MUST always return the SCSI status for
+   a SCSI command in a separate SCSI Response PDU.  "Phase collapse" for
+   transferring SCSI status in a SCSI Data-in PDU MUST NOT be used.  The
+   iSER layer at the target sends the SCSI Response PDU according to the
+   following rules:
+
+   *  If no STags are Advertised by the initiator in the iSER Message
+      containing the SCSI command PDU, then the iSER layer at the target
+      MUST send a SendSE Message containing the SCSI Response PDU.
+
+
+
+Ko, et al.                  Standards Track                    [Page 42]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   *  If the initiator Advertised a Read STag in the iSER Message
+      containing the SCSI Command PDU, then the iSER layer at the target
+      MUST send a SendInvSE Message containing the SCSI Response PDU.
+      The header of the SendInvSE Message MUST carry the Read STag to be
+      invalidated at the initiator.
+
+   *  If the initiator Advertised only the Write STag in the iSER
+      Message containing the SCSI Command PDU, then the iSER layer at
+      the target MUST send a SendInvSE Message containing the SCSI
+      Response PDU.  The header of the SendInvSE Message MUST carry the
+      Write STag to be invalidated at the initiator.
+
+   When the iSCSI layer at the target invokes the Send_Control
+   Operational Primitive to send the SCSI Response PDU, the iSER layer
+   at the target MUST invalidate the Remote Mapping that associates the
+   ITT to the Advertised STag(s) before transferring the SCSI Response
+   PDU to the initiator.
+
+   Upon receiving the SendInvSE Message containing the SCSI Response PDU
+   from the target, the RCaP layer at the initiator will invalidate the
+   STag specified in the header.  The iSER layer at the initiator MUST
+   ensure that the correct STag is invalidated.  If both the Read and
+   the Write STags are Advertised earlier by the initiator, then the
+   iSER layer at the initiator MUST explicitly invalidate the Write STag
+   upon receiving the SendInvSE Message because the header of the
+   SendInvSE Message can only carry one STag (in this case, the Read
+   STag) to be invalidated.
+
+   The iSER layer at the initiator MUST ensure the invalidation of the
+   STag(s) used in a command before notifying the iSCSI layer at the
+   initiator by invoking the Control_Notify Operational Primitive
+   qualified with the SCSI Response.  This precludes the possibility of
+   using the STag(s) after the completion of the command, thereby
+   causing data corruption.
+
+   When the iSER layer at the initiator receives the SendSE or the
+   SendInvSE Message containing the SCSI Response PDU, it SHOULD
+   invalidate the Local Mapping that associates the ITT to the local
+   STag(s).  The iSER layer MUST ensure that all local STag(s)
+   associated with the ITT are invalidated before notifying the iSCSI
+   layer of the SCSI Response PDU by invoking the Control_Notify
+   Operational Primitive qualified with the SCSI Response PDU.
+
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 43]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+7.3.3.  Task Management Function Request/Response
+
+      Type:  control-type PDU
+
+      PDU-specific qualifiers (for TMF Request):  DataDescriptorOut,
+      DataDescriptorIn
+
+   The iSER layer MUST use a SendSE Message to send the Task Management
+   Function Request/Response PDU.
+
+   For the Task Management Function Request with the TASK REASSIGN
+   function, the iSER layer at the initiator MUST do the following:
+
+   *  It MUST use the ITT as specified in the Referenced Task Tag from
+      the Task Management Function Request PDU to locate the existing
+      STag(s), if any, in the Local Mapping(s) that associates the ITT
+      to the local STag(s).
+
+   *  It MUST invalidate the existing STag(s), if any, and the Local
+      Mapping(s) that associates the ITT to the local STag(s).
+
+   *  It MUST allocate a Read STag for the I/O Buffer as defined by the
+      qualifier DataDescriptorIn if the Send_Control Operational
+      Primitive invocation is qualified with DataDescriptorIn.
+
+   *  It MUST allocate a Write STag for the I/O Buffer as defined by the
+      qualifier DataDescriptorOut if the Send_Control Operational
+      Primitive invocation is qualified with DataDescriptorOut.
+
+   *  If STags are allocated, it MUST establish a new Local Mapping(s)
+      that associate the ITT to the allocated STag(s).
+
+   *  It MUST Advertise the STags, if allocated, to the target in the
+      iSER header of the SendSE Message carrying the iSCSI PDU, as
+      described in Section 9.2.
+
+   For the Task Management Function Request with the TASK REASSIGN
+   function for a SCSI read or bidirectional command, the iSCSI layer at
+   the initiator MUST set ExpDataSN to 0 since the data transfer and
+   acknowledgements happen transparently to the iSCSI layer at the
+   initiator.  This provides the flexibility to the iSCSI layer at the
+   target to request transmission of only the unacknowledged data as
+   specified in [RFC3720].
+
+   When the iSER layer at the target receives the Task Management
+   Function Request with the TASK REASSIGN function, it MUST do the
+   following:
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 44]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   *  It MUST use the ITT as specified in the Referenced Task Tag from
+      the Task Management Function Request PDU to locate the mappings
+      that associate the ITT to the Advertised STag(s) and the local
+      STag(s), if any.
+
+   *  It MUST invalidate the local STag(s), if any, associated with the
+      ITT.
+
+   *  It MUST replace the Advertised STag(s) in the Remote Mapping that
+      associates the ITT to the Advertised STag(s) with the Write STag
+      and the Read STag if present in the iSER header.  The Write STag
+      is used in the handling of the R2T PDU(s) from the iSCSI layer at
+      the target as described in Section 7.3.6.  The Read STag is used
+      in the handling of the SCSI Data-in PDU(s) from the iSCSI layer at
+      the target as described in Section 7.3.5.
+
+7.3.4.  SCSI Data-Out
+
+      Type:  control-type PDU
+
+      PDU-specific qualifiers:  DataDescriptorOut
+
+   The iSCSI layer at the initiator MUST invoke the Send_Control
+   Operational Primitive qualified with DataDescriptorOut, which defines
+   the initiator's I/O Buffer containing unsolicited SCSI Write data.
+
+   If the amount of unsolicited data to be transferred as SCSI Data-out
+   exceeds TargetRecvDataSegmentLength, then the iSCSI layer at the
+   initiator MUST segment the data into multiple iSCSI control-type
+   PDUs, with the DataSegmentLength having the value of
+   TargetRecvDataSegmentLength in all PDUs generated except the last
+   one.  The DataSegmentLength of the last iSCSI control-type PDU
+   carrying the unsolicited data can be up to
+   TargetRecvDataSegmentLength.  The iSCSI layer at the target MUST
+   perform the reassembly function for the unsolicited data.
+
+   For unsolicited data, if the F bit is set to 0 in a SCSI Data-out
+   PDU, the iSER layer at the initiator MUST use a Send Message to send
+   the SCSI Data-out PDU.  If the F bit is set to 1, the iSER layer at
+   the initiator MUST use a SendSE Message to send the SCSI Data-out
+   PDU.
+
+   Note that for solicited data, the SCSI Data-out PDUs are not used
+   since R2T PDUs are not delivered to the iSCSI layer at the initiator;
+   instead, R2T PDUs are transformed by the iSER layer at the target
+   into RDMA Read operations.  (See Section 7.3.6.)
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 45]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+7.3.5.  SCSI Data-In
+
+      Type:  data-type PDU
+
+      PDU-specific qualifiers:  DataDescriptorIn
+
+   When the iSCSI layer at the target is ready to return the SCSI Read
+   data to the initiator, it MUST invoke the Put_Data Operational
+   Primitive qualified with DataDescriptorIn, which defines the SCSI
+   Data-in buffer.  See Section 7.1 on the general requirement on the
+   handling of iSCSI data-type PDUs.  SCSI Data-in PDU(s) are used in
+   SCSI Read data transfer as described in Section 9.5.2.
+
+   The iSER layer at the target MUST do the following for each
+   invocation of the Put_Data Operational Primitive:
+
+   1.  It MUST use the ITT in the SCSI Data-in PDU to locate the remote
+       Read STag in the Remote Mapping that associates the ITT to
+       Advertised STag(s).  The Remote Mapping was established earlier
+       by the iSER layer at the target when the SCSI read command was
+       received from the initiator.
+
+   2.  It MUST generate and send an RDMA Write Message containing the
+       read data to the initiator.
+
+       a.  It MUST use the remote Read STag as the Data Sink STag of the
+           RDMA Write Message.
+
+       b.  It MUST use the Buffer Offset from the SCSI Data-in PDU as
+           the Data Sink Tagged Offset of the RDMA Write Message.
+
+       c.  It MUST use DataSegmentLength from the SCSI Data-in PDU to
+           determine the amount of data to be sent in the RDMA Write
+           Message.
+
+   3.  It MUST associate DataSN and ITT from the SCSI Data-in PDU with
+       the RDMA Write operation.  If the Put_Data Operational Primitive
+       invocation was qualified with Notify_Enable set, then when the
+       iSER layer at the target receives a completion from the RCaP
+       layer for the RDMA Write Message, the iSER layer at the target
+       MUST notify the iSCSI layer by invoking the
+       Data_Completion_Notify Operational Primitive qualified with
+       DataSN and ITT.  Conversely, if the Put_Data Operational
+       Primitive invocation was qualified with Notify_Enable cleared,
+       then the iSER layer at the target MUST NOT notify the iSCSI layer
+       on completion and MUST NOT invoke the Data_Completion_Notify
+       Operational Primitive.
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 46]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   When the A-bit is set to 1 in the SCSI Data-in PDU, the iSER layer at
+   the target MUST notify the iSCSI layer at the target when the data
+   transfer is complete at the initiator.  To perform this additional
+   function, the iSER layer at the target can take advantage of the
+   operational ErrorRecoveryLevel if previously disclosed by the iSCSI
+   layer via an earlier invocation of the Notice_Key_Values Operational
+   Primitive.  There are two approaches that can be taken:
+
+   1.  If the iSER layer at the target knows that the operational
+       ErrorRecoveryLevel is 2, or if the iSER layer at the target does
+       not know the operational ErrorRecoveryLevel, then the iSER layer
+       at the target MUST issue a zero-length RDMA Read Request Message
+       following the RDMA Write Message.  When the iSER layer at the
+       target receives a completion for the RDMA Read Request Message
+       from the RCaP layer, implying that the RDMA-Capable Controller at
+       the initiator has completed processing the RDMA Write Message due
+       to the completion ordering semantics of RCaP, the iSER layer at
+       the target MUST notify the iSCSI layer at the target by invoking
+       the Data_Ack_Notify Operational Primitive qualified with ITT and
+       DataSN (see Section 3.2.3).
+
+   2.  If the iSER layer at the target knows that the operational
+       ErrorRecoveryLevel is 1, then the iSER layer at the target MUST
+       do one of the following:
+
+       a.  It MUST notify the iSCSI layer at the target by invoking the
+           Data_Ack_Notify Operational Primitive qualified with ITT and
+           DataSN (see Section 3.2.3) when it receives the local
+           completion from the RCaP layer for the RDMA Write Message.
+           This is allowed since digest errors do not occur in iSER (see
+           Section 10.1.4.2) and a CRC error will cause the connection
+           to be terminated and the task to be terminated anyway.  The
+           local RDMA Write completion from the RCaP layer guarantees
+           that the RCaP layer will not access the I/O Buffer again to
+           transfer the data associated with that RDMA Write operation.
+
+       b.  Alternatively, it MUST use the same procedure for handling
+           the data transfer completion at the initiator as for
+           ErrorRecoveryLevel 2.
+
+   Note that the iSCSI layer at the target cannot set the A-bit to 1 if
+   the ErrorRecoveryLevel=0.
+
+   The SCSI status MUST always be returned in a separate SCSI Response
+   PDU.  The S bit in the SCSI Data-in PDU MUST always be set to 0.
+   There MUST NOT be a "phase collapse" in the SCSI Data-in PDU.
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 47]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   Since the RDMA Write Message only transfers the data portion of the
+   SCSI Data-in PDU but not the control information in the header, such
+   as ExpCmdSN, if timely updates of such information are crucial, the
+   iSCSI layer at the initiator MAY issue NOP-Out PDUs to request that
+   the iSCSI layer at the target respond with the information using NOP-
+   In PDUs.
+
+7.3.6.  Ready to Transfer (R2T)
+
+      Type:  data-type PDU
+
+      PDU-specific qualifiers:  DataDescriptorOut
+
+   In order to send an R2T PDU, the iSCSI layer at the target MUST
+   invoke the Get_Data Operational Primitive qualified with
+   DataDescriptorOut, which defines the I/O Buffer for receiving the
+   SCSI Write data from the initiator.  See Section 7.1 on the general
+   requirements on the handling of iSCSI data-type PDUs.
+
+   The iSER layer at the target MUST do the following for each
+   invocation of the Get_Data Operational Primitive:
+
+   1.  It MUST ensure a valid local STag for the I/O Buffer and a valid
+       Local Mapping that associates the Initiator Task Tag (ITT) to the
+       local STag.  This may involve allocating a valid local STag and
+       establishing a Local Mapping.
+
+   2.  It MUST use the ITT in the R2T to locate the remote Write STag in
+       the Remote Mapping that associates the ITT to Advertised STag(s).
+       The Remote Mapping is established earlier by the iSER layer at
+       the target when the iSER Message containing the Advertised Write
+       STag and the SCSI Command PDU for a SCSI write or bidirectional
+       command is received from the initiator.
+
+   3.  If the iSER-ORD value at the target is set to 0, the iSER layer
+       at the target MUST terminate the connection and free up the
+       resources associated with the connection (as described in Section
+       5.2.3) if it receives the R2T PDU from the iSCSI layer at the
+       target.  Upon termination of the connection, the iSER layer at
+       the target MUST notify the iSCSI layer at the target by invoking
+       the Connection_Terminate_Notify Operational Primitive.
+
+   4.  If the iSER-ORD value at the target is set to greater than 0, the
+       iSER layer at the target MUST transform the R2T PDU into an RDMA
+       Read Request Message.  While transforming the R2T PDU, the iSER
+       layer at the target MUST ensure that the number of outstanding
+       RDMA Read Request Messages does not exceed the iSER-ORD value.
+       To transform the R2T PDU, the iSER layer at the target:
+
+
+
+Ko, et al.                  Standards Track                    [Page 48]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+       a.  MUST derive the local STag and local Tagged Offset from the
+           DataDescriptorOut that qualified the Get_Data invocation.
+
+       b.  MUST use the local STag as the Data Sink STag of the RDMA
+           Read Request Message.
+
+       c.  MUST use the local Tagged Offset as the Data Sink Tagged
+           Offset of the RDMA Read Request Message.
+
+       d.  MUST use the Desired Data Transfer Length from the R2T PDU as
+           the RDMA Read Message Size of the RDMA Read Request Message.
+
+       e.  MUST use the remote Write STag as the Data Source STag of the
+           RDMA Read Request Message.
+
+       f.  MUST use the Buffer Offset from the R2T PDU as the Data
+           Source Tagged Offset of the RDMA Read Request Message.
+
+   5.  It MUST associate R2TSN and ITT from the R2T PDU with the RDMA
+       Read operation.  If the Get_Data Operational Primitive invocation
+       is qualified with Notify_Enable set, then when the iSER layer at
+       the target receives a completion from the RCaP layer for the RDMA
+       Read operation, the iSER layer at the target MUST notify the
+       iSCSI layer by invoking the Data_Completion_Notify Operational
+       Primitive qualified with R2TSN and ITT.  Conversely, if the
+       Get_Data Operational Primitive invocation is qualified with
+       Notify_Enable cleared, then the iSER layer at the target MUST NOT
+       notify the iSCSI layer on completion and MUST NOT invoke the
+       Data_Completion_Notify Operational Primitive.
+
+   When the RCaP layer at the initiator receives a valid RDMA Read
+   Request Message, it will return an RDMA Read Response Message
+   containing the solicited write data to the target.  When the RCaP
+   layer at target receives the RDMA Read Response Message from the
+   initiator, it will place the solicited data in the I/O Buffer
+   referenced by the Data Sink STag in the RDMA Read Response Message.
+
+   Since the RDMA Read Request Message from the target does not transfer
+   the control information in the R2T PDU, such as ExpCmdSN, if timely
+   updates of such information are crucial, the iSCSI layer at the
+   initiator MAY issue NOP-Out PDUs to request that the iSCSI layer at
+   the target respond with the information using NOP-In PDUs.
+
+   Similarly, since the RDMA Read Response Message from the initiator
+   only transfers the data but not the control information normally
+   found in the SCSI Data-out PDU, such as ExpStatSN, if timely updates
+   of such information are crucial, the iSCSI layer at the target MAY
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 49]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   issue NOP-In PDUs to request that the iSCSI layer at the initiator
+   respond with the information using NOP-Out PDUs.
+
+7.3.7.  Asynchronous Message
+
+      Type:  control-type PDU
+
+      PDU-specific qualifiers:  DataDescriptorSense
+
+   The iSCSI layer MUST invoke the Send_Control Operational Primitive
+   qualified with DataDescriptorSense, which defines the buffer
+   containing the sense and iSCSI Event information.  The iSER layer
+   MUST use a SendSE Message to send the Asynchronous Message PDU.
+
+7.3.8.  Text Request and Text Response
+
+      Type:  control-type PDU
+
+      PDU-specific qualifiers:  DataDescriptorTextOut (for Text
+      Request), DataDescriptorIn (for Text Response)
+
+   The iSCSI layer MUST invoke the Send_Control Operational Primitive
+   qualified with DataDescriptorTextOut (or DataDescriptorIn), which
+   defines the Text Request (or Text Response) buffer.  The iSER layer
+   MUST use SendSE Messages to send the Text Request (or Text Response
+   PDUs).
+
+7.3.9.  Login Request and Login Response
+
+   During the login negotiation, the iSCSI layer interacts with the
+   transport layer directly and the iSER layer is not involved.  See
+   Section 5.1 on iSCSI/iSER connection setup.  If the underlying
+   transport is TCP, the Login Request PDUs and the Login Response PDUs
+   are exchanged when the connection between the initiator and the
+   target is still in the byte stream mode.
+
+   The iSCSI layer MUST not send a Login Request (or a Login Response)
+   PDU during the Full Feature Phase.  A Login Request (or a Login
+   Response) PDU, if used, MUST be treated as an iSCSI protocol error.
+   The iSER layer MAY reject such a PDU from the iSCSI layer with an
+   appropriate error code.  If a Login Request PDU is received by the
+   iSCSI layer at the target, it MUST respond with a Reject PDU with a
+   reason code of "protocol error".
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 50]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+7.3.10.  Logout Request and Logout Response
+
+      Type:  control-type PDU
+
+      PDU-specific qualifiers:  None
+
+   The iSER layer MUST use a SendSE Message to send the Logout Request
+   or Logout Response PDU.  Sections 5.2.1 and 5.2.2 describe the
+   handling of the Logout Request and the Logout Response at the
+   initiator and the target and the interactions between the initiator
+   and the target to terminate a connection.
+
+7.3.11.  SNACK Request
+
+   Since HeaderDigest and DataDigest must be negotiated to "None", there
+   are no digest errors when the connection is in iSER-assisted mode.
+   Also, since RCaP delivers all messages in the order they were sent,
+   there are no sequence errors when the connection is in iSER-assisted
+   mode.  Therefore, the iSCSI layer MUST NOT send SNACK Request PDUs.
+   A SNCAK Request PDU, if used, MUST be treated as an iSCSI protocol
+   error.  The iSER layer MAY reject such a PDU from the iSCSI layer
+   with an appropriate error code.  If a SNACK Request PDU is received
+   by the iSCSI layer at the target, it MUST respond with a Reject PDU
+   with a reason code of "protocol error".
+
+7.3.12.  Reject
+
+      Type:  control-type PDU
+
+      PDU-specific qualifiers:  DataDescriptorReject
+
+   The iSCSI layer MUST invoke the Send_Control Operational Primitive
+   qualified with DataDescriptorReject, which defines the Reject buffer.
+   The iSER layer MUST use a SendSE Message to send the Reject PDU.
+
+7.3.13.  NOP-Out and NOP-In
+
+      Type:  control-type PDU
+
+      PDU-specific qualifiers:  DataDescriptorNOPOut (for NOP-Out),
+      DataDescriptorNOPIn (for NOP-In)
+
+   The iSCSI layer MUST invoke the Send_Control Operational Primitive
+   qualified with DataDescriptorNOPOut (or DataDescriptorNOPIn), which
+   defines the Ping (or Return Ping) data buffer.  The iSER layer MUST
+   use SendSE Messages to send the NOP-Out (or NOP-In) PDU.
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 51]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+8.  Flow Control and STag Management
+
+8.1.  Flow Control for RDMA Send Message Types
+
+   Send Message Types in RCaP are used by the iSER layer to transfer
+   iSCSI control-type PDUs.  Each Send Message Type in RCaP consumes an
+   Untagged Buffer at the Data Sink.  However, neither the RCaP layer
+   nor the iSER layer provides an explicit flow control mechanism for
+   the Send Message Types.  Therefore, the iSER layer SHOULD provision
+   enough Untagged buffers for handling incoming Send Message Types to
+   prevent buffer exhaustion at the RCaP layer.  If buffer exhaustion
+   occurs, it may result in the termination of the connection.
+
+   An implementation may choose to satisfy the buffer requirement by
+   using a common buffer pool shared across multiple connections, with
+   usage limits on a per-connection basis and usage limits on the buffer
+   pool itself.  In such an implementation, exceeding the buffer usage
+   limit for a connection or the buffer pool itself may trigger
+   interventions from the iSER layer to replenish the buffer pool and/or
+   to isolate the connection causing the problem.
+
+   iSER also provides the MaxOutstandingUnexpectedPDUs key to be used by
+   the initiator and the target to declare the maximum number of
+   outstanding "unexpected" control-type PDUs that it can receive.  It
+   is intended to allow the receiving side to determine the amount of
+   buffer resources needed beyond the normal flow control mechanism
+   available in iSCSI.
+
+   The buffer resources required at both the initiator and the target as
+   a result of control-type PDUs sent by the initiator is described in
+   Section 8.1.1.  The buffer resources required at both the initiator
+   and target as a result of control-type PDUs sent by the target is
+   described in Section 8.1.2.
+
+8.1.1.  Flow Control for Control-Type PDUs from the Initiator
+
+   The control-type PDUs that can be sent by an initiator to a target
+   can be grouped into the following categories:
+
+   1.  Regulated:  Control-type PDUs in this category are regulated by
+       the iSCSI CmdSN window mechanism and the immediate flag is not
+       set.
+
+   2.  Unregulated but Expected:  Control-type PDUs in this category are
+       not regulated by the iSCSI CmdSN window mechanism but are
+       expected by the target.
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 52]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   3.  Unregulated and Unexpected:  Control-type PDUs in this category
+       are not regulated by the iSCSI CmdSN window mechanism and are
+       "unexpected" by the target.
+
+8.1.1.1.  Control-Type PDUs from the Initiator in the Regulated Category
+
+   Control-type PDUs that can be sent by the initiator in this category
+   are regulated by the iSCSI CmdSN window mechanism and the immediate
+   flag is not set.
+
+   The queuing capacity required of the iSCSI layer at the target is
+   described in Section 3.2.2.1 of [RFC3720].  For each of the control-
+   type PDUs that can be sent by the initiator in this category, the
+   initiator MUST provision for the buffer resources required for the
+   corresponding control-type PDU sent as a response from the target.
+   The following is a list of the PDUs that can be sent by the initiator
+   and the PDUs that are sent by the target in response:
+
+       a.  When an initiator sends a SCSI Command PDU, it expects a SCSI
+           Response PDU from the target.
+
+       b.  When the initiator sends a Task Management Function Request
+           PDU, it expects a Task Management Function Response PDU from
+           the target.
+
+       c.  When the initiator sends a Text Request PDU, it expects a
+           Text Response PDU from the target.
+
+       d.  When the initiator sends a Logout Request PDU, it expects a
+           Logout Response PDU from the target.
+
+       e.  When the initiator sends a NOP-Out PDU as a ping request with
+           ITT != 0xffffffff and TTT = 0xffffffff, it expects a NOP-In
+           PDU from the target with the same ITT and TTT as in the ping
+           request.
+
+   The response from the target for any of the PDUs enumerated here may
+   alternatively be in the form of a Reject PDU sent instead before the
+   task is active, as described in Section 6.3 of [RFC3720].
+
+8.1.1.2.  Control-Type PDUs from the Initiator in the Unregulated but
+          Expected Category
+
+   For the control-type PDUs in the Unregulated but Expected category,
+   the amount of buffering resources required at the target can be
+   predetermined.  The following is a list of the PDUs in this category:
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 53]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+       a.  SCSI Data-out PDUs are used by the initiator to send
+           unsolicited data.  The amount of buffer resources required by
+           the target can be determined using FirstBurstLength.  Note
+           that SCSI Data-out PDUs are not used for solicited data since
+           the R2T PDU that is used for solicitation is transformed into
+           RDMA Read operations by the iSER layer at the target.  See
+           Section 7.3.4.
+
+       b.  A NOP-Out PDU with TTT != 0xffffffff is sent as a ping
+           response by the initiator to the NOP-In PDU sent as a ping
+           request by the target.
+
+8.1.1.3.  Control-Type PDUs from the Initiator in the Unregulated and
+          Unexpected Category
+
+   PDUs in the Unregulated and Unexpected category are PDUs with the
+   immediate flag set.  The number of PDUs in this category that can be
+   sent by an initiator is controlled by the value of
+   MaxOutstandingUnexpectedPDUs declared by the target (see Section
+   6.7).  After a PDU in this category is sent by the initiator, it is
+   outstanding until it is retired.  At any time, the number of
+   outstanding unexpected PDUs MUST not exceed the value of
+   MaxOutstandingUnexpectedPDUs declared by the target.
+
+   The target uses the value of MaxOutstandingUnexpectedPDUs that it
+   declared to determine the amount of buffer resources required for
+   control-type PDUs in this category that can be sent by an initiator.
+   For the initiator, for each of the control-type PDUs that can be sent
+   in this category, the initiator MUST provision for the buffer
+   resources if required for the corresponding control-type PDU that can
+   be sent as a response from the target.
+
+   An outstanding PDU in this category is retired as follows.  If the
+   CmdSN of the PDU sent by the initiator in this category is x, the PDU
+   is outstanding until the initiator sends a non-immediate control-type
+   PDU on the same connection with CmdSN = y (where y is at least x) and
+   the target responds with a control-type PDU on any connection where
+   ExpCmdSN is at least y+1.
+
+   When the number of outstanding unexpected control-type PDUs equals
+   MaxOutstandingUnexpectedPDUs, the iSCSI layer at the initiator MUST
+   NOT generate any unexpected PDUs that otherwise it would have
+   generated, even if it is intended for immediate delivery.
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 54]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+8.1.2.  Flow Control for Control-Type PDUs from the Target
+
+   Control-type PDUs that can be sent by a target and are expected by
+   the initiator are listed in the Regulated category (see Section
+   8.1.1.1).
+
+   For the control-type PDUs that can be sent by a target and are
+   unexpected by the initiator, the number is controlled by
+   MaxOutstandingUnexpectedPDUs declared by the initiator (see Section
+   6.7).  After a PDU in this category is sent by a target, it is
+   outstanding until it is retired.  At any time, the number of
+   outstanding unexpected PDUs MUST not exceed the value of
+   MaxOutstandingUnexpectedPDUs declared by the initiator.  The
+   initiator uses the value of MaxOutstandingUnexpectedPDUs that it
+   declared to determine the amount of buffer resources required for
+   control-type PDUs in this category that can be sent by a target.  The
+   following is a list of the PDUs in this category and the conditions
+   for retiring the outstanding PDU:
+
+       a.  For an Asynchronous Message PDU with StatSN = x, the PDU is
+           outstanding until the initiator sends a control-type PDU with
+           ExpStatSN set to at least x+1.
+
+       b.  For a Reject PDU with StatSN = x that is sent after a task is
+           active, the PDU is outstanding until the initiator sends a
+           control-type PDU with ExpStatSN set to at least x+1.
+
+       c.  For a NOP-In PDU with ITT = 0xffffffff and StatSN = x, the
+           PDU is outstanding until the initiator responds with a
+           control-type PDU on the same connection where ExpStatSN is at
+           least x+1.  But if the NOP-In PDU is sent as a ping request
+           with TTT != 0xffffffff, the PDU can also be retired when the
+           initiator sends a NOP-Out PDU with the same ITT and TTT as in
+           the ping request.  Note that when a target sends a NOP-In PDU
+           as a ping request, it must provision a buffer for the NOP-Out
+           PDU sent as a ping response from the initiator.
+
+   When the number of outstanding unexpected control-type PDUs equals
+   MaxOutstandingUnexpectedPDUs, the iSCSI layer at the target MUST NOT
+   generate any unexpected PDUs that otherwise it would have generated,
+   even if its intent is to indicate an iSCSI error condition (e.g.,
+   Asynchronous Message, Reject).  Task timeouts, as in the initiator
+   waiting for a command completion or other connection and session
+   level exceptions, will ensure that correct operational behavior will
+   result in these cases despite not generating the PDU.  This rule
+   overrides any other requirements elsewhere that require that a Reject
+   PDU MUST be sent.
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 55]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   (Implementation note:  A SCSI task timeout and recovery can be a
+   lengthy process and hence SHOULD be avoided by proper provisioning of
+   resources.)
+
+   (Implementation note:  To ensure that the initiator has a means to
+   inform the target that outstanding PDUs have been retired, the target
+   should reserve the last unexpected control-type PDU allowable by the
+   value of MaxOutstandingUnexpectedPDUs declared by the initiator for
+   sending a NOP-In ping request with TTT != 0xffffffff to allow the
+   initiator to return the NOP-Out ping response with the current
+   ExpStatSN.)
+
+8.2.  Flow Control for RDMA Read Resources
+
+   The total number of RDMA Read operations that can be active
+   simultaneously on an iSCSI/iSER connection depends on the amount of
+   resources allocated as declared in the iSER Hello exchange described
+   in Section 5.1.3.  Exceeding the number of RDMA Read operations
+   allowed on a connection will result in the connection being
+   terminated by the RCaP layer.  The iSER layer at the target maintains
+   the iSER-ORD to keep track of the maximum number of RDMA Read
+   Requests that can be issued by the iSER layer on a particular RCaP
+   Stream.
+
+   During connection setup (see Section 5.1), iSER-IRD is known at the
+   initiator and iSER-ORD is known at the target after the iSER layers
+   at the initiator and the target have respectively allocated the
+   connection resources necessary to support RCaP, as directed by the
+   Allocate_Connection_Resources Operational Primitive from the iSCSI
+   layer before the end of the iSCSI Login Phase.  In the Full Feature
+   Phase, the first message sent by the initiator is the iSER Hello
+   Message (see Section 9.3), which contains the value of iSER-IRD.  In
+   response to the iSER Hello Message, the target sends the iSER
+   HelloReply Message (see Section 9.4), which contains the value of
+   iSER-ORD.  The iSER layer at both the initiator and the target MAY
+   adjust (lower) the resources associated with iSER-IRD and iSER-ORD
+   respectively to match the iSER-ORD value declared in the HelloReply
+   Message.  The iSER layer at the target MUST flow control the RDMA
+   Read Request Messages to not exceed the iSER-ORD value at the target.
+
+8.3.  STag Management
+
+   An STag, as defined in [RDMAP], is an identifier of a Tagged Buffer
+   used in an RDMA operation.  The allocation and the subsequent
+   invalidation of the STags are specified in this document if the STags
+   are exposed on the wire by being Advertised in the iSER header or
+   declared in the header of an RCaP Message.
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 56]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+8.3.1.  Allocation of STags
+
+   When the iSCSI layer at the initiator invokes the Send_Control
+   Operational Primitive to request that the iSER layer at the initiator
+   process a SCSI command, zero, one, or two STags may be allocated by
+   the iSER layer.  See Section 7.3.1 for details.  The number of STags
+   allocated depends on whether the command is unidirectional or
+   bidirectional and whether or not solicited write data transfer is
+   involved.
+
+   When the iSCSI layer at the initiator invokes the Send_Control
+   Operational Primitive to request that the iSER layer at the initiator
+   process a Task Management Function Request with the TASK REASSIGN
+   function, besides allocating zero, one, or two STags, the iSER layer
+   MUST invalidate the existing STags, if any, associated with the ITT.
+   See Section 7.3.3 for details.
+
+   The iSER layer at the target allocates a local Data Sink STag when
+   the iSCSI layer at the target invokes the Get_Data Operational
+   Primitive to request that the iSER layer process an R2T PDU.  See
+   Section 7.3.6 for details.
+
+8.3.2.  Invalidation of STags
+
+   The invalidation of the STags at the initiator at the completion of a
+   unidirectional or bidirectional command when the associated SCSI
+   Response PDU is sent by the target is described in Section 7.3.2.
+
+   When a unidirectional or bidirectional command concludes without the
+   associated SCSI Response PDU being sent by the target, the iSCSI
+   layer at the initiator MUST request that the iSER layer at the
+   initiator invalidate the STags by invoking the
+   Deallocate_Task_Resources Operational Primitive qualified with ITT.
+   In response, the iSER layer at the initiator MUST locate the STag(s)
+   (if any) in the Local Mapping that associates the ITT to the local
+   STag(s).  The iSER layer at the initiator MUST invalidate the STag(s)
+   (if any) and the Local Mapping.
+
+   For an RDMA Read operation used to realize a SCSI Write data
+   transfer, the iSER layer at the target SHOULD invalidate the Data
+   Sink STag at the conclusion of the RDMA Read operation referencing
+   the Data Sink STag (to permit the immediate reuse of buffer
+   resources).
+
+   For an RDMA Write operation used to realize a SCSI Read data
+   transfer, the Data Source STag at the target is not declared to the
+   initiator and is not exposed on the wire.  Invalidation of the STag
+   is thus not specified.
+
+
+
+Ko, et al.                  Standards Track                    [Page 57]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   When a unidirectional or bidirectional command concludes without the
+   associated SCSI Response PDU being sent by the target, the iSCSI
+   layer at the target MUST request that the iSER layer at the target
+   invalidate the STags by invoking the Deallocate_Task_Resources
+   Operational Primitive qualified with ITT.  In response, the iSER
+   layer at the target MUST locate the local STag(s) (if any) in the
+   Local Mapping that associates the ITT to the local STag(s).  The iSER
+   layer at the target MUST invalidate the local STag(s) (if any) and
+   the mapping.
+
+9.  iSER Control and Data Transfer
+
+   For iSCSI data-type PDUs (see Section 7.1), the iSER layer uses RDMA
+   Read and RDMA Write operations to transfer the solicited data.  For
+   iSCSI control-type PDUs (see Section 7.2), the iSER layer uses Send
+   Message Types of RCaP.
+
+9.1.  iSER Header Format
+
+   An iSER header MUST be present in every Send Message Type of RCaP.
+   The iSER header is located in the first 12 bytes of the message
+   payload of the Send Message Type of RCaP, as shown in Figure 2.
+
+       0                   1                   2                   3
+       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      | Opcode|                  Opcode Specific Fields               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                    Opcode Specific Fields                     |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                    Opcode Specific Fields                     |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+                       Figure 2.  iSER Header Format
+
+   Opcode - Operation Code: 4 bits
+
+        The Opcode field identifies the type of iSER Messages:
+
+           0001b = iSCSI control-type PDU
+
+           0010b = iSER Hello Message
+
+           0011b = iSER HelloReply Message
+
+           All other opcodes are reserved.
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 58]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+9.2.  iSER Header Format for the iSCSI Control-Type PDU
+
+   The iSER layer uses Send Message Types of RCaP to transfer iSCSI
+   control-type PDUs (see Section 7.2).  The message payload of each of
+   the Send Message Types of RCaP used for transferring an iSER Message
+   contains an iSER Header followed by an iSCSI control-type PDU.
+
+   The iSER header in a Send Message Type of RCaP carrying an iSCSI
+   control-type PDU MUST have the format as described in Figure 3.
+
+       0                   1                   2                   3
+       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |       |W|R|                                                   |
+      | 0001b |S|S|                  Reserved                         |
+      |       |V|V|                                                   |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                        Write STag (or N/A)                    |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                         Read STag (or N/A)                    |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+          Figure 3.  iSER Header Format for iSCSI Control-Type PDU
+
+   WSV - Write STag Valid flag: 1 bit
+
+       This flag indicates the validity of the Write STag field of the
+       iSER Header.  If set to one, the Write STag field in this iSER
+       Header is valid.  If set to zero, the Write STag field in this
+       iSER Header MUST be ignored at the receiver.  The Write STag
+       Valid flag is set to one when there is solicited data to be
+       transferred for a SCSI write or bidirectional command, or when
+       there are non-immediate unsolicited and solicited data to be
+       transferred for the referenced task specified in a Task
+       Management Function Request with the TASK REASSIGN function.
+
+   RSV - Read STag Valid flag: 1 bit
+
+       This flag indicates the validity of the Read STag field of the
+       iSER Header.  If set to one, the Read STag field in this iSER
+       Header is valid.  If set to zero, the Read STag field in this
+       iSER Header MUST be ignored at the receiver.  The Read STag Valid
+       flag is set to one for a SCSI read or bidirectional command, or
+       for a Task Management Function Request with the TASK REASSIGN
+       function.
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 59]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   Write STag - Write Steering Tag: 32 bits
+
+       This field contains the Write STag when the Write STag Valid flag
+       is set to one.  For a SCSI write or bidirectional command, the
+       Write STag is used to Advertise the initiator's I/O Buffer
+       containing the solicited data.  For a Task Management Function
+       Request with the TASK REASSIGN function, the Write STag is used
+       to Advertise the initiator's I/O Buffer containing the non-
+       immediate unsolicited data and solicited data.  This Write STag
+       is used as the Data Source STag in the resultant RDMA Read
+       operation(s).  When the Write STag Valid flag is set to zero,
+       this field MUST be set to zero.
+
+   Read STag - Read Steering Tag: 32 bits
+
+       This field contains the Read STag when the Read STag Valid flag
+       is set to one.  The Read STag is used to Advertise the
+       initiator's Read I/O Buffer of a SCSI read or bidirectional
+       command, or of a Task Management Function Request with the TASK
+       REASSIGN function.  This Read STag is used as the Data Sink STag
+       in the resultant RDMA Write operation(s).  When the Read STag
+       Valid flag is zero, this field MUST be set to zero.
+
+   Reserved:
+
+       Reserved fields MUST be set to zero on transmit and MUST be
+       ignored on reception.
+
+9.3.  iSER Header Format for the iSER Hello Message
+
+   An iSER Hello Message MUST only contain the iSER header, which MUST
+   have the format as described in Figure 4.  The iSER Hello Message is
+   the first iSER Message sent on the RCaP Stream from the iSER layer at
+   the initiator to the iSER layer at the target.
+
+       0                   1                   2                   3
+       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |       |       |       |       |                               |
+      | 0010b | Rsvd  | MaxVer| MinVer|           iSER-IRD            |
+      |       |       |       |       |                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           Reserved                            |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           Reserved                            |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+            Figure 4.  iSER Header Format for iSER Hello Message
+
+
+
+Ko, et al.                  Standards Track                    [Page 60]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   MaxVer - Maximum Version: 4 bits
+
+       This field specifies the maximum version of the iSER protocol
+       supported.  It MUST be set to one to indicate the version of the
+       specification described in this document.
+
+   MinVer - Minimum Version: 4 bits
+
+       This field specifies the minimum version of the iSER protocol
+       supported.  It MUST be set to one to indicate the version of the
+       specification described in this document.
+
+   iSER-IRD: 16 bits
+
+       This field contains the value of the iSER-IRD at the initiator.
+
+   Reserved (Rsvd):
+
+       Reserved fields MUST be set to zero on transmit, and MUST be
+       ignored on reception.
+
+9.4.  iSER Header Format for the iSER HelloReply Message
+
+   An iSER HelloReply Message MUST only contain the iSER header which
+   MUST have the format as described in Figure 5.  The iSER HelloReply
+   Message is the first iSER Message sent on the RCaP Stream from the
+   iSER layer at the target to the iSER layer at the initiator.
+
+       0                   1                   2                   3
+       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |       |     |R|       |       |                               |
+      | 0011b |Rsvd |E| MaxVer| CurVer|           iSER-ORD            |
+      |       |     |J|       |       |                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           Reserved                            |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           Reserved                            |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+         Figure 5.  iSER Header Format for iSER HelloReply Message
+
+   REJ - Reject flag: 1 bit
+
+       This flag indicates whether the target is rejecting this
+       connection.  If set to one, the target is rejecting the
+       connection.
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 61]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   MaxVer - Maximum Version: 4 bits
+
+       This field specifies the maximum version of the iSER protocol
+       supported.  It MUST be set to one to indicate the version of the
+       specification described in this document.
+
+   CurVer - Current Version: 4 bits
+
+       This field specifies the current version of the iSER protocol
+       supported.  It MUST be set to one to indicate the version of the
+       specification described in this document.
+
+   iSER-ORD: 16 bits
+
+       This field contains the value of the iSER-ORD at the target.
+
+   Reserved (Rsvd):
+
+       Reserved fields MUST be set to zero on transmit, and MUST be
+       ignored on reception.
+
+9.5.  SCSI Data Transfer Operations
+
+   The iSER layer at the initiator and the iSER layer at the target
+   handle each SCSI Write, SCSI Read, and bidirectional operation as
+   described below.
+
+9.5.1.  SCSI Write Operation
+
+   The iSCSI layer at the initiator MUST invoke the Send_Control
+   Operational Primitive to request that the iSER layer at the initiator
+   send the SCSI write command.  The iSER layer at the initiator MUST
+   request that the RCaP layer transmit a SendSE Message with the
+   message payload consisting of the iSER header followed by the SCSI
+   Command PDU and immediate data (if any).  If there is solicited data,
+   the iSER layer MUST Advertise the Write STag in the iSER header of
+   the SendSE Message, as described in Section 9.2.  Upon receiving the
+   SendSE Message, the iSER layer at the target MUST notify the iSCSI
+   layer at the target by invoking the Control_Notify Operational
+   Primitive qualified with the SCSI Command PDU.  See Section 7.3.1 for
+   details on the handling of the SCSI write command.
+
+   For the non-immediate unsolicited data, the iSCSI layer at the
+   initiator MUST invoke a Send_Control Operational Primitive qualified
+   with the SCSI Data-out PDU.  Upon receiving each Send or SendSE
+   Message containing the non-immediate unsolicited data, the iSER layer
+   at the target MUST notify the iSCSI layer at the target by invoking
+   the Control_Notify Operational Primitive qualified with the SCSI
+
+
+
+Ko, et al.                  Standards Track                    [Page 62]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   Data-out PDU.  See Section 7.3.4 for details on the handling of the
+   SCSI Data-out PDU.
+
+   For the solicited data, when the iSCSI layer at the target has an I/O
+   Buffer available, it MUST invoke the Get_Data Operational Primitive
+   qualified with the R2T PDU.  See Section 7.3.6 for details on the
+   handling of the R2T PDU.
+
+   When the data transfer associated with this SCSI Write operation is
+   complete, the iSCSI layer at the target MUST invoke the Send_Control
+   Operational Primitive when it is ready to send the SCSI Response PDU.
+   Upon receiving a SendSE or SendInvSE Message containing the SCSI
+   Response PDU, the iSER layer at the initiator MUST notify the iSCSI
+   layer at the initiator by invoking the Control_Notify Operational
+   Primitive qualified with the SCSI Response PDU.  See Section 7.3.2
+   for details on the handling of the SCSI Response PDU.
+
+9.5.2.  SCSI Read Operation
+
+   The iSCSI layer at the initiator MUST invoke the Send_Control
+   Operational Primitive to request that the iSER layer at the initiator
+   to send the SCSI read command.  The iSER layer at the initiator MUST
+   request that the RCaP layer transmit a SendSE Message with the
+   message payload consisting of the iSER header followed by the SCSI
+   Command PDU.  The iSER layer at the initiator MUST Advertise the Read
+   STag in the iSER header of the SendSE Message, as described in
+   Section 9.2.  Upon receiving the SendSE Message, the iSER layer at
+   the target MUST notify the iSCSI layer at the target by invoking the
+   Control_Notify Operational Primitive qualified with the SCSI Command
+   PDU.  See Section 7.3.1 for details on the handling of the SCSI read
+   command.
+
+   When the requested SCSI data is available in the I/O Buffer, the
+   iSCSI layer at the target MUST invoke the Put_Data Operational
+   Primitive qualified with the SCSI Data-in PDU.  See Section 7.3.5 for
+   details on the handling of the SCSI Data-in PDU.
+
+   When the data transfer associated with this SCSI Read operation is
+   complete, the iSCSI layer at the target MUST invoke the Send_Control
+   Operational Primitive when it is ready to send the SCSI Response PDU.
+   Upon receiving the SendInvSE Message containing the SCSI Response
+   PDU, the iSER layer at the initiator MUST notify the iSCSI layer at
+   the initiator by invoking the Control_Notify Operational Primitive
+   qualified with the SCSI Response PDU.  See Section 7.3.2 for details
+   on the handling of the SCSI Response PDU.
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 63]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+9.5.3.  Bidirectional Operation
+
+   The initiator and the target handle the SCSI Write and the SCSI Read
+   portions of this bidirectional operation the same as described in
+   Sections 9.5.1 and 9.5.2, respectively.
+
+10.  iSER Error Handling and Recovery
+
+   RCaP provides the iSER layer with reliable in-order delivery.
+   Therefore, the error management needs of an iSER-assisted connection
+   are somewhat different than those of a Traditional iSCSI connection.
+
+10.1.  Error Handling
+
+   iSER error handling is described in the following sections,
+   classified loosely based on the sources of errors:
+
+   1.  Those originating at the transport layer (e.g., TCP).
+
+   2.  Those originating at the RCaP layer.
+
+   3.  Those originating at the iSER layer.
+
+   4.  Those originating at the iSCSI layer.
+
+10.1.1.  Errors in the Transport Layer
+
+   If the transport layer is TCP, then TCP packets with detected errors
+   are silently dropped by the TCP layer and result in retransmission at
+   the TCP layer.  This has no impact on the iSER layer.  However,
+   connection loss (e.g., link failure) and unexpected termination
+   (e.g., TCP graceful or abnormal close without the iSCSI Logout
+   exchanges) at the transport layer will cause the iSCSI/iSER
+   connection to be terminated as well.
+
+10.1.1.1.  Failure in the Transport Layer before RCaP Mode Is Enabled
+
+   If the connection is lost or terminated before the iSCSI layer
+   invokes the Allocate_Connection_Resources Operational Primitive, the
+   login process is terminated and no further action is required.
+
+   If the connection is lost or terminated after the iSCSI layer has
+   invoked the Allocate_Connection_Resources Operational Primitive, then
+   the iSCSI layer MUST request that the iSER layer deallocate all
+   connection resources by invoking the Deallocate_Connection_Resources
+   Operational Primitive.
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 64]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+10.1.1.2.  Failure in the Transport Layer after RCaP Mode Is Enabled
+
+   If the connection is lost or terminated after the iSCSI layer has
+   invoked the Enable_Datamover Operational Primitive, the iSER layer
+   MUST notify the iSCSI layer of the connection loss by invoking the
+   Connection_Terminate_Notify Operational Primitive.  Prior to invoking
+   the Connection_Terminate_Notify Operational Primitive, the iSER layer
+   MUST perform the actions described in Section 5.2.3.2.
+
+10.1.2.  Errors in the RCaP Layer
+
+   The RCaP layer does not have error recovery operations built in.  If
+   errors are detected at the RCaP layer, the RCaP layer will terminate
+   the RCaP Stream and the associated connection.
+
+10.1.2.1.  Errors Detected in the Local RCaP Layer
+
+   If an error is encountered at the local RCaP layer, the RCaP layer
+   MAY send a Terminate Message to the Remote Peer to report the error
+   if possible.  (For iWARP, see [RDMAP] for the list of errors where a
+   Terminate Message is sent.)  The RCaP layer is responsible for
+   terminating the connection.  After the RCaP layer notifies the iSER
+   layer that the connection is terminated, the iSER layer MUST notify
+   the iSCSI layer by invoking the Connection_Terminate_Notify
+   Operational Primitive.  Prior to invoking the
+   Connection_Terminate_Notify Operational Primitive, the iSER layer
+   MUST perform the actions described in Section 5.2.3.2.
+
+10.1.2.2.  Errors Detected in the RCaP Layer at the Remote Peer
+
+   If an error is encountered at the RCaP layer at the Remote Peer, the
+   RCaP layer at the Remote Peer may send a Terminate Message to report
+   the error if possible.  If it is unable to send the Terminate
+   Message, the connection is terminated.  This is treated the same as a
+   failure in the transport layer after RDMA is enabled as described in
+   Section 10.1.1.2.
+
+   If an error is encountered at the RCaP layer at the Remote Peer and
+   it is able to send a Terminate Message, the RCaP layer at the Remote
+   Peer is responsible for terminating the connection.  After the local
+   RCaP layer notifies the iSER layer that the connection is terminated,
+   the iSER layer MUST notify the iSCSI layer by invoking the
+   Connection_Terminate_Notify Operational Primitive.  Prior to invoking
+   the Connection_Terminate_Notify Operational Primitive, the iSER layer
+   MUST perform the actions described in Section 5.2.3.2.
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 65]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+10.1.3.  Errors in the iSER Layer
+
+   The error handling due to errors at the iSER layer is described in
+   the following sections.
+
+10.1.3.1.  Insufficient Connection Resources to Support RCaP at
+           Connection Setup
+
+   After the iSCSI layer at the initiator invokes the
+   Allocate_Connection_Resources Operational Primitive during the iSCSI
+   Login Negotiation Phase, if the iSER layer at the initiator fails to
+   allocate the connection resources necessary to support RCaP, it MUST
+   return a status of failure to the iSCSI layer at the initiator.  The
+   iSCSI layer at the initiator MUST terminate the connection as
+   described in Section 5.2.3.1.
+
+   After the iSCSI layer at the target invokes the
+   Allocate_Connection_Resources Operational Primitive during the iSCSI
+   Login Negotiation Phase, if the iSER layer at the target fails to
+   allocate the connection resources necessary to support RCaP, it MUST
+   return a status of failure to the iSCSI layer at the target.  The
+   iSCSI layer at the target MUST send a Login Response with a status
+   class of 3 (Target Error), and a status code of "0302" (Out of
+   Resources).  The iSCSI layers at the initiator and the target MUST
+   terminate the connection as described in Section 5.2.3.1.
+
+10.1.3.2.  iSER Negotiation Failures
+
+   If the RCaP or iSER related parameters declared by the initiator in
+   the iSER Hello Message are unacceptable to the iSER layer at the
+   target, the iSER layer at the target MUST set the Reject (REJ) flag,
+   as described in Section 9.4, in the iSER HelloReply Message.  The
+   following are the cases when the iSER layer MUST set the REJ flag to
+   one in the HelloReply Message:
+
+   *  The initiator-declared iSER-IRD value is greater than 0 and the
+      target-declared iSER-ORD value is 0.
+
+   *  The initiator-supported and the target-supported iSER protocol
+      versions do not overlap.
+
+   After requesting that the RCaP layer send the iSER HelloReply
+   Message, the handling of the error situation is the same as that for
+   iSER format errors as described in Section 10.1.3.3.
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 66]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+10.1.3.3.  iSER Format Errors
+
+   The following types of errors in an iSER header are considered format
+   errors:
+
+   *  Illegal contents of any iSER header field
+
+   *  Inconsistent field contents in an iSER header
+
+   *  Length error for an iSER Hello or HelloReply Message (see Section
+      9.3 and 9.4)
+
+   When a format error is detected, the following events MUST occur in
+   the specified sequence:
+
+   1.  The iSER layer MUST request that the RCaP layer terminate the
+       RCaP Stream.  The RCaP layer MUST terminate the associated
+       connection.
+
+   2.  The iSER layer MUST notify the iSCSI layer of the connection
+       termination by invoking the Connection_Terminate_Notify
+       Operational Primitive.  Prior to invoking the
+       Connection_Terminate_Notify Operational Primitive, the iSER layer
+       MUST perform the actions described in Section 5.2.3.2.
+
+10.1.3.4.  iSER Protocol Errors
+
+   The first iSER Message sent by the iSER layer at the initiator after
+   transitioning into iSER-assisted mode MUST be the iSER Hello Message
+   (see Section 9.3).  Likewise, the first iSER Message sent by the iSER
+   layer at the target after transitioning into iSER-assisted mode MUST
+   be the iSER HelloReply Message (see Section 9.4).  Failure to send
+   the iSER Hello or HelloReply Message, as indicated by the wrong
+   Opcode in the iSER header, is a protocol error.  The handling of this
+   error situation is the same as that for iSER format errors as
+   described in Section 10.1.3.3.
+
+   If the sending side of an iSER-enabled connection acts in a manner
+   not permitted by the negotiated or declared login/text operational
+   key values as described in Section 6, this is a protocol error, and
+   the receiving side MAY handle this the same as for iSER format errors
+   as described in Section 10.1.3.3.
+
+10.1.4.  Errors in the iSCSI Layer
+
+   The error handling due to errors at the iSCSI layer is described in
+   the following sections.  For error recovery, see Section 10.2.
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 67]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+10.1.4.1.  iSCSI Format Errors
+
+   When an iSCSI format error is detected, the iSCSI layer MUST request
+   that the iSER layer terminate the RCaP Stream by invoking the
+   Connection_Terminate Operational Primitive.  For more details on the
+   connection termination, see Section 5.2.3.1.
+
+10.1.4.2.  iSCSI Digest Errors
+
+   In the iSER-assisted mode, the iSCSI layer will not see any digest
+   error because both the HeaderDigest and the DataDigest keys are
+   negotiated to "None".
+
+10.1.4.3.  iSCSI Sequence Errors
+
+   For Traditional iSCSI, sequence errors are caused by dropped PDUs due
+   to header or data digest errors.  Since digests are not used in
+   iSER-assisted mode and the RCaP layer will deliver all messages in
+   the order they were sent, sequence errors will not occur in iSER-
+   assisted mode.
+
+10.1.4.4.  iSCSI Protocol Error
+
+   When the iSCSI layer handles certain protocol errors by dropping the
+   connection, the error handling is the same as that for iSCSI format
+   errors as described in Section 10.1.4.1.
+
+   When the iSCSI layer uses the iSCSI Reject PDU and response codes to
+   handle certain other protocol errors, no special handling at the iSER
+   layer is required.
+
+10.1.4.5.  SCSI Timeouts and Session Errors
+
+   SCSI Timeouts and Session Errors are handled at the iSCSI layer and
+   no special handling at the iSER layer is required.
+
+10.1.4.6.  iSCSI Negotiation Failures
+
+   For negotiation failures that happen during the Login Phase at the
+   initiator after the iSCSI layer has invoked the
+   Allocate_Connection_Resources Operational Primitive and before the
+   Enable_Datamover Operational Primitive has been invoked, the iSCSI
+   layer MUST request that the iSER layer deallocate all connection
+   resources by invoking the Deallocate_Connection_Resources Operational
+   Primitive.  The iSCSI layer at the initiator MUST terminate the
+   connection.
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 68]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   For negotiation failures during the Login Phase at the target, the
+   iSCSI layer can use a Login Response with a status class other than 0
+   (success) to terminate the Login Phase.  If the iSCSI layer has
+   invoked the Allocate_Connection_Resources Operational Primitive
+   before the Enable_Datamover Operational Primitive has been invoked,
+   the iSCSI layer at the target MUST request that the iSER layer at the
+   target deallocate all connection resources by invoking the
+   Deallocate_Connection_Resources Operational Primitive.  The iSCSI
+   layer at both the initiator and the target MUST terminate the
+   connection.
+
+   During the iSCSI Login Phase, if the iSCSI layer at the initiator
+   receives a Login Response from the target with a status class other
+   than 0 (Success) after the iSCSI layer at the initiator has invoked
+   the Allocate_Connection_Resources Operational Primitive, the iSCSI
+   layer MUST request the iSER layer to deallocate all connection
+   resources by invoking the Deallocate_Connection_Resources Operational
+   Primitive.  The iSCSI layer MUST terminate the connection in this
+   case.
+
+   For negotiation failures during the Full Feature Phase, the error
+   handling is left to the iSCSI layer and no special handling at the
+   iSER layer is required.
+
+10.2.  Error Recovery
+
+   Error recovery requirements of iSCSI/iSER are the same as that of
+   Traditional iSCSI.  All three ErrorRecoveryLevels as defined in
+   [RFC3720] are supported in iSCSI/iSER.
+
+   *  For ErrorRecoveryLevel 0, session recovery is handled by iSCSI and
+      no special handling by the iSER layer is required.
+
+   *  For ErrorRecoveryLevel 1, see Section 10.2.1 on PDU Recovery.
+
+   *  For ErrorRecoveryLevel 2, see Section 10.2.2 on Connection
+      Recovery.
+
+   The iSCSI layer may invoke the Notice_Key_Values Operational
+   Primitive during connection setup to request that the iSER layer take
+   note of the value of the operational ErrorRecoveryLevel, as described
+   in Sections 5.1.1 and 5.1.2.
+
+10.2.1.  PDU Recovery
+
+   As described in Sections 10.1.4.2 and 10.1.4.3, digest and sequence
+   errors will not occur in the iSER-assisted mode.  If the RCaP layer
+   detects an error, it will close the iSCSI/iSER connection, as
+
+
+
+Ko, et al.                  Standards Track                    [Page 69]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   described in Section 10.1.2.  Therefore, PDU recovery is not useful
+   in the iSER-assisted mode.
+
+   The iSCSI layer at the initiator SHOULD disable iSCSI timeout-driven
+   PDU retransmissions.
+
+10.2.2.  Connection Recovery
+
+   The iSCSI layer at the initiator MAY reassign connection allegiance
+   for non-immediate commands that are still in progress and are
+   associated with the failed connection by using a Task Management
+   Function Request with the TASK REASSIGN function.  See Section 7.3.3
+   for more details.
+
+   When the iSCSI layer at the initiator does a task reassignment for a
+   SCSI write command, it MUST qualify the Send_Control Operational
+   Primitive invocation with DataDescriptorOut, which defines the I/O
+   Buffer for both the non-immediate unsolicited data and the solicited
+   data.  This allows the iSCSI layer at the target to use recovery R2Ts
+   to request data originally sent as unsolicited and solicited from the
+   initiator.
+
+   When the iSCSI layer at the target accepts a reassignment request for
+   a SCSI read command, it MUST request that the iSER layer process SCSI
+   Data-in for all unacknowledged data by invoking the Put_Data
+   Operational Primitive.  See Section 7.3.5 on the handling of SCSI
+   Data-in.
+
+   When the iSCSI layer at the target accepts a reassignment request for
+   a SCSI write command, it MUST request that the iSER layer process a
+   recovery R2T for any non-immediate unsolicited data and any solicited
+   data sequences that have not been received by invoking the Get_Data
+   Operational Primitive.  See Section 7.3.6 on the handling of Ready To
+   Transfer (R2T).
+
+   The iSCSI layer at the target MUST NOT issue recovery R2Ts on an
+   iSCSI/iSER connection for a task for which the connection allegiance
+   was never reassigned.  The iSER layer at the target MAY reject such a
+   recovery R2T received via the Get_Data Operational Primitive
+   invocation from the iSCSI layer at the target, with an appropriate
+   error code.
+
+   The iSER layer at the target will process the requests invoked by the
+   Put_Data and Get_Data Operational Primitives for a reassigned task in
+   the same way as for the original commands.
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 70]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+11.  Security Considerations
+
+   When iSER is layered on top of an RCaP layer and provides the RDMA
+   extensions to the iSCSI protocol, the security considerations of iSER
+   are the same as that of the underlying RCaP layer.  For iWARP, this
+   is described in [RDMAP] and [RDDPSEC].
+
+   Since the iSER-assisted iSCSI protocol is still functionally iSCSI
+   from a security considerations perspective, all of the iSCSI security
+   requirements as described in [RFC3720] and [RFC3723] apply.  If the
+   IPsec [IPSEC] mechanism is used, then it MUST be established before
+   the connection transitions to the iSER-assisted mode.  If iSER is
+   layered on top of a non-IP based RCaP layer, all the security
+   protocol mechanisms applicable to that RCaP layer are also applicable
+   to an iSCSI/iSER connection.  If iSER is layered on top of a non-IP
+   protocol, the IPsec mechanism as specified in [RFC3720] MUST be
+   implemented at any point where the iSER protocol enters the IP
+   network (e.g., via gateways), and the non-IP protocol SHOULD
+   implement (optional to use) a packet-by packet security protocol
+   equal in strength to the IPsec mechanism specified by [RFC3720].
+
+   To minimize the potential for a denial-of-service attack, the iSCSI
+   layer MUST NOT request that the iSER layer allocate the connection
+   resources necessary to support RCaP until the iSCSI layer is
+   sufficiently far along in the iSCSI Login Phase that it is reasonably
+   certain that the peer side is not an attacker, as described in
+   Sections 5.1.1 and 5.1.2.
+
+   Note that the IPsec requirements for this document are based on the
+   version of IPsec specified in RFC 2401 [IPSEC] and related RFCs, as
+   profiled by RFC 3723 [RFC3723], despite the existence of a newer
+   version of IPsec specified in RFC 4301 [RFC4301] and related RFCs.
+
+12.  References
+
+12.1.  Normative References
+
+   [RFC3720] Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and
+             E. Zeidner, "Internet Small Computer Systems Interface
+             (iSCSI)", RFC 3720, April 2004.
+
+   [RFC3723] Aboba, B., Tseng, J., Walker, J., Rangan, V., and F.
+             Travostino, "Securing Block Storage Protocols over IP", RFC
+             3723, April 2004.
+
+   [RDMAP]   Recio, R., Culley, P., Garcia, D., Hilland, J., and B.
+             Metzler, "A Remote Direct Memory Access Protocol
+             Specification", RFC 5040, October 2007.
+
+
+
+Ko, et al.                  Standards Track                    [Page 71]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   [DDP]     Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
+             Data Placement over Reliable Transports", RFC 5041, October
+             2007.
+
+   [IPSEC]   Kent, S. and R. Atkinson, "Security Architecture for the
+             Internet Protocol", RFC 2401, November 1998.
+
+   [MPA]     Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
+             Carrier, "Marker PDU Aligned Framing for TCP
+             Specification", RFC 5044, October 2007.
+
+   [RDDPSEC] Pinkerton, J. and E. Deleganes, "Direct Data Placement
+             Protocol (DDP) / Remote Direct Memory Access Protocol
+             (RDMAP) Security", RFC 5042, October 2007.
+
+   [TCP]     Postel, J., "Transmission Control Protocol", STD 7, RFC
+             793, September 1981.
+
+   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
+             Requirement Levels", BCP 14, RFC 2119, March 1997.
+
+12.2.  Informative References
+
+   [SAM2]    T10/1157D, SCSI Architecture Model - 2 (SAM-2)
+
+   [DA]      Chadalapaka, M., Hufferd, J., Satran, J., and H. Shah, "DA:
+             Datamover Architecture for the Internet Small Computer
+             System Interface (iSCSI)", RFC 5047, October 2007.
+
+   [IB]      InfiniBand Architecture Specification Volume 1 Release 1.2,
+             October 2004
+
+   [IPoIB]   Chu, J. and V. Kashyap, "Transmission of IP over InfiniBand
+             (IPoIB)", RFC 4391, April 2006.
+
+   [RFC4301] Kent, S. and K. Seo, "Security Architecture for the
+             Internet Protocol", RFC 4301, December 2005.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 72]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+Appendix A.  iWARP Message Format for iSER
+
+   This section is for information only and is NOT part of the standard.
+   It simply depicts the iWARP Message format for the various iSER
+   Messages when the transport layer is TCP.
+
+A.1.  iWARP Message Format for iSER Hello Message
+
+   The following figure depicts an iSER Hello Message encapsulated in an
+   iWARP SendSE Message.
+
+       0                   1                   2                   3
+       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |         MPA Header            |  DDP Control  | RDMA Control  |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                        Reserved                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                       (Send) Queue Number                     |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                 (Send) Message Sequence Number                |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                      (Send) Message Offset                    |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      | 0010b | Zeros | 0001b | 0001b |           iSER-IRD            |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           All Zeros                           |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           All Zeros                           |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           MPA CRC                             |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+         Figure 6.  SendSE Message Containing an iSER Hello Message
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 73]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+A.2.  iWARP Message Format for iSER HelloReply Message
+
+   The following figure depicts an iSER HelloReply Message encapsulated
+   in an iWARP SendSE Message.  The Reject (REJ) flag is set to 0.
+
+       0                   1                   2                   3
+       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |         MPA Header            |  DDP Control  | RDMA Control  |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                        Reserved                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                       (Send) Queue Number                     |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                 (Send) Message Sequence Number                |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                      (Send) Message Offset                    |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      | 0011b |Zeros|0| 0001b | 0001b |           iSER-ORD            |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           All Zeros                           |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           All Zeros                           |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           MPA CRC                             |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+       Figure 7.  SendSE Message Containing an iSER HelloReply Message
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 74]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+A.3.  iWARP Message Format for SCSI Read Command PDU
+
+   The following figure depicts a SCSI Read Command PDU embedded in an
+   iSER Message encapsulated in an iWARP SendSE Message.  For this
+   particular example, in the iSER header, the Write STag Valid flag is
+   set to zero, the Read STag Valid flag is set to one, the Write STag
+   field is set to all zeros, and the Read STag field contains a valid
+   Read STag.
+
+       0                   1                   2                   3
+       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |         MPA Header            |  DDP Control  | RDMA Control  |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                        Reserved                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                       (Send) Queue Number                     |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                 (Send) Message Sequence Number                |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                      (Send) Message Offset                    |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      | 0001b |0|1|                  All zeros                        |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                         All Zeros                             |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                         Read STag                             |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                       SCSI Read Command PDU                   |
+      //                                                             //
+      |                                                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           MPA CRC                             |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+        Figure 8.  SendSE Message Containing a SCSI Read Command PDU
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 75]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+A.4.  iWARP Message Format for SCSI Read Data
+
+   The following figure depicts an iWARP RDMA Write Message carrying
+   SCSI Read data in the payload:
+
+       0                   1                   2                   3
+       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |         MPA Header            |   DDP Control | RDMA Control  |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                       Data Sink STag                          |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                   Data Sink Tagged Offset                     |
+      +                                                               +
+      |                                                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                      SCSI Read data                           |
+      //                                                             //
+      |                                                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           MPA CRC                             |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+           Figure 9.  RDMA Write Message Containing SCSI Read Data
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 76]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+A.5.  iWARP Message Format for SCSI Write Command PDU
+
+   The following figure depicts a SCSI Write Command PDU embedded in an
+   iSER Message encapsulated in an iWARP SendSE Message.  For this
+   particular example, in the iSER header, the Write STag Valid flag is
+   set to one, the Read STag Valid flag is set to zero, the Write STag
+   field contains a valid Write STag, and the Read STag field is set to
+   all zeros since it is not used.
+
+       0                   1                   2                   3
+       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |         MPA Header            |  DDP Control  | RDMA Control  |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                        Reserved                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                       (Send) Queue Number                     |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                 (Send) Message Sequence Number                |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                      (Send) Message Offset                    |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      | 0001b |1|0|                  All zeros                        |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                        Write STag                             |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                         All Zeros                             |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                       SCSI Write Command PDU                  |
+      //                                                             //
+      |                                                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           MPA CRC                             |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+       Figure 10.  SendSE Message Containing a SCSI Write Command PDU
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 77]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+A.6.  iWARP Message Format for RDMA Read Request
+
+   An iSCSI R2T is transformed into an iWARP RDMA Read Request Message.
+   The following figure depicts an iWARP RDMA Read Request Message:
+
+       0                   1                   2                   3
+       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |         MPA Header            |  DDP Control  | RDMA Control  |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                      Reserved (Not Used)                      |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |              DDP (RDMA Read Request) Queue Number             |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |        DDP (RDMA Read Request) Message Sequence Number        |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |             DDP (RDMA Read Request) Message Offset            |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                     Data Sink STag (SinkSTag)                 |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                                                               |
+      +                  Data Sink Tagged Offset (SinkTO)             +
+      |                                                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                  RDMA Read Message Size (RDMARDSZ)            |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                     Data Source STag (SrcSTag)                |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                                                               |
+      +                 Data Source Tagged Offset (SrcTO)             +
+      |                                                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           MPA CRC                             |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+                   Figure 11.  RDMA Read Request Message
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 78]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+A.7.  iWARP Message Format for Solicited SCSI Write Data
+
+   The following figure depicts an iWARP RDMA Read Response Message
+   carrying the solicited SCSI Write data in the payload:
+
+       0                   1                   2                   3
+       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |         MPA Header            |  DDP Control  | RDMA Control  |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                       Data Sink STag                          |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                   Data Sink Tagged Offset                     |
+      +                                                               +
+      |                                                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                       SCSI Write Data                         |
+      //                                                             //
+      |                                                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           MPA CRC                             |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+      Figure 12.  RDMA Read Response Message Containing SCSI Write Data
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 79]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+A.8.  iWARP Message Format for SCSI Response PDU
+
+   The following figure depicts a SCSI Response PDU embedded in an iSER
+   Message encapsulated in an iWARP SendInvSE Message:
+
+       0                   1                   2                   3
+       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |         MPA Header            |  DDP Control  | RDMA Control  |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                      Invalidate STag                          |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                       (Send) Queue Number                     |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                 (Send) Message Sequence Number                |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                      (Send) Message Offset                    |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      | 0001b |0|0|                  All Zeros                        |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           All Zeros                           |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           All Zeros                           |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                       SCSI Response PDU                       |
+      //                                                             //
+      |                                                               |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+      |                           MPA CRC                             |
+      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+         Figure 13.  SendInvSE Message Containing SCSI Response PDU
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 80]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+Appendix B.  Architectural Discussion of iSER over InfiniBand
+
+   This section explains how an InfiniBand network (with Gateways) would
+   be structured.  It is informational only and is intended to provide
+   insight on how iSER is used in an InfiniBand environment.
+
+B.1.  The Host Side of the iSCSI and iSER Connections in InfiniBand
+
+   Figure 14 defines the topologies in which iSCSI and iSER will be able
+   to operate on an InfiniBand Network.
+
+   +---------+ +---------+ +---------+ +---------+ +--- -----+
+   |  Host   | |  Host   | |   Host  | |   Host  | |   Host  |
+   |         | |         | |         | |         | |         |
+   +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+
+   |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA|
+   +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+
+     |----+------|-----+-----|-----+-----|-----+-----|-----+---> To IB
+   IB|        IB |        IB |        IB |        IB |    SubNet2 SWTCH
+   +-v-----------v-----------v-----------v-----------v---------+
+   |                  InfiniBand Switch for Subnet1            |
+   +---+-----+--------+-----+--------+-----+------------v------+
+       | TCA |        | TCA |        | TCA |            |
+       +-----+        +-----+        +-----+            | IB
+      /  IB   \      /  IB   \      /       \     +--+--v--+--+
+     |  iSER   |    |  iSER   |    |  IPoIB  |    |  | TCA |  |
+     | Gateway |    | Gateway |    | Gateway |    |  +-----+  |
+     |   to    |    |   to    |    |   to    |    | Storage   |
+     |  iSCSI  |    |  iSER   |    |   IP    |    | Controller|
+     |   TCP   |    |  iWARP  |    |Ethernet |    +-----+-----+
+     +---v-----|    +---v-----|    +----v----+
+         | EN           | EN            | EN
+         +--------------+---------------+----> to IP based storage
+           Ethernet links that carry iSCSI or iWARP
+
+                   Figure 14.  iSCSI and iSER on IB
+
+   In Figure 14, the Host systems are connected via the InfiniBand Host
+   Channel Adapters (HCAs) to the InfiniBand links.  With the use of IB
+   switch(es), the InfiniBand links connect the HCA to InfiniBand Target
+   Channel Adapters (TCAs) located in gateways or Storage Controllers.
+   An iSER-capable IB-IP Gateway converts the iSER Messages encapsulated
+   in IB protocols to either standard iSCSI, or iSER Messages for iWARP.
+   An [IPoIB] Gateway converts the InfiniBand [IPoIB] protocol to IP
+   protocol, and in the iSCSI case, permits iSCSI to be operated on an
+   IB Network between the Hosts and the [IPoIB] Gateway.
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 81]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+B.2.  The Storage Side of the iSCSI and iSER Mixed Network Environment
+
+   Figure 15 shows a storage controller that has three different portal
+   groups: one supporting only iSCSI (TPG-4), one supporting iSER/iWARP
+   or iSCSI (TPG-2), and one supporting iSER/IB (TPG-1).
+
+                  |                |                |
+                  |                |                |
+            +--+--v--+----------+--v--+----------+--v--+--+
+            |  | IB  |          |iWARP|          | EN  |  |
+            |  |     |          | TCP |          | NIC |  |
+            |  |(TCA)|          | RNIC|          |     |  |
+            |  +-----|          +-----+          +-----+  |
+            |   TPG-1            TPG-2            TPG-4   |
+            |  9.1.3.3          9.1.2.4          9.1.2.6  |
+            |                                             |
+            |                  Storage Controller         |
+            |                                             |
+            +---------------------------------------------+
+
+   Figure 15.  Storage Controller with TCP, iWARP, and IB Connections
+
+   The normal iSCSI portal group advertising processes (via the Service
+   Location Protocol (SLP), the Internet Storage Name Service (iSNS), or
+   SendTargets) are available to a Storage Controller.
+
+B.3.  Discovery Processes for an InfiniBand Host
+
+   An InfiniBand Host system can gather portal group IP addresses from
+   SLP, iSNS, or the SendTargets discovery processes by using TCP/IP via
+   [IPoIB].  After obtaining one or more remote portal IP addresses, the
+   Initiator uses the standard IP mechanisms to resolve the IP address
+   to a local outgoing interface and the destination hardware address
+   (Ethernet MAC or IB GID of the target or a gateway leading to the
+   target).  If the resolved interface is an [IPoIB] network interface,
+   then the target portal can be reached through an InfiniBand fabric.
+   In this case, the Initiator can establish an iSCSI/TCP or iSCSI/iSER
+   session with the Target over that InfiniBand interface, using the
+   Hardware Address (InfiniBand GID) obtained through the standard
+   Address Resolution (ARP) processes.
+
+   If more than one IP address is obtained through the discovery
+   process, the Initiator should select a Target IP address that is on
+   the same IP subnet as the Initiator, if one exists.  This will avoid
+   a potential overhead of going through a gateway when a direct path
+   exists.
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 82]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+   In addition, a user can configure manual static IP route entries if a
+   particular path to the target is preferred.
+
+B.4.  IBTA Connection Specifications
+
+   The InfiniBand Trade Association (IBTA) connection specifications are
+   outside the scope of this document, but it is expected that the IBTA
+   has or will define:
+
+   *  The iSER ServiceID.
+
+   *  A Means for permitting a Host to establish a connection with a
+      peer InfiniBand end-node, and to fall back to iSCSI/TCP over
+      [IPoIB] if that peer indicates iSER is not supported.
+
+   *  A Means for permitting the Host to establish connections with IB
+      iSER connections on storage controllers or IB iSER connected
+      Gateways in preference to [IPoIB] connected Gateways/Bridges or
+      connections to Target Storage Controllers that also accept iSCSI
+      via [IPoIB].
+
+   *  A Means for combining the IB ServiceID for iSER and the IP port
+      number such that the IB Host can use normal IB connection
+      processes, yet ensure that the iSER target peer can actually
+      connect to the required IP port number.
+
+Acknowledgments
+
+   This protocol was developed by a design team that, in addition to the
+   authors, included Dwight Barron (HP), John Carrier (formerly from
+   Adaptec), Ted Compton (EMC), Paul R. Culley (HP), Yaron Haviv
+   (Voltaire), Jeff Hilland (HP), Mike Krause (HP), Alex Nezhinsky
+   (Voltaire), Jim Pinkerton (Microsoft), Renato J. Recio (IBM), Julian
+   Satran (IBM), Tom Talpey (Network Appliance), and Jim Wendt (HP).
+   Special thanks to David Black (EMC) for his extensive review
+   comments.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 83]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+Author's Address
+
+   Mallikarjun Chadalapaka
+   Hewlett-Packard Company
+   8000 Foothills Blvd.
+   Roseville, CA 95747-5668, USA
+   Phone: +1-916-785-5621
+   EMail: cbm@rose.hp.com
+
+   Uri Elzur
+   Broadcom Corporation
+   5300 California Avenue
+   Irvine, CA 92617, USA
+   Phone: +1-949-926-6432
+   EMail: Uri@Broadcom.com
+
+   John Hufferd
+   Brocade Communications Systems, Inc.
+   1745 Technology Drive
+   San Jose, CA 95110, USA
+   Phone: +1-408-333-5244
+   EMail: jhufferd@brocade.com
+
+   Mike Ko
+   IBM Corp.
+   650 Harry Rd.
+   San Jose, CA 95120, USA
+   Phone: +1-408-927-2085
+   EMail: mako@us.ibm.com
+
+   Hemal Shah
+   Broadcom Corporation
+   5300 California Avenue
+   Irvine, CA 92617, USA
+   Phone: +1-949-926-6941
+   EMail: hemal@broadcom.com
+
+   Patricia Thaler
+   Broadcom Corporation
+   5300 California Avenue
+   Irvine, CA 92617, USA
+   Phone: +1-916-570-2707
+   EMail: pthaler@broadcom.com
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 84]
+
+RFC 5046                   iSER Specification               October 2007
+
+
+Full Copyright Statement
+
+   Copyright (C) The IETF Trust (2007).
+
+   This document is subject to the rights, licenses and restrictions
+   contained in BCP 78, and except as set forth therein, the authors
+   retain all their rights.
+
+   This document and the information contained herein are provided on an
+   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
+   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
+   THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
+   OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
+   THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
+   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
+
+Intellectual Property
+
+   The IETF takes no position regarding the validity or scope of any
+   Intellectual Property Rights or other rights that might be claimed to
+   pertain to the implementation or use of the technology described in
+   this document or the extent to which any license under such rights
+   might or might not be available; nor does it represent that it has
+   made any independent effort to identify any such rights.  Information
+   on the procedures with respect to rights in RFC documents can be
+   found in BCP 78 and BCP 79.
+
+   Copies of IPR disclosures made to the IETF Secretariat and any
+   assurances of licenses to be made available, or the result of an
+   attempt made to obtain a general license or permission for the use of
+   such proprietary rights by implementers or users of this
+   specification can be obtained from the IETF on-line IPR repository at
+   http://www.ietf.org/ipr.
+
+   The IETF invites any interested party to bring to its attention any
+   copyrights, patents or patent applications, or other proprietary
+   rights that may cover technology that may be required to implement
+   this standard.  Please address the information to the IETF at
+   ietf-ipr@ietf.org.
+
+
+
+
+
+
+
+
+
+
+
+
+Ko, et al.                  Standards Track                    [Page 85]
+