summaryrefslogtreecommitdiff
path: root/doc/rfc/rfc5047.txt
diff options
context:
space:
mode:
authorThomas Voss <mail@thomasvoss.com> 2024-11-27 20:54:24 +0100
committerThomas Voss <mail@thomasvoss.com> 2024-11-27 20:54:24 +0100
commit4bfd864f10b68b71482b35c818559068ef8d5797 (patch)
treee3989f47a7994642eb325063d46e8f08ffa681dc /doc/rfc/rfc5047.txt
parentea76e11061bda059ae9f9ad130a9895cc85607db (diff)
doc: Add RFC documents
Diffstat (limited to 'doc/rfc/rfc5047.txt')
-rw-r--r--doc/rfc/rfc5047.txt2747
1 files changed, 2747 insertions, 0 deletions
diff --git a/doc/rfc/rfc5047.txt b/doc/rfc/rfc5047.txt
new file mode 100644
index 0000000..32b97a3
--- /dev/null
+++ b/doc/rfc/rfc5047.txt
@@ -0,0 +1,2747 @@
+
+
+
+
+
+
+Network Working Group M. Chadalapaka
+Request for Comments: 5047 HP
+Category: Informational J. Hufferd
+ Brocade Inc.
+ J. Satran
+ IBM
+ H. Shah
+ Broadcom Corporation
+ October 2007
+
+
+ DA: Datamover Architecture for
+ the Internet Small Computer System Interface (iSCSI)
+
+Status of This Memo
+
+ This memo provides information for the Internet community. It does
+ not specify an Internet standard of any kind. Distribution of this
+ memo is unlimited.
+
+Abstract
+
+ The Internet Small Computer System Interface (iSCSI) is a SCSI
+ transport protocol that maps the SCSI family of application protocols
+ onto TCP/IP. Datamover Architecture for iSCSI (DA) defines an
+ abstract model in which the movement of data between iSCSI end nodes
+ is logically separated from the rest of the iSCSI protocol in order
+ to allow iSCSI to adapt to innovations available in new IP
+ transports. While DA defines the architectural functions required of
+ the class of Datamover protocols, it does not define any specific
+ Datamover protocols. Each such Datamover protocol, defined in a
+ separate document, provides a reliable transport for all iSCSI PDUs,
+ but actually moves the data required for certain iSCSI PDUs without
+ involving the remote iSCSI layer itself. This document begins with
+ an introduction of a few new abstractions, defines a layered
+ architecture for iSCSI and Datamover protocols, and then models the
+ interactions within an iSCSI end node between the iSCSI layer and the
+ Datamover layer that happen in order to transparently perform remote
+ data movement within an IP fabric. It is intended that this
+ definition will help map iSCSI to generic Remote Direct Memory Access
+ (RDMA)-capable IP fabrics in the future comprising TCP, the Stream
+ Control Transmission Protocol (SCTP), and possibly other underlying
+ network transport layers, such as InfiniBand.
+
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 1]
+
+RFC 5047 DA October 2007
+
+
+Table of Contents
+
+ 1. Motivation ......................................................4
+ 1.1. Intent .....................................................4
+ 1.2. Interpretation of Requirements .............................5
+ 2. Definitions and Acronyms ........................................5
+ 2.1. Definitions ................................................5
+ 2.2. Acronyms ...................................................6
+ 3. Architectural Layering of iSCSI and Datamover Layers ............7
+ 4. Design Overview .................................................9
+ 5. Architectural Concepts .........................................10
+ 5.1. iSCSI PDU Types ...........................................10
+ 5.1.1. iSCSI Data-Type PDUs ...............................10
+ 5.1.2. iSCSI Control-Type PDUs ............................11
+ 5.2. Data_Descriptor ...........................................11
+ 5.3. Connection_Handle .........................................11
+ 5.4. Operational Primitive .....................................12
+ 5.5. Transport Connection ......................................13
+ 6. Datamover Layer and Datamover Protocol .........................13
+ 7. Functional Overview ............................................14
+ 7.1. Startup ...................................................14
+ 7.2. Full Feature Phase ........................................15
+ 7.3. Wrap-up ...................................................15
+ 8. Operational Primitives Provided by the Datamover Layer .........16
+ 8.1. Send_Control ..............................................16
+ 8.2. Put_Data ..................................................17
+ 8.3. Get_Data ..................................................17
+ 8.4. Allocate_Connection_Resources .............................18
+ 8.5. Deallocate_Connection_Resources ...........................19
+ 8.6. Enable_Datamover ..........................................19
+ 8.7. Connection_Terminate ......................................20
+ 8.8. Notice_Key_Values .........................................20
+ 8.9. Deallocate_Task_Resources .................................20
+ 9. Operational Primitives Provided by the iSCSI Layer .............21
+ 9.1. Control_Notify ............................................21
+ 9.2. Connection_Terminate_Notify ...............................22
+ 9.3. Data_Completion_Notify ....................................22
+ 9.4. Data_ACK_Notify ...........................................23
+ 10. Datamover Interface (DI) ......................................23
+ 10.1. Overview .................................................23
+ 10.2. Interactions for Handling Asynchronous Notifications .....24
+ 10.2.1. Connection Termination ............................24
+ 10.2.2. Data Transfer Completion ..........................24
+ 10.2.3. Data Acknowledgement ..............................25
+ 10.3. Interactions for Sending an iSCSI PDU ....................25
+ 10.3.1. SCSI Command ......................................26
+ 10.3.2. SCSI Response .....................................26
+ 10.3.3. Task Management Function Request ..................26
+
+
+
+Chadalapaka, et al. Informational [Page 2]
+
+RFC 5047 DA October 2007
+
+
+ 10.3.4. Task Management Function Response .................27
+ 10.3.5. SCSI Data-Out and SCSI Data-In ....................27
+ 10.3.6. Ready To Transfer (R2T) ...........................28
+ 10.3.7. Asynchronous Message ..............................28
+ 10.3.8. Text Request ......................................28
+ 10.3.9. Text Response .....................................28
+ 10.3.10. Login Request ....................................29
+ 10.3.11. Login Response ...................................29
+ 10.3.12. Logout Command ...................................29
+ 10.3.13. Logout Response ..................................30
+ 10.3.14. SNACK Request ....................................30
+ 10.3.15. Reject ...........................................30
+ 10.3.16. NOP-Out ..........................................30
+ 10.3.17. NOP-In ...........................................30
+ 10.4. Interactions for Receiving an iSCSI PDU ..................31
+ 10.4.1. General Control-Type PDU Notification .............31
+ 10.4.2. SCSI Data Transfer PDUs ...........................31
+ 10.4.3. Login Request .....................................32
+ 10.4.4. Login Response ....................................32
+ 11. Security Considerations .......................................33
+ 11.1. Architectural Considerations .............................33
+ 11.2. Wire Protocol Considerations .............................33
+ 12. References ....................................................34
+ 12.1. Normative References .....................................34
+ 12.2. Informative References ...................................34
+ Appendix A. Design Considerations and Examples ....................35
+ A.1. Design Considerations for a Datamover Protocol ............35
+ A.2. Examples of Datamover Interactions ........................35
+ Acknowledgements ..................................................44
+
+Table of Figures
+
+ Figure 1. Datamover Architecture Diagram, with the RDMAP Example ...8
+ Figure 2. A Successful iSCSI Login on Initiator ...................37
+ Figure 3. A Successful iSCSI Login on Target ......................37
+ Figure 4. A Failed iSCSI Login on Initiator .......................38
+ Figure 5. A Failed iSCSI Login on Target ..........................38
+ Figure 6. iSCSI Does Not Enable the Datamover .....................39
+ Figure 7. A Normal iSCSI Connection Termination ...................40
+ Figure 8. An Abnormal iSCSI Connection Termination ................40
+ Figure 9. A SCSI Write Data Transfer ..............................41
+ Figure 10. A SCSI Read Data Transfer ..............................42
+ Figure 11. A SCSI Read Data Acknowledgement .......................43
+ Figure 12. Task Resource Cleanup on Abort .........................44
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 3]
+
+RFC 5047 DA October 2007
+
+
+1. Motivation
+
+1.1. Intent
+
+ There are relatively new standard protocols that enable Remote Direct
+ Memory Access (RDMA) and Remote Direct Data Placement (RDDP)
+ technologies to work over IP fabrics. The principal value
+ proposition of these technologies is that they enable one end node to
+ place data in the final intended buffer on the remote end node, thus
+ eliminating the need for a receive path data copy that moves the data
+ to its final location. The data copy avoidance in turn eliminates
+ unnecessary memory bandwidth consumption, substantially decreases the
+ reassembly buffer size requirements, and preserves CPU cycles that
+ would otherwise be spent in copying.
+
+ The iSCSI specification [RFC3720] defines a very detailed data
+ transfer model that employs SCSI Data-In PDUs, SCSI Data-Out PDUs,
+ and R2T PDUs, in addition to the SCSI Command and SCSI Response PDUs
+ that respectively create and conclude the task context for the data
+ transfer. In the traditional iSCSI model, the iSCSI protocol layer
+ plays the central role in pacing the data transfer and carrying out
+ the ensuing data transfer itself. An alternative architecture would
+ be for iSCSI to delegate a large part of this data transfer role to a
+ separate protocol layer exclusively designed to move data, which in
+ turn is possibly aided by a data movement and placement technology
+ such as RDMA.
+
+ If iSCSI were operating in such RDMA environments, iSCSI would be
+ shielded from the low-level data transfer mechanics but would only be
+ privy to the conclusion of the requested data transfer. Thus, there
+ would be an effective "off-loading" of the work that an iSCSI
+ protocol layer is expected to perform, compared to today's iSCSI end
+ nodes. For such RDMA environments, it is highly desirable that there
+ be a standard architecture to separate the data movement part of the
+ iSCSI protocol definition from the rest of the iSCSI functionality.
+ This architecture precisely defines what a Datamover layer is and
+ also describes the model of interactions between the iSCSI layer and
+ the Datamover layer (Section 6). In order to satisfy this need, this
+ document presents a Datamover Architecture for iSCSI (DA) and
+ summarizes a reasonable model for interactions between the iSCSI
+ layer and the Datamover layer for each of the iSCSI PDUs that are
+ defined in [RFC3720]. Note that while DA is motivated by the advent
+ of RDMA over TCP/IP technology, the architecture is not dependent on
+ RDMA in its design. DA is intended to be a generic architectural
+ framework for allowing different types of Datamovers based on
+ different types of RDMA and transport protocols. Adoption of this
+ model will help iSCSI proliferate into more environments.
+
+
+
+
+Chadalapaka, et al. Informational [Page 4]
+
+RFC 5047 DA October 2007
+
+
+1.2. Interpretation of Requirements
+
+ This document introduces certain architectural abstractions and
+ builds an abstract functional interface model between iSCSI and
+ Datamover protocol layers based on those abstractions. This
+ architectural style is motivated by the following desires:
+
+ a) Provide guidance to Datamover protocol designers with respect
+ to the functional boundary between iSCSI and the Datamover
+ protocols. This guidance is critical since a significant part
+ of the [RFC3720] protocol definition is left unchanged by DA
+ architecture and the iSCSI notions from [RFC3720] (e.g., tasks,
+ ITTs) are leveraged by the Datamover protocol.
+
+ b) Aid existing iSCSI implementations to rapidly adapt to DA
+ architecture, largely by leveraging the architectural
+ abstractions into implementation constructs -- e.g., functions,
+ APIs, modules.
+
+ However, note that DA architecture does not intend to impose any
+ implementation specifics per se. When a DA architectural concept
+ (e.g., Operational Primitive) is described as mandatory ("MUST") or
+ recommended ("SHOULD") of a layer (iSCSI or Datamover) in this
+ document, the intent is that an implementation respectively MUST or
+ SHOULD produce the same protocol action as what the model describes.
+ Specifically, no implementation compliance in terms of names, modules
+ or API arguments etc. is implied by this Architecture by such use of
+ [RFC2119] terms, only a functional compliance is sought.
+
+2. Definitions and Acronyms
+
+2.1. Definitions
+
+ I/O Buffer - A buffer that is used in a SCSI Read or Write operation
+ so that SCSI data may be sent from or received by the buffer.
+
+ Datamover protocol - A Datamover protocol is a data transfer wire
+ protocol for iSCSI that meets the requirements stated in Section
+ 6.
+
+ Datamover layer - A Datamover layer is a protocol layer within an end
+ node that implements the Datamover protocol.
+
+ Datamover-assisted - An iSCSI connection is said to be "Datamover-
+ assisted" when a Datamover layer is enabled for moving control and
+ data information on that iSCSI connection.
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 5]
+
+RFC 5047 DA October 2007
+
+
+ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+ "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
+ document are to be interpreted as described in [RFC2119].
+
+2.2. Acronyms
+
+ Acronym Definition
+ -------------------------------------------------------------
+
+ DA Datamover Architecture for iSCSI
+
+ DDP Direct Data Placement Protocol
+
+ DI Datamover Interface
+
+ IANA Internet Assigned Numbers Authority
+
+ IETF Internet Engineering Task Force
+
+ I/O Input - Output
+
+ IP Internet Protocol
+
+ iSCSI Internet SCSI
+
+ iSER iSCSI Extensions for RDMA
+
+ ITT Initiator Task Tag
+
+ LO Leading Only
+
+ MPA Marker PDU Aligned Framing for TCP
+
+ PDU Protocol Data Unit
+
+ RDDP Remote Direct Data Placement
+
+ RDMA Remote Direct Memory Access
+
+ R2T Ready To Transfer
+
+ R2TSN Ready To Transfer Sequence Number
+
+ RDMA Remote Direct Memory Access
+
+ RDMAP Remote Direct Memory Access Protocol
+
+ RFC Request For Comments
+
+
+
+Chadalapaka, et al. Informational [Page 6]
+
+RFC 5047 DA October 2007
+
+
+ SAM SCSI Architecture Model
+
+ SCSI Small Computer Systems Interface
+
+ SN Sequence Number
+
+ SNACK Selective Negative Acknowledgment - also
+ Sequence Number Acknowledgement for Data
+
+ TCP Transmission Control Protocol
+
+ TTT Target Transfer Tag
+
+3. Architectural Layering of iSCSI and Datamover Layers
+
+ Figure 1 illustrates an example of the architectural layering of
+ iSCSI and Datamover layers, in conjunction with a TCP/IP
+ implementation of RDMAP/DDP ([DDP]) layers in an iSCSI end node.
+ Note that RDMAP/DDP/MPA and TCP protocol layers are shown here only
+ as an example, and in reality, DA is completely oblivious to protocol
+ layers below the Datamover layer. The RDMAP/DDP/MPA protocol stack
+ provides a generic transport service with direct data placement.
+ There is no need to tailor the implementation of this protocol stack
+ to the specific ULP to benefit from these services.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 7]
+
+RFC 5047 DA October 2007
+
+
+ Initiator stack Target stack
+
+ +----------------+ SCSI application +----------------+
+ | SCSI Layer | protocols | SCSI Layer |
+ +----------------+ +----------------+
+ ^ ^
+ | |
+ v v
+ +----------------+ iSCSI protocol +----------------+
+ | iSCSI Layer | (excluding data | iSCSI Layer |
+ +----------------+ movement) +----------------+
+ ^ ^
+ -- ---+-- ---- DI (Datamover Interface)--- ----+--- ----
+ v v
+ +----------------+ a Datamover +----------------+
+ | Datamover Layer| protocol | Datamover Layer|
+ +----------------+ +----------------+
+ ^ ^
+ +-------+----------+ +---------+-----------+
+ | v | | v |
+ |+---------------+ | | +-----------------+ |
+ || RDMAP/DDP/MPA | | RDMAP/DDP/MPA | | RDMAP/DDP/MPA | |
+ || Layers | | protocols | | Layers | |
+ |+---------------+ | | +-----------------+ |
+ | ^ | | ^ |
+ | | network | | | network |
+ | | transport| | | transport |
+ | v | | v |
+ |+---------------+ | | +----------------+ |
+ || TCP Layer | | TCP protocol | | TCP Layer | |
+ |+---------------+ | | +----------------+ |
+ | ^ | | ^ |
+ +-------+----------+ +---------+-----------+
+ +------------------------------------------+
+
+ Figure 1. Datamover Architecture Diagram,
+ with the RDMAP Example
+
+ The scope of this document is limited to:
+
+ 1. Defining the notion of a Datamover layer and a Datamover
+ protocol (Section 6).
+
+ 2. Defining the functionality distribution between the iSCSI layer
+ and the Datamover layer, along with the communication model
+ between the two (Operational Primitives).
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 8]
+
+RFC 5047 DA October 2007
+
+
+ 3. Modeling the interactions between the blocks labeled as "iSCSI
+ Layer" and "Datamover Layer" in Figure 1 -- i.e., defining the
+ interface labeled "DI" in the figure -- for each defined iSCSI
+ PDU, based on the Operational Primitives.
+
+4. Design Overview
+
+ This document discusses and defines a model for interactions between
+ the iSCSI layer and a "Datamover layer" (see Section 6) operating
+ within an iSCSI end node, presumably communicating with one or more
+ iSCSI end nodes with similar layering. The model for interactions
+ for handling different iSCSI operations is called the "Datamover
+ Interface" (DI, Section 10), while the architecture itself is called
+ the "Datamover Architecture for iSCSI" (DA). It is likely that the
+ architecture will have implications on the Datamover wire protocols
+ as DA places certain requirements and functionality expectations on
+ the Datamover layer. However, this document itself neither defines
+ any new wire protocol for the Datamover layer, nor any potential
+ modifications to the iSCSI wire protocol to employ the Datamover
+ layer. The scope of this document is strictly limited to specifying
+ the architectural framework and the minimally required interactions
+ that happen within an iSCSI end node to leverage the Datamover layer.
+
+ The design ideas behind DA can be summarized as follows:
+
+ 1) DA defines an abstract functional interface model of the iSCSI
+ layer's interactions with a Datamover layer below -- i.e., DA
+ models the interactions between the logical "bottom" interface
+ of iSCSI and the logical "top" interface of a Datamover.
+
+ 2) DA guides the wire protocol for a Datamover layer by defining
+ the iSCSI knowledge that the Datamover layer may utilize in its
+ protocol definition (as an example, this document completely
+ limits the notion of "iSCSI session" to the iSCSI layer).
+
+ 3) DA is designed to allow implementation of the Datamover layer
+ either in hardware or in software.
+
+ 4) DA is not a wire protocol spec, but an architecture that also
+ models the interactions between iSCSI and Datamover layers
+ operating within an iSCSI end node.
+
+ 5) DA by design seeks to model the iSCSI-Datamover interactions in
+ a way that the modeling is independent of the specifics of
+ either a particular iSCSI revision or an instantiation of a
+ Datamover layer.
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 9]
+
+RFC 5047 DA October 2007
+
+
+ 6) DA introduces and relies on the notion of a defined set of
+ Operational Primitives (could be seen as entry point
+ definitions in implementation terms) provided by each layer to
+ the other to carry out the request-response interactions.
+
+ 7) DA is intended to allow Datamover protocol definitions with
+ minimal changes to existing iSCSI implementations.
+
+ 8) DA is designed to allow the iSCSI layer to completely rely on
+ the Datamover layer for all data transport needs.
+
+ 9) DA models the architecturally required minimal interactions
+ between an operational iSCSI layer and a Datamover layer to
+ realize the iSCSI-transparent data movement. There may be
+ several other interactions in a typical implementation in order
+ to bootstrap a Datamover layer (or an iSCSI layer) into
+ operation, but they are outside the scope of this document.
+
+ Note that in summary, DA is architected to support many different
+ Datamover protocols operating under the iSCSI layer. One such
+ example of a Datamover protocol is iSER [iSER].
+
+5. Architectural Concepts
+
+5.1. iSCSI PDU Types
+
+ This section defines the iSCSI PDU classification terminology, as
+ defined and used in this document. Out of the set of legal iSCSI
+ PDUs defined in [RFC3720], as we will see in Section 5.1.1, the iSCSI
+ layer does not request a SCSI Data-Out PDU carrying solicited data
+ for transmission across the Datamover Interface per this
+ architecture. For this reason, the SCSI Data-Out PDU carrying
+ solicited data is excluded in the iSCSI PDU classification we
+ introduce in this section (for SCSI Data-Out PDUs for unsolicited
+ Data, see Section 5.1.2). The rest of the legal iSCSI PDUs that may
+ be exchanged across the Datamover Interface are defined to consist of
+ two classes:
+
+ 1) iSCSI data-type PDUs
+
+ 2) iSCSI control-type PDUs
+
+5.1.1. iSCSI Data-Type PDUs
+
+ An iSCSI data-type PDU is defined as an iSCSI PDU that causes data
+ transfer, transparent to the remote iSCSI layer, to take place
+ between the peer iSCSI nodes on a Full Feature Phase iSCSI
+ connection. A data-type PDU, when requested for transmission by the
+
+
+
+Chadalapaka, et al. Informational [Page 10]
+
+RFC 5047 DA October 2007
+
+
+ sender iSCSI layer, results in the associated data transfer without
+ the participation of the remote iSCSI layer, i.e., the PDU itself is
+ not delivered as-is to the remote iSCSI layer. The following iSCSI
+ PDUs constitute the set of iSCSI data-type PDUs:
+
+ 1) SCSI Data-In PDU
+
+ 2) R2T PDU
+
+ In an iSCSI end node structured as an iSCSI layer and a Datamover
+ layer as defined in this document, the solicitation for Data-Out
+ (i.e., R2T PDU) is not delivered to the initiator iSCSI layer, per
+ the definition of an iSCSI data-type PDU. The data transfer is
+ instead performed via the mechanisms known to the Datamover layer
+ (e.g., RDMA Read). This in turn implies that a SCSI Data-Out PDU for
+ solicited data is never requested for transmission across the
+ Datamover Interface at the initiator.
+
+5.1.2. iSCSI Control-Type PDUs
+
+ Any iSCSI PDU that is not an iSCSI data-type PDU and also not a
+ solicited SCSI Data-Out PDU is defined as an iSCSI control-type PDU.
+ Specifically, note that SCSI Data-Out PDUs for unsolicited Data are
+ defined as iSCSI control-type PDUs.
+
+5.2. Data_Descriptor
+
+ A Data_Descriptor is an information element that describes an
+ iSCSI/SCSI data buffer, provided by the iSCSI layer to its local
+ Datamover layer or provided by the Datamover layer to its local iSCSI
+ layer for identifying the data associated respectively with the
+ requested or completed operation.
+
+ In implementation terms, a Data_Descriptor may be a scatter-gather
+ list describing a local buffer, the exact structure of which is
+ subject to the constraints imposed by the operating environment on
+ the local iSCSI node.
+
+5.3. Connection_Handle
+
+ A Connection_Handle is an information element that identifies the
+ particular iSCSI connection for which an inbound or outbound iSCSI
+ PDU is intended. A connection handle is unique for a given pair of
+ an iSCSI layer instance and a Datamover layer instance. The
+ Connection_Handle qualifier is used in all invocations of any
+ Operational Primitive for connection identification.
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 11]
+
+RFC 5047 DA October 2007
+
+
+ Note that the Connection_Handle is conceptually different from the
+ Connection Identifier (CID) defined by the iSCSI specification.
+ While the CID is a unique identifier of an iSCSI connection within an
+ iSCSI session, the uniqueness of the Connection_Handle extends to the
+ entire iSCSI layer instance coupled with the Datamover layer
+ instance, across possibly multiple iSCSI sessions.
+
+ In implementation terms, a Connection_Handle could be an opaque
+ identifier exchanged between the iSCSI layer and the Datamover layer
+ at the connection login time. One may also consider it to be similar
+ in scope of uniqueness to a socket identifier. The exact structure
+ and modalities of exchange of a Connection_Handle between the two
+ layers is implementation-specific.
+
+5.4. Operational Primitive
+
+ An Operational Primitive, in this document, is an abstract functional
+ interface procedure that requests another layer perform a specific
+ action on the requestor's behalf or notifies the other layer of some
+ event. The Datamover Interface between an iSCSI layer instance and a
+ Datamover layer instance within an iSCSI end node uses a set of
+ Operational Primitives to define the functional interface between the
+ two layers. Note that not every invocation of an Operational
+ Primitive may elicit a response from the requested layer. This
+ document describes the types of Operational Primitives that are
+ implicitly required and provided by the iSCSI protocol layer as
+ defined in [RFC3720], and the semantics of these Primitives.
+
+ Note that ownership of buffers and data structures is likely to be
+ exchanged between the iSCSI layer and its local Datamover layer in
+ invoking the Operational Primitives defined in this architecture.
+ The buffer management details, including how buffers are allocated
+ and released, are implementation-specific and thus are outside the
+ scope of this document.
+
+ Each Operational Primitive invocation needs a certain "information
+ context" (e.g., Connection_Handle) for performing the specific action
+ being requested. The required information context is described in
+ this document by a listing of "qualifiers" on each invocation, in the
+ style of function call arguments. There is no specific
+ implementation implied in this notation. The "qualifiers" of any
+ Operational Primitive invocation specified in this document thus
+ represent the mandatory information context that the Operational
+ Primitive invocation MUST consider in performing the action. While
+ the qualifiers are required, the method of realizing the qualifiers
+ (passed synchronously with invocation, or retrieved from task
+ context, or retrieved from shared memory etc.) is really up to the
+ implementations.
+
+
+
+Chadalapaka, et al. Informational [Page 12]
+
+RFC 5047 DA October 2007
+
+
+ When an Operational Primitive implementation is described as
+ mandatory ("MUST") or recommended ("SHOULD") of a layer (iSCSI or
+ Datamover) in this document, the intent is that an implementation
+ respectively MUST or SHOULD produce the same protocol action as what
+ the model describes.
+
+5.5. Transport Connection
+
+ The term "Transport Connection" is used in this document as a generic
+ term to represent the end-to-end logical connection as defined by the
+ underlying reliable transport protocol. For this document, all
+ instances of Transport Connection refer to a TCP connection.
+
+6. Datamover Layer and Datamover Protocol
+
+ This section introduces the notion of a "Datamover layer" and
+ "Datamover protocol" as meant in this document, and defines the
+ requirements on a Datamover protocol.
+
+ A Datamover layer is the implementation component that realizes a
+ Datamover protocol functionality in an iSCSI-capable end node in
+ communicating with other iSCSI end nodes with similar capabilities.
+ More specifically, a "Datamover layer" MUST provide the following
+ functionality and the "Datamover protocol" MUST consist of the wire
+ protocol required to realize the following functionality:
+
+ 1) guarantee that all the necessary data transfers take place when
+ the local iSCSI layer requests transmitting a command (in order
+ to complete a SCSI command, for an initiator), or
+ sending/receiving an iSCSI data sequence (in order to complete
+ part of a SCSI command for a target).
+
+ 2) transport an iSCSI control-type PDU as-is to the peer Datamover
+ layer when requested to do so by the local iSCSI layer.
+
+ 3) provide notification and delivery to the iSCSI layer upon
+ arrival of an iSCSI control-type PDU.
+
+ 4) provide an initiator-to-target data acknowledgement of SCSI
+ read data back to the target iSCSI layer, when requested.
+
+ 5) provide an asynchronous notification upon completion of a
+ requested data transfer operation that moved data without
+ involving the iSCSI layer.
+
+ 6) place the SCSI data into the I/O buffers or pick up the SCSI
+ data for transmission out of the data buffers that the iSCSI
+ layer had requested to be used for a SCSI I/O.
+
+
+
+Chadalapaka, et al. Informational [Page 13]
+
+RFC 5047 DA October 2007
+
+
+ 7) provide an error-free (i.e., must have at least the same level
+ of assurance of data integrity as the CRC32C iSCSI data
+ digest), reliable, in-order delivery transport mechanism over
+ IP networks in performing the data transfer, and asynchronously
+ notify the iSCSI layer upon iSCSI connection termination.
+
+ Note that this architecture expects that each compliant Datamover
+ protocol will define the precise means of satisfying the requirements
+ specified in this section.
+
+ In order to meet the functional requirements listed in this section,
+ certain Datamover protocols may require pre-posted buffers from the
+ local iSCSI protocol layer via mechanisms outside the scope of this
+ document. In some implementations, the absence of such buffers may
+ result in a connection failure. Datamover protocols may also realize
+ these functional requirements via methods not explicitly listed in
+ this document.
+
+7. Functional Overview
+
+ This section presents an overview of the functional interactions
+ between the iSCSI layer and the Datamover layer as intended by this
+ Architecture.
+
+7.1. Startup
+
+ The iSCSI Login Phase on an iSCSI connection occurs as defined in
+ [RFC3720]. The Architecture assumes that at the end of the Login
+ Phase, both the initiator and target, if they had so decided,
+ transition the connection to being Datamover-assisted. The precise
+ means of how an iSCSI initiator and an iSCSI target agree on having
+ the connection Datamover-assisted is defined by the Datamover
+ protocol. The only architectural requirement is that all iSCSI
+ interactions in the iSCSI Full Feature Phase MUST be Datamover-
+ assisted subject to the prior agreement, meaning that the Datamover
+ protocol is in the iSCSI-to-iSCSI communication path below the iSCSI
+ layer on either side as shown in Figure 1. DA defines the
+ Enable_Datamover Operational Primitive (Section 8.6) to bring about
+ this transition to a Datamover-assisted connection.
+
+ The Architecture also assumes that the Datamover layer may require a
+ certain number of opaque local resources for making a connection
+ Datamover-assisted. DA thus defines the
+ Allocate_Connection_Resources Operational Primitive (Section 8.4) to
+ model this interaction. This Primitive is intended to be invoked on
+ each side once the two sides decide (as previously noted) to have the
+ connection be Datamover-assisted. The expected sequence of Primitive
+ invocations is depicted in Figures 2 and 3 in Section 13.2. Figures
+
+
+
+Chadalapaka, et al. Informational [Page 14]
+
+RFC 5047 DA October 2007
+
+
+ 4, 5, and 6 illustrate how the Primitives may be employed to deal
+ with various legal login outcomes.
+
+7.2. Full Feature Phase
+
+ All iSCSI peer communication in the Full Feature Phase happens
+ through the Datamover layers if the iSCSI connection is Datamover-
+ assisted. The Architecture assumes that a Datamover layer may
+ require a certain number of opaque local resources for each new iSCSI
+ task. In the normal course of execution, these task-level resources
+ in the Datamover layer are assumed to be transparently allocated on
+ each task initiation and deallocated on the conclusion of each task
+ as appropriate. In exception scenarios however -- scenarios that do
+ not yield a SCSI Response for each task such as ABORT TASK operation
+ -- the Architecture assumes that the Datamover layer needs to be
+ notified of the individual task terminations to aid its task-level
+ resource management. DA thus defines the Deallocate_Task_Resources
+ Operational Primitive (Section 8.9) to model this task-resource
+ management. In specifying the ITT qualifier for the
+ Deallocate_Task_Resources Primitive, the Architecture further assumes
+ that the Datamover layer tracks its opaque task-level local resources
+ by the iSCSI ITT. DA also defines Send_Control (Section 8.1),
+ Put_Data (Section 8.2), Get_Data (Section 8.3),
+ Data_Completion_Notify (Section 9.3), Data_ACK_Notify (Section 9.4),
+ and Control_Notify (Section 9.1) Operational Primitives to model the
+ various Full Feature Phase interactions.
+
+ Figures 9, 10, and 11 in Section 13.2 show some Full Feature Phase
+ interactions -- SCSI Write task, SCSI Read task, and a SCSI Read Data
+ acknowledgement, respectively. Figure 12 in Section 13.2 illustrates
+ how an ABORT TASK operation can be modeled leading to deterministic
+ resource cleanup on the Datamover layer.
+
+7.3. Wrap-up
+
+ Once an iSCSI connection becomes Datamover-assisted, the connection
+ continues in that state until the end of the Full Feature Phase,
+ i.e., the termination of the connection. The Architecture assumes
+ that when a connection is normally logged out, the Datamover layer
+ needs to be notified so that its connection-level opaque resources
+ (see Section 7.1) may be freed up. DA thus defines a
+ Connection_Terminate Operational Primitive (Section 8.7) to model
+ this interaction. The Architecture further assumes that when a
+ connection termination happens without iSCSI layer's involvement
+ (e.g., TCP RST), the Datamover layer is capable of locally cleaning
+ up its task-level and connection-level resources before notifying the
+ iSCSI layer of the fact. DA thus defines the
+
+
+
+
+Chadalapaka, et al. Informational [Page 15]
+
+RFC 5047 DA October 2007
+
+
+ Connection_Terminate_Notify Operational Primitive (Section 9.2) to
+ model this interaction.
+
+ Figures 7 and 8 in Section 13.2 illustrate the interactions between
+ the iSCSI and Datamover layers in normal and unexpected connection
+ termination scenarios.
+
+8. Operational Primitives Provided by the Datamover Layer
+
+ While the iSCSI specification itself does not have a notion of
+ Operational Primitives, any iSCSI layer implementing the iSCSI
+ specification functionally requires the following Operational
+ Primitives from its Datamover layer. Thus, any Datamover protocol
+ compliant with this architecture MUST implement the Operational
+ Primitives described in this section. These Operational Primitives
+ are invoked by the iSCSI layer as appropriate. Unless otherwise
+ stated, all the following Operational Primitives may be used both on
+ the initiator side and the target side. In general programming
+ terminology, this set of Operational Primitives may be construed as
+ "down calls".
+
+ 1) Send_Control
+
+ 2) Put_Data
+
+ 3) Get_Data
+
+ 4) Allocate_Connection_Resources
+
+ 5) Deallocate_Connection_Resources
+
+ 6) Enable_Datamover
+
+ 7) Connection_Terminate
+
+ 8) Notice_Key_Values
+
+ 9) Deallocate_Task_Resources
+
+8.1. Send_Control
+
+ Input qualifiers: Connection_Handle, iSCSI PDU-specific qualifiers
+
+ Return Results: Not specified.
+
+ An iSCSI layer requests that its local Datamover layer transmit an
+ iSCSI control-type PDU to the peer iSCSI layer operating in the
+ remote iSCSI node by this Operational Primitive. The Datamover layer
+
+
+
+Chadalapaka, et al. Informational [Page 16]
+
+RFC 5047 DA October 2007
+
+
+ performs the requested operation, and may add its own protocol
+ headers in doing so. The iSCSI layer MUST NOT invoke the
+ Send_Control Operational Primitive on an iSCSI connection that is not
+ yet Datamover-assisted.
+
+ An initiator iSCSI layer requesting the transfer of a SCSI Command
+ PDU or a target iSCSI layer requesting the transfer of a SCSI
+ response PDU are examples of invoking the Send_Control Operational
+ Primitive. As Section 10.3.1 illustrates later on, the iSCSI PDU-
+ specific qualifiers in this example are: BHS and AHS,
+ DataDescriptorOut, DataDescriptorIn, ImmediateDataSize, and
+ UnsolicitedDataSize.
+
+8.2. Put_Data
+
+ Input qualifiers: Connection_Handle, contents of a SCSI Data-In PDU
+ header, Data_Descriptor, Notify_Enable
+
+ Return Results: Not specified.
+
+ An iSCSI layer requests that its local Datamover layer transmit the
+ data identified by the Data_Descriptor for the SCSI Data-In PDU to
+ the peer iSCSI layer on the remote iSCSI node by this Operational
+ Primitive. The Datamover layer performs the operation by using its
+ own protocol means, completely transparent to the remote iSCSI layer.
+ The iSCSI layer MUST NOT invoke the Put_Data Operational Primitive on
+ an iSCSI connection that is not yet Datamover-assisted.
+
+ The Notify_Enable qualifier is used to request the local Datamover
+ layer to generate or not generate the eventual local completion
+ notification to the iSCSI layer for this Put_Data invocation. For
+ detailed semantics of this qualifier, see Section 9.3.
+
+ A Put_Data Primitive may only be invoked by an iSCSI layer on the
+ target to its local Datamover layer.
+
+ A target iSCSI layer requesting the transfer of an iSCSI read data
+ sequence (also known as a read burst) is an example of invoking the
+ Put_Data Operational Primitive.
+
+8.3. Get_Data
+
+ Input qualifiers: Connection_Handle, contents of an R2T PDU,
+ Data_Descriptor, Notify_Enable
+
+ Return Results: Not specified.
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 17]
+
+RFC 5047 DA October 2007
+
+
+ An iSCSI layer requests that its local Datamover layer retrieve
+ certain data identified by the R2T PDU from the peer iSCSI layer on
+ the remote iSCSI node and place it into the buffer identified by the
+ Data_Descriptor by invoking this Operational Primitive. The
+ Datamover layer performs the operation by using its own protocol
+ means, completely transparent to the remote iSCSI layer. The iSCSI
+ layer MUST NOT invoke the Get_Data Operational Primitive on an iSCSI
+ connection that is not yet Datamover-assisted.
+
+ The Notify_Enable qualifier is used to request that the local
+ Datamover layer generate or not generate the eventual local
+ completion notification to the iSCSI layer for this Get_Data
+ invocation. For detailed semantics of this qualifier, see Section
+ 9.3.
+
+ A Get_Data Primitive may only be invoked by an iSCSI layer on the
+ target to its local Datamover layer.
+
+ A target iSCSI layer requesting the transfer of an iSCSI write data
+ sequence (also known as a write burst) is an example of invoking the
+ Get_Data Operational Primitive.
+
+8.4. Allocate_Connection_Resources
+
+ Input qualifiers: Connection_Handle[, Resource_Descriptor ]
+
+ Return Results: Status.
+
+ By invoking this Operational Primitive, an iSCSI layer requests that
+ its local Datamover layer perform all the Datamover-specific resource
+ allocations required for the Full Feature Phase of an iSCSI
+ connection. The Connection_Handle identifies the connection for
+ which the iSCSI layer is requesting resources to be allocated.
+ Allocation of these resources is a step towards eventually
+ transitioning the connection to become a Datamover-assisted iSCSI
+ connection. Note that the Datamover layer however does not allocate
+ any Datamover-specific task-level resources upon invocation of this
+ Primitive.
+
+ An iSCSI layer, in addition, optionally specifies the
+ implementation-specific resource requirements for the iSCSI
+ connection to the Datamover layer by passing an input qualifier
+ called Resource_Descriptor. The exact structure of a
+ Resource_Descriptor is implementation-dependent, and hence
+ structurally opaque to DA.
+
+ A return result of Status=success means that the
+ Allocate_Connection_Resources invocation corresponding to that
+
+
+
+Chadalapaka, et al. Informational [Page 18]
+
+RFC 5047 DA October 2007
+
+
+ Connection_Handle succeeded. If an Allocate_Connection_Resources
+ invocation is made for a Connection_Handle for which an earlier
+ invocation succeeded, the return Status must be success and the
+ request will be ignored by the Datamover layer. A return result of
+ Status=failure means that the Allocate_Connection_Resources
+ invocation corresponding to that Connection_Handle failed. There
+ MUST NOT be more than one Allocate_Connection_Resources Primitive
+ invocation outstanding for a given Connection_Handle at any time.
+
+ The iSCSI layer must invoke the Allocate_Connection_Resources
+ Primitive before the invocation of the Enable_Datamover Primitive.
+
+8.5. Deallocate_Connection_Resources
+
+ Input qualifiers: Connection_Handle
+
+ Return Results: Not specified.
+
+ By invoking this Operational Primitive, an iSCSI layer requests that
+ its local Datamover layer deallocate all the Datamover-specific
+ resources that may have been allocated earlier for the Transport
+ Connection identified by the Connection_Handle. The iSCSI layer may
+ invoke this Operational Primitive when the Datamover-specific
+ resources associated with the Connection_Handle are no longer
+ necessary (such as the Login failure of the corresponding iSCSI
+ connection).
+
+8.6. Enable_Datamover
+
+ Input qualifiers: Connection_Handle, Transport_Connection_Descriptor
+ [, Final_Login_Response_PDU]
+
+ Return Results: Not specified.
+
+ By invoking this Operational Primitive, an iSCSI layer requests that
+ its local Datamover layer assist all further iSCSI exchanges on the
+ iSCSI connection (i.e., to make the connection Datamover-assisted)
+ identified by the Connection_Handle, for which the Datamover-specific
+ resource allocation was earlier made. The iSCSI layer MUST NOT
+ invoke the Enable_Datamover Operational Primitive for an iSCSI
+ connection unless there is a corresponding prior resource allocation.
+
+ The Final_Login_Response_PDU input qualifier is applicable only for a
+ target, and contains the final Login Response that concludes the
+ iSCSI Login Phase and which must be sent as a byte stream as expected
+ by the initiator iSCSI layer. When this qualifier is used, the
+ target-Datamover layer MUST transmit this final Login Response before
+ Datamover assistance is enabled for the Transport Connection.
+
+
+
+Chadalapaka, et al. Informational [Page 19]
+
+RFC 5047 DA October 2007
+
+
+ The iSCSI layer identifies the specific Transport Connection
+ associated with the Connection_Handle to the Datamover layer by
+ specifying the Transport_Connection_Descriptor. The exact structure
+ of this Descriptor is implementation-dependent.
+
+8.7. Connection_Terminate
+
+ Input qualifiers: Connection_Handle
+
+ Return Results: Not specified.
+
+ By invoking this Operational Primitive, an iSCSI layer requests that
+ its local Datamover layer terminate the Transport Connection and
+ deallocate all the connection and task resources associated with the
+ Connection_Handle. When this Operational Primitive invocation
+ returns to the iSCSI layer, the iSCSI layer may assume the full
+ ownership of all the iSCSI-level resources, e.g., I/O Buffers,
+ associated with the connection. This Operational Primitive may be
+ invoked only with a valid Connection_Handle, and the Transport
+ Connection associated with the Connection_Handle must already be
+ Datamover-assisted.
+
+8.8. Notice_Key_Values
+
+ Input qualifiers: Connection_Handle, Number of keys, a list of Key-
+ Value pairs.
+
+ Return Results: Not specified.
+
+ By invoking this Operational Primitive, an iSCSI layer requests that
+ its local Datamover layer take note of the negotiated values of the
+ listed keys for the Transport Connection. This Operational Primitive
+ may be invoked only with a valid Connection_Handle, and the Key-Value
+ pairs MUST be the current values that were successfully agreed upon
+ by the iSCSI peers for the connection. The Datamover layer may use
+ the values of the keys to aid the Datamover operation as it deems
+ appropriate. The specific keys to be passed as input qualifiers and
+ the point(s) in time this Operational Primitive is invoked are
+ implementation-dependent.
+
+8.9. Deallocate_Task_Resources
+
+ Input qualifiers: Connection_Handle, ITT
+
+ Return Results: Not specified.
+
+ By invoking this Operational Primitive, an iSCSI layer requests that
+ its local Datamover layer deallocate all Datamover-specific resources
+
+
+
+Chadalapaka, et al. Informational [Page 20]
+
+RFC 5047 DA October 2007
+
+
+ that earlier may have been allocated for the task identified by the
+ ITT qualifier. The iSCSI layer uses this Operational Primitive
+ during exception processing when one or more active tasks are to be
+ terminated without corresponding SCSI Response PDUs. This Primitive
+ MUST be invoked for each active task terminated without a SCSI
+ Response PDU. This Primitive MUST NOT be invoked by the iSCSI layer
+ when a SCSI Response PDU normally concludes a task. When a SCSI
+ Response PDU normally concludes a task (even if the SCSI Status was
+ not a success), the Datamover layer is assumed to have automatically
+ deallocated all Datamover-specific task resources for that task.
+ Refer to Section 7.2 for a related discussion on the Architectural
+ assumptions on the task-level Datamover resource management,
+ especially with respect to when the resources are assumed to be
+ allocated.
+
+9. Operational Primitives Provided by the iSCSI Layer
+
+ While the iSCSI specification itself does not have a notion of
+ Operational Primitives, any iSCSI layer implementing the iSCSI
+ specification would have to provide the following Operational
+ Primitives to its local Datamover layer. Thus, any iSCSI protocol
+ implementation compliant with this architecture MUST implement the
+ Operational Primitives described in this section. These Operational
+ Primitives are invoked by the Datamover layer as appropriate and when
+ the iSCSI connection is Datamover-assisted. Unless otherwise stated,
+ all the following Operational Primitives may be used both on the
+ initiator side and the target side. In general programming
+ terminology, this set of Operational Primitives may be construed as
+ "up calls".
+
+ 1) Control_Notify
+
+ 2) Connection_Terminate_Notify
+
+ 3) Data_Completion_Notify
+
+ 4) Data_ACK_Notify
+
+9.1. Control_Notify
+
+ Input qualifiers: Connection_Handle, an iSCSI control-type PDU.
+
+ Return Results: Not specified.
+
+ A Datamover layer notifies its local iSCSI layer, via this
+ Operational Primitive, of the arrival of an iSCSI control-type PDU
+ from the peer Datamover layer on the remote iSCSI node. The iSCSI
+ layer processes the control-type PDU as defined in [RFC3720].
+
+
+
+Chadalapaka, et al. Informational [Page 21]
+
+RFC 5047 DA October 2007
+
+
+ A target iSCSI layer being notified of the arrival of a SCSI command
+ is an example of invoking the Control_Notify Operational Primitive.
+
+ Note that implementations may choose to describe the "iSCSI control-
+ type PDU" qualifier in this notification using a Data_Descriptor
+ (Section 5.2) and not necessarily one contiguous buffer.
+
+9.2. Connection_Terminate_Notify
+
+ Input qualifiers: Connection_Handle
+
+ Return Results: Not specified.
+
+ A Datamover layer notifies its local iSCSI layer on an unsolicited
+ termination or failure of an iSCSI connection providing the
+ Connection_Handle associated with the iSCSI Connection. The iSCSI
+ layer MUST consider the Connection_Handle to be invalid upon being so
+ notified. The iSCSI layer processes the connection termination as
+ defined in [RFC3720]. The Datamover layer MUST deallocate the
+ connection and task resources associated with the terminated
+ connection before notifying the iSCSI layer of the termination via
+ this Operational Primitive.
+
+ A target iSCSI layer is notified of an ungraceful connection
+ termination by the Datamover layer when the underlying Transport
+ Connection is torn down. Such a Connection_Terminate_Notify
+ Operational Primitive may be triggered, for example, by a TCP RESET
+ in cases where the underlying Transport Connection uses TCP.
+
+9.3. Data_Completion_Notify
+
+ Input qualifiers: Connection_Handle, ITT, SN
+
+ Return Results: Not specified.
+
+ A Datamover layer notifies its local iSCSI layer on completing the
+ retrieval of the data or upon sending the data, as requested in a
+ prior iSCSI data-type PDU, from/to the peer Datamover layer on the
+ remote iSCSI node via this Operational Primitive. The iSCSI layer
+ processes the operation as defined in [RFC3720].
+
+ SN may be either the DataSN associated with the SCSI Data-In PDU or
+ R2TSN associated with the R2T PDU depending on the SCSI operation.
+ Note that, for targets, a TTT (see [RFC3720]) could have been
+ specified instead of an SN. However, the considered choice was to
+ leave the SN to be the qualifier for two reasons -- a) it is generic
+ and applicable to initiators and targets as well as Data-In and
+ Data-Out, and b) having both SN and TTT qualifiers for the
+
+
+
+Chadalapaka, et al. Informational [Page 22]
+
+RFC 5047 DA October 2007
+
+
+ notification is considered onerous on the Datamover layer, in terms
+ of state maintenance for each completion notification. The
+ implication of this choice is that iSCSI target implementations will
+ have to adapt to using the ITT-SN tuple in associating the solicited
+ data to the appropriate task, rather than the ITT-TTT tuple for doing
+ the same.
+
+ If Notify_Enable is set in either a Put_Data or a Get_Data
+ invocation, the Datamover layer MUST invoke the
+ Data_Completion_Notify Operational Primitive upon completing that
+ requested data transfer. If the Notify_Enable was cleared in either
+ a Put_Data or a Get_Data invocation, the Datamover layer MUST NOT
+ invoke the Data_Completion_Notify Operational Primitive upon
+ completing that requested data transfer.
+
+ A Data_Completion_Notify invocation serves to notify the iSCSI layer
+ of the Put_Data or Get_Data completion, respectively. As earlier
+ noted in Sections 8.2 and 8.3, specific Datamover protocol
+ definitions may restrict the usage scope of Put_Data and Get_Data,
+ and thus implicitly the usage scope of Data_Completion_Notify.
+
+ A target iSCSI layer being notified of the retrieval of a write data
+ sequence is an example of invoking the Data_Completion_Notify
+ Operational Primitive.
+
+9.4. Data_ACK_Notify
+
+ Input qualifiers: Connection_Handle, ITT, DataSN
+
+ Return Results: Not specified.
+
+ A target Datamover layer notifies its local iSCSI layer of the
+ arrival of a previously requested data acknowledgement from the peer
+ Datamover layer on the remote (initiator) iSCSI node via this
+ Operational Primitive. The iSCSI layer processes the data
+ acknowledgement notification as defined in [RFC3720].
+
+ A target iSCSI layer being notified of the arrival of a data
+ acknowledgement for a certain SCSI Read data PDU is the only example
+ of invoking the Data_ACK_Notify Operational Primitive.
+
+10. Datamover Interface (DI)
+
+10.1. Overview
+
+ This section describes the model of interactions between iSCSI and
+ Datamover layers when the iSCSI connection is Datamover-assisted so
+ the iSCSI layer may carry out the following:
+
+
+
+Chadalapaka, et al. Informational [Page 23]
+
+RFC 5047 DA October 2007
+
+
+ - send iSCSI data-type PDUs and exchange iSCSI control-type PDUs,
+ and
+
+ - handle asynchronous notifications such as completion of data
+ sequence transfer and connection failure.
+
+ This chapter relies on the notion of Operational Primitives (Section
+ 5.4) to define DI.
+
+10.2. Interactions for Handling Asynchronous Notifications
+
+10.2.1. Connection Termination
+
+ As stated in Section 9.2, the Datamover layer notifies the iSCSI
+ layer of a failed or terminated connection via the
+ Connection_Terminate_Notify Operational Primitive. The iSCSI layer
+ MUST consider the connection unusable upon the invocation of this
+ Primitive and handle the connection termination as specified in
+ [RFC3720].
+
+10.2.2. Data Transfer Completion
+
+ As stated in Section 9.3, the Datamover layer notifies the iSCSI
+ layer of a completed data transfer operation via the
+ Data_Completion_Notify Operational Primitive. The iSCSI layer
+ processes the transfer completion as specified in [RFC3720].
+
+10.2.2.1. Completion of a Requested SCSI Data Transfer
+
+ To notify the iSCSI layer of the completion of a requested iSCSI
+ data-type PDU transfer, the Datamover layer uses the
+ Data_Completion_Notify Operational Primitive with the following input
+ qualifiers.
+
+ a) Connection_Handle.
+
+ b) ITT: Initiator Task Tag semantics as defined in [RFC3720].
+
+ c) SN: DataSN for a SCSI Data-in/Data-out PDU, and R2TSN for an
+ iSCSI R2T PDU. The semantics for both types of sequence
+ numbers are as defined in [RFC3720].
+
+ The rationale for choosing SN is explained in Section 9.3.
+
+ Every invocation of the Data_Completion_Notify Operational Primitive
+ MUST be preceded by an invocation of the Put_Data or Get_Data
+ Operational Primitive with the Notify_Enable qualifier set by the
+ iSCSI layer at an earlier point in time.
+
+
+
+Chadalapaka, et al. Informational [Page 24]
+
+RFC 5047 DA October 2007
+
+
+10.2.3. Data Acknowledgement
+
+ [RFC3720] allows the iSCSI targets to optionally solicit data
+ acknowledgement from the initiator for one or more Data-In PDUs, via
+ setting of the A-bit on a Data-In PDU. The Data_ACK_Notify
+ Operational Primitive with the following input qualifiers is used by
+ the target Datamover layer to notify the local iSCSI layer of the
+ arrival of data acknowledgement of a previously solicited iSCSI read
+ data acknowledgement. This Operational Primitive thus is applicable
+ only to iSCSI targets.
+
+ a) Connection_Handle.
+
+ b) ITT: Initiator Task Tag semantics as defined in [RFC3720].
+
+ c) DataSN: of the next SCSI Data-In PDU, which immediately follows
+ the SCSI Data-In PDU with the A-bit set to which this
+ notification corresponds, with semantics as defined in
+ [RFC3720].
+
+ Every invocation of the Data_ACK_Notify Operational Primitive MUST be
+ preceded by an invocation of the Put_Data Operational Primitive by
+ the iSCSI target layer with the A-bit set to 1 at an earlier point in
+ time.
+
+10.3. Interactions for Sending an iSCSI PDU
+
+ This section discusses the model of interactions for sending each of
+ the iSCSI PDUs defined in [RFC3720]. A Connection_Handle (see
+ Section 5.3) is assumed to qualify each of these interactions so that
+ the Datamover layer can route it to the appropriate Transport
+ Connection. The qualifying Connection_Handle is not explicitly
+ listed in the subsequent sections.
+
+ Note that the defined list of input qualifiers represents the
+ semantically required set for the Datamover layer to consider in
+ implementing the Primitive in each interaction described in this
+ section (see Section 5.4 for an elaboration). Implementations may
+ choose to deduce the qualifiers in ways that are optimized for the
+ implementation specifics. Two examples of this are:
+
+ 1. For SCSI command (Section 10.3.1), deducing the
+ ImmediateDataSize input qualifier from the DataSegmentLength
+ field of the SCSI Command PDU.
+
+ 2. For SCSI Data-Out (Section 10.3.5.1), deducing the
+ DataDescriptorOut input qualifier from the associated SCSI
+ command invocation qualifiers (assuming such state is
+
+
+
+Chadalapaka, et al. Informational [Page 25]
+
+RFC 5047 DA October 2007
+
+
+ maintained) in conjunction with BHS fields of the SCSI Data-Out
+ PDU.
+
+10.3.1. SCSI Command
+
+ The Send_Control Operational Primitive with the following input
+ qualifiers is used for requesting the transmission of a SCSI Command
+ PDU.
+
+ a) BHS and AHS, if any, of the SCSI Command PDU as defined in
+ [RFC3720].
+
+ b) DataDescriptorOut: that defines the I/O Buffer meant for Data-
+ Out for the entire command, in the case of a write or
+ bidirectional command.
+
+ c) DataDescriptorIn: that defines the I/O Buffer meant for Data-In
+ for the entire command, in the case of a read or bidirectional
+ command.
+
+ d) ImmediateDataSize: that defines the number of octets of
+ immediate unsolicited data for a write/bidirectional command.
+
+ e) UnsolicitedDataSize: that defines the number of octets of
+ immediate and non-immediate unsolicited data for a
+ write/bidirectional command.
+
+10.3.2. SCSI Response
+
+ The Send_Control Operational Primitive with the following input
+ qualifiers is used for requesting the transmission of a SCSI Response
+ PDU.
+
+ a) BHS of the SCSI Response PDU as defined in [RFC3720].
+
+ b) DataDescriptorStatus: that defines the iSCSI buffer that
+ contains the sense and response information for the command.
+
+10.3.3. Task Management Function Request
+
+ The Send_Control Operational Primitive with the following input
+ qualifiers is used for requesting the transmission of a Task
+ Management Function Request PDU.
+
+ a) BHS of the Task Management Function Request PDU as defined in
+ [RFC3720].
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 26]
+
+RFC 5047 DA October 2007
+
+
+ b) DataDescriptorOut: that defines the I/O Buffer meant for Data-
+ Out for the entire command, in the case of a write or
+ bidirectional command. (Only valid if Function="TASK REASSIGN"
+ - [RFC3720].)
+
+ c) DataDescriptorIn: that defines the I/O Buffer meant for Data-In
+ for the entire command, in the case of a read or bidirectional
+ command. (Only valid if Function="TASK REASSIGN" - [RFC3720].)
+
+10.3.4. Task Management Function Response
+
+ The Send_Control Operational Primitive with the following input
+ qualifier is used for requesting the transmission of a Task
+ Management Function Response PDU.
+
+ a) BHS of the Task Management Function Response PDU as defined in
+ [RFC3720].
+
+10.3.5. SCSI Data-Out and SCSI Data-In
+
+10.3.5.1. SCSI Data-Out
+
+ The Send_Control Operational Primitive with the following input
+ qualifiers is used by the initiator iSCSI layer for requesting the
+ transmission of a SCSI Data-Out PDU carrying the non-immediate
+ unsolicited data.
+
+ a) BHS of the SCSI Data-Out PDU as defined in [RFC3720].
+
+ b) DataDescriptorOut: that defines the I/O Buffer with the Data-
+ Out to be carried in the iSCSI data segment of the PDU.
+
+10.3.5.2. SCSI Data-In
+
+ The Put_Data Operational Primitive with the following input
+ qualifiers is used by the target iSCSI layer for requesting the
+ transmission of the data carried by a SCSI Data-In PDU.
+
+ a) BHS of the SCSI Data-In PDU as defined in [RFC3720].
+
+ b) DataDescriptorIn: that defines the I/O Buffer with the Data-In
+ being requested for transmission.
+
+
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 27]
+
+RFC 5047 DA October 2007
+
+
+10.3.6. Ready To Transfer (R2T)
+
+ The Get_Data Operational Primitive with the following input
+ qualifiers is used by the target iSCSI layer for requesting the
+ retrieval of the data as specified by the semantic content of an R2T
+ PDU.
+
+ a) BHS of the Ready To Transfer PDU as defined in [RFC3720].
+
+ b) DataDescriptorOut: that defines the I/O Buffer for the Data-Out
+ being requested for retrieval.
+
+10.3.7. Asynchronous Message
+
+ The Send_Control Operational Primitive with the following input
+ qualifiers is used for requesting the transmission of an Asynchronous
+ Message PDU.
+
+ a) BHS of the Asynchronous Message PDU as defined in [RFC3720].
+
+ b) DataDescriptorSense: that defines an iSCSI buffer that contains
+ the sense and iSCSI Event information.
+
+10.3.8. Text Request
+
+ The Send_Control Operational Primitive with the following input
+ qualifiers is used for requesting the transmission of a Text Request
+ PDU.
+
+ a) BHS of the Text Request PDU as defined in [RFC3720].
+
+ b) DataDescriptorTextOut: that defines the iSCSI Text Request
+ buffer.
+
+10.3.9. Text Response
+
+ The Send_Control Operational Primitive with the following input
+ qualifiers is used for requesting the transmission of a Text Response
+ PDU.
+
+ a) BHS of the Text Response PDU as defined in [RFC3720].
+
+ b) DataDescriptorTextIn: that defines the iSCSI Text Response
+ buffer.
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 28]
+
+RFC 5047 DA October 2007
+
+
+10.3.10. Login Request
+
+ The Send_Control Operational Primitive with the following input
+ qualifiers is used for requesting the transmission of a Login Request
+ PDU.
+
+ a) BHS of the Login Request PDU as defined in [RFC3720].
+
+ b) DataDescriptorLoginRequest: that defines the iSCSI Login
+ Request buffer.
+
+ Note that specific Datamover protocols may choose to disallow the
+ standard DA Primitives from being used for the iSCSI Login Phase.
+ When used in conjunction with such Datamover protocols, an attempt to
+ send a Login Request via the Send_Control Operational Primitive
+ invocation is clearly an error scenario, as the Login Request PDU is
+ being sent while the connection is in the iSCSI Full Feature Phase.
+ It is outside the scope of this document to specify the resulting
+ implementation behavior in this case -- [RFC3720] already defines the
+ error handling for this error scenario.
+
+10.3.11. Login Response
+
+ The Send_Control Operational Primitive with the following input
+ qualifiers is used for requesting the transmission of a Login
+ Response PDU.
+
+ a) BHS of the Login Response PDU as defined in [RFC3720].
+
+ b) DataDescriptorLoginResponse: that defines the iSCSI Login
+ Response buffer.
+
+ Note that specific Datamover protocols may choose to disallow the
+ standard DA Primitives from being used for the iSCSI Login Phase.
+ When used in conjunction with such Datamover protocols, an attempt to
+ send a Login Response via the Send_Control Operational Primitive
+ invocation is clearly an error scenario, as the Login Response PDU is
+ being sent while in the iSCSI Full Feature Phase. It is outside the
+ scope of this document to specify the resulting implementation
+ behavior in this case -- [RFC3720] already defines the error handling
+ for this error scenario.
+
+10.3.12. Logout Command
+
+ The Send_Control Operational Primitive with the following input
+ qualifier is used for requesting the transmission of a Logout Command
+ PDU.
+
+
+
+
+Chadalapaka, et al. Informational [Page 29]
+
+RFC 5047 DA October 2007
+
+
+ a) BHS of the Logout Command PDU as defined in [RFC3720].
+
+10.3.13. Logout Response
+
+ The Send_Control Operational Primitive with the following input
+ qualifier is used for requesting the transmission of a Logout
+ Response PDU.
+
+ a) BHS of the Logout Response PDU as defined in [RFC3720].
+
+10.3.14. SNACK Request
+
+ The Send_Control Operational Primitive with the following input
+ qualifier is used for requesting the transmission of a SNACK Request
+ PDU.
+
+ a) BHS of the SNACK Request PDU as defined in [RFC3720].
+
+10.3.15. Reject
+
+ The Send_Control Operational Primitive with the following input
+ qualifiers is used for requesting the transmission of a Reject PDU.
+
+ a) BHS of the Reject PDU as defined in [RFC3720].
+
+ b) DataDescriptorReject: that defines the iSCSI Reject buffer.
+
+10.3.16. NOP-Out
+
+ The Send_Control Operational Primitive with the following input
+ qualifiers is used for requesting the transmission of a NOP-Out PDU.
+
+ a) BHS of the NOP-Out PDU as defined in [RFC3720].
+
+ b) DataDescriptorNOPOut: that defines the iSCSI Ping data buffer.
+
+10.3.17. NOP-In
+
+ The Send_Control Operational Primitive with the following input
+ qualifiers is used for requesting the transmission of a NOP-In PDU.
+
+ a) BHS of the NOP-In PDU as defined in [RFC3720].
+
+ b) DataDescriptorNOPIn: that defines the iSCSI Return Ping data
+ buffer.
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 30]
+
+RFC 5047 DA October 2007
+
+
+10.4. Interactions for Receiving an iSCSI PDU
+
+ The only PDUs that are received by an iSCSI layer operating on a
+ Datamover layer are the iSCSI control-type PDUs. The Datamover layer
+ delivers the iSCSI control-type PDUs as they arrive, qualifying each
+ with the Connection_Handle (see Section 5.3) that identifies the
+ iSCSI connection for which the PDU is meant. The subsequent
+ processing of the iSCSI control-type PDUs proceeds as defined in
+ [RFC3720].
+
+10.4.1. General Control-Type PDU Notification
+
+ This sub-section describes the general mechanics applicable to
+ several control-type PDUs. The following sub-sections note
+ additional considerations for control-type PDUs that are not covered
+ in this sub-section.
+
+ The Control_Notify Operational Primitive is used to notify the iSCSI
+ layer of the arrival of the following iSCSI control-type PDUs: SCSI
+ Command, SCSI Response, Task Management Function Request, Task
+ Management Function Response, Asynchronous Message, Text Request,
+ Text Response, Logout Command, Logout Response, SNACK, Reject, NOP-
+ Out, NOP-In.
+
+10.4.2. SCSI Data Transfer PDUs
+
+10.4.2.1. SCSI Data-Out
+
+ The Control_Notify Operational Primitive is used to notify the iSCSI
+ layer of the arrival of a SCSI Data-Out PDU carrying the non-
+ immediate unsolicited data. Note however that the solicited SCSI
+ Data-Out arriving on the target does not cause a notification to the
+ iSCSI layer using the Control_Notify Primitive because the solicited
+ SCSI Data-Out was not sent by the initiator iSCSI layer as control-
+ type PDUs.
+
+10.4.2.2. SCSI Data-In
+
+ The Datamover layer does not notify the iSCSI layer of the arrival of
+ the SCSI Data-in at the initiator, because SCSI Data-in is an iSCSI
+ data-type PDU (see section 5.1). The iSCSI layer at the initiator
+ however may infer the arrival of the SCSI Data-In when it receives a
+ subsequent notification of the SCSI Response PDU via a Control_Notify
+ invocation.
+
+ While this document does not contemplate the possibility of a Data-In
+ PDU being received at the initiator iSCSI layer, specific Datamover
+ protocols may define how to deal with an unexpected inbound SCSI
+
+
+
+Chadalapaka, et al. Informational [Page 31]
+
+RFC 5047 DA October 2007
+
+
+ Data-In PDU that may result in the initiator iSCSI layer receiving
+ the Data-In PDU. This document leaves the details of handling this
+ error scenario to the specific Datamover protocols, so each may
+ define the appropriate error handling specific to the Datamover
+ environment.
+
+10.4.2.3. Ready To Transfer (R2T)
+
+ Because an R2T PDU is an iSCSI data-type PDU (see Section 5.1) that
+ is not delivered as-is to the initiator iSCSI layer, the Datamover
+ layer does not notify the iSCSI layer of the arrival of an R2T PDU.
+ When an iSCSI node sends an R2T PDU to its local Datamover layer, the
+ local and remote Datamover layers transparently bring about the data
+ transfer requested by the R2T PDU.
+
+ While this document does not contemplate the possibility of an R2T
+ PDU being received at the initiator iSCSI layer, specific Datamover
+ protocols may define how to deal with an unexpected inbound R2T PDU
+ that may result in the initiator iSCSI layer receiving the R2T PDU.
+ This document leaves the details of handling this error scenario to
+ the specific Datamover protocols, so each may define the appropriate
+ error handling specific to the Datamover environment.
+
+10.4.3. Login Request
+
+ The Control_Notify Operational Primitive is used for notifying the
+ target iSCSI layer of the arrival of a Login Request PDU. Note that
+ specific Datamover protocols may choose to disallow the standard DA
+ Primitives from being used for the iSCSI Login Phase. When used in
+ conjunction with such Datamover protocols, the arrival of a Login
+ Request necessitating the Control_Notify Operational Primitive
+ invocation is clearly an error scenario, as the Login Request PDU is
+ arriving in the iSCSI Full Feature Phase. It is outside the scope of
+ this document to specify the resulting implementation behavior in
+ this case -- [RFC3720] already defines the error handling in this
+ error scenario.
+
+10.4.4. Login Response
+
+ The Control_Notify Operational Primitive is used to notify the
+ initiator iSCSI layer of the arrival of a Login Response PDU. Note
+ that specific Datamover protocols may choose to disallow the standard
+ DA Primitives from being used for the iSCSI Login Phase. When used
+ in conjunction with such Datamover protocols, the arrival of a Login
+ Response necessitating the Control_Notify Operational Primitive
+ invocation is clearly an error scenario, as the Login Response PDU is
+ arriving in the iSCSI Full Feature Phase. It is outside the scope of
+ this document to specify the resulting implementation behavior in
+
+
+
+Chadalapaka, et al. Informational [Page 32]
+
+RFC 5047 DA October 2007
+
+
+ this case -- [RFC3720] already defines the error handling in this
+ error scenario.
+
+11. Security Considerations
+
+11.1. Architectural Considerations
+
+ DA enables compliant iSCSI implementations to realize a control and
+ data separation in the way they interact with their Datamover
+ protocols. Note however that this separation does not imply a
+ separation in transport mediums between control traffic and data
+ traffic -- the basic iSCSI architecture with respect to tasks and PDU
+ relationships to tasks remains unchanged. [RFC3720] defines several
+ MUST requirements on ordering relationships across control and data
+ for a given task besides a mandatory deterministic task allegiance
+ model -- DA does not change this basic architecture (DA has a
+ normative reference to [RFC3720]) for allow any additional
+ flexibility in compliance in this area. To summarize, sending bulk
+ data transfers (prompted by Put_Data and Get_Data Primitive
+ invocations) on a different transport medium would be as ill-advised
+ as sending just the Data-Out/Data-In PDUs on a different TCP
+ connection in RFC 3720-based iSCSI implementations. Consequently,
+ all the iSCSI-related security text in [RFC3723] is directly
+ applicable to a DA-enabled iSCSI implementation.
+
+ Another area with security implications is the Datamover connection
+ resource management model, which DA defines -- particularly the
+ Allocate_Connection_Resources Primitive. An inadvertent realization
+ of this model could leave an iSCSI implementation exposed to denial-
+ of-service attacks. As Figures 2 and 3 in Section 13.2 illustrate,
+ the most effective countermeasure to this potential attack consists
+ of performing the Datamover resource allocation when the iSCSI layer
+ is sufficiently far along in the iSCSI Login Phase that it is
+ reasonably certain that the peer side is not an attacker. In
+ particular, if the Login Phase includes a SecurityNegotiation stage,
+ an iSCSI end node MUST defer the Datamover connection resource
+ allocation (i.e., invoking the Allocate_Connection_Resources
+ Primitive) to the LoginOperationalNegotiation stage [RFC3720] so that
+ the resource allocation happens post-authentication. This
+ considerably minimizes the potential for a denial-of service attack.
+
+11.2. Wire Protocol Considerations
+
+ In view of the fact that the DA architecture itself does not define
+ any new wire protocol or propose modifications to the existing
+ protocols, there are no additional wire protocol security
+ considerations in employing DA itself. However, a DA-compliant iSCSI
+ implementation MUST comply with all the iSCSI-related requirements
+
+
+
+Chadalapaka, et al. Informational [Page 33]
+
+RFC 5047 DA October 2007
+
+
+ stipulated in [RFC3723] and [RFC3720]. Note further that in
+ realizing DA, each Datamover protocol must define and elaborate as
+ appropriate on any additional security considerations resulting from
+ the use of that Datamover protocol.
+
+ All Datamover protocol designers are strongly recommended to refer to
+ [RDDPSEC] for the types of security issues to consider. While
+ [RDDPSEC] elaborates on the security considerations applicable to an
+ RDDP-based Datamover [iSER], the document is representative of the
+ type of analysis of resource exhaustion and the application of
+ countermeasures that need to be done for any Datamover protocol.
+
+12. References
+
+12.1. Normative References
+
+ [RFC3720] Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and
+ E. Zeidner, "Internet Small Computer Systems Interface
+ (iSCSI)", RFC 3720, April 2004.
+
+ [RFC3723] Aboba, B., Tseng, J., Walker, J., Rangan, V., and F.
+ Travostino, "Securing Block Storage Protocols over IP", RFC
+ 3723, April 2004.
+
+ [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
+ Requirement Levels", BCP 14, RFC 2119, March 1997.
+
+12.2. Informative References
+
+ [DDP] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
+ Data Placement over Reliable Transports", RFC 5041, October
+ 2007.
+
+ [iSER] Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah, H.,
+ and P. Thaler, "Internet Small Computer System Interface
+ (iSCSI) Extensions for Remote Direct Memory Access (RDMA)",
+ RFC 5046, October 2007.
+
+ [RDDPSEC] Pinkerton, J. and E. Deleganes, "Direct Data Placement
+ Protocol (DDP) / Remote Direct Memory Access Protocol
+ (RDMAP) Security", RFC 5042, October 2007.
+
+
+
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 34]
+
+RFC 5047 DA October 2007
+
+
+Appendix A. Design Considerations and Examples
+
+A.1. Design Considerations for a Datamover Protocol
+
+ This section discusses the specific considerations for RDMA-based and
+ RDDP-based Datamover protocols.
+
+ a) Note that the modeling of interactions for SCSI Data-Out
+ (Section 10.3.5.1) is only used for unsolicited data transfer.
+
+ b) The modeling of interactions for SNACK (Sections 10.3.14 and
+ 10.4.1) is not expected to be used given that one of the design
+ requirements on the Datamover is that it "guarantees an error-
+ free, reliable, in-order transport mechanism" (Section 6). The
+ interactions for sending and receiving a SNACK are nevertheless
+ modeled in this document because the receiving iSCSI layer can
+ deterministically deal with an inadvertent SNACK. This also
+ shows the DA designers' intent that DI is not meant to filter
+ certain types of PDUs.
+
+ c) The onus is on a reliable Datamover (per requirements stated in
+ Section 6) to realize end-to-end data acknowledgements via
+ Datamover-specific means. In view of this, even use of data-
+ ACK-type SNACKs are unnecessary. Consequently, an initiator
+ may never request sending a SNACK Request in this model
+ assuming that the proactive (timeout-driven) SNACK
+ functionality is turned off in the legacy iSCSI code.
+
+ d) Note that the current DA model for bootstrapping a
+ Connection_Handle into service -- i.e., associating a new iSCSI
+ connection with a Connection_Handle -- clearly implies that the
+ iSCSI connection must already be in Full Feature Phase when the
+ Datamover layer comes into the stack. This further implies
+ that the iSCSI Login Phase must be carried out in the
+ traditional "Byte streaming mode" with no assistance or
+ involvement from the Datamover layer.
+
+A.2. Examples of Datamover Interactions
+
+ The figures described in this section provide some examples of the
+ usage of Operational Primitives in interactions between the iSCSI
+ layer and the Datamover layer. The following abbreviations are used
+ in this section.
+
+ Avail - Available
+
+ Abted - Aborted
+
+
+
+
+Chadalapaka, et al. Informational [Page 35]
+
+RFC 5047 DA October 2007
+
+
+ Buf - I/O Buffer
+
+ Cmd - Command
+
+ Compl - Complete
+
+ Conn - Connection
+
+ Ctrl_Ntfy - Control_Notify
+
+ Dal_Tk_Res - Deallocate_Task_Resources
+
+ Data_Cmp_Nfy - Data_Completion_Notify
+
+ Data_ACK_Nfy - Data_ACK_Notify
+
+ DM - Datamover
+
+ Imm - Immediate
+
+ Snd_Ctrl - Send_Control
+
+ Msg - Message
+
+ Resp - Response
+
+ Sol - Solicited
+
+ TMF Req - Task Management Function Request
+
+ TMF Res - Task Management Function Response
+
+ Trans - Transfer
+
+ Unsol - Unsolicited
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 36]
+
+RFC 5047 DA October 2007
+
+
+ | | Allocate_Connection_Resources | D | ^
+ | |------------------------------->| a | |
+ | | Connection resources are | t | |
+ | i | successfully allocated | a | | iSCSI
+ | S | | m | | Login
+ | C | | o | | Phase
+ | S | | v | |
+ | I | | e | |
+ | | | r | | Login Phase
+ | L | Final Login Response (success) v succeeds
+ | a |<----------------------------------------^
+ | y | | L | | iSCSI
+ | e | Enable_Datamover | a | | Full
+ | r |------------------------------->| y | | Feature
+ | | Datamover is enabled | e | | Phase
+ | | | r | |
+ | | Full Feature Phase | | |
+ | | control and data Transfer | | v
+
+ Figure 2. A Successful iSCSI Login on Initiator
+
+
+ | | Notice_Key_Values | | |
+ | |------------------------------->| | |
+ | | Datamover layer is notified | | |
+ | | of the negotiated key values | | |
+ | | | | |
+ | | Allocate_Connection_Resources | | |
+ | |------------------------------->| D | |
+ | | Connection resources are | a | |
+ | i | successfully allocated | t | | iSCSI
+ | S | | a | | Login
+ | C | | m |Final | Phase
+ | S | | o |Login |
+ | I |Enable_Datamover(Login Response)| v |Resp |
+ | |------------------------------->| e |---->vLogin Phase
+ | L | Datamover is enabled | r | ^ succeeds
+ | a | | | |
+ | y | | L | | iSCSI
+ | e | | a | | Full
+ | r | | y | | Feature
+ | | | e | | Phase
+ | | Full Feature Phase | r | |
+ | | control and data Transfer | | |
+ | | | | v
+
+ Figure 3. A Successful iSCSI Login on Target
+
+
+
+
+Chadalapaka, et al. Informational [Page 37]
+
+RFC 5047 DA October 2007
+
+
+ | | Allocate_Connection_Resources | D | ^
+ | |------------------------------->| a | |
+ | | Connection resources are | t | |
+ | i | successfully allocated | a | | iSCSI
+ | S | | m | | Login
+ | C | | o | | Phase
+ | S | | v | |
+ | I | | e | |
+ | | | r | | Login
+ | | | | | Phase
+ | L | Final Login Response (failure) v fails
+ | a |<------------------------------------------
+ | y | | L |
+ | e | Deallocate_Connection_Resources| a |
+ | r |------------------------------->| y |
+ | | Datamover-specific | e |
+ | | connection resources freed | r |
+ | | | |
+ | |
+ | | Connection terminated by standard means
+ | |--------------------------------------------->
+
+ Figure 4. A Failed iSCSI Login on Initiator
+
+
+ | | Allocate_Connection_Resources | D | ^
+ | |------------------------------->| a | |
+ | | Connection resources are | t | |
+ | i | successfully allocated | a | | iSCSI
+ | S | | m | | Login
+ | C | | o | | Phase
+ | S | | v | |
+ | I | | e | |
+ | | | r | | Login
+ | | | | | Phase
+ | L | Final Login Response (failure) v fails
+ | a |---------------------------------------------->
+ | y | | L |
+ | e | Deallocate_Connection_Resources| a |
+ | r |------------------------------->| y |
+ | | Datamover-specific | e |
+ | | connection resources freed | r |
+ | | | |
+ | |
+ | | Connection terminated by standard means
+ | |-------------------------------------------->
+
+ Figure 5. A Failed iSCSI Login on Target
+
+
+
+Chadalapaka, et al. Informational [Page 38]
+
+RFC 5047 DA October 2007
+
+
+
+ | | Allocate_Connection_Resources | D | ^
+ | |------------------------------->| a | |
+ | | Connection resources are | t | |
+ | i | successfully allocated | a | | iSCSI
+ | S | | m | | Login
+ | C | | o | | Phase
+ | S | | v | |
+ | I | | e | |
+ | | | r | |
+ | L | Login non-Final Request/Response |
+ | a |<-----------------------------------------|
+ | y | iSCSI layer decides not to | L | |
+ | e | enable Datamover for this | a | |
+ | r | connection | y | |
+ | | | e | |
+ | | Deallocate_Connection_Resources| r | |
+ | |------------------------------->| | |
+ | | All Datamover-specific | | |
+ | | resources deallocated | | |
+ | | | | | Login
+ | | | | | Phase
+ | | | continues
+ | | Regular Login negotiation continues |
+ | |<---------------------------------------->|
+ | | .
+ | | .
+ | | .
+
+ Figure 6. iSCSI Does Not Enable the Datamover
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 39]
+
+RFC 5047 DA October 2007
+
+
+ | | | | ^
+ | | Full Feature Phase Control & | | |
+ | | Data Transfer Using DM | D | | iSCSI
+ | | | a | | Full Feature
+ | i | | t | | Phase
+ | S | | a | | (DM Enabled)
+ | C | | m | |
+ | S | Successful iSCSI Logout | o | |
+ | I | | v | v
+ | | Connection_Terminate | e |
+ | L |------------------------------->| r |
+ | a | Connection is terminated | |
+ | y | Datamover-specific resources | L | Transport
+ | e | deallocated, both connection | a | Connection
+ | r | level & task level | y | is terminated
+ | | | e |
+ | | | r |
+ | | | |
+ | | | |
+
+ Figure 7. A Normal iSCSI Connection Termination
+
+
+ | | | | ^
+ | | Full Feature Phase Control & | D | | iSCSI
+ | | Data Transfer Using DM | a | | Full Feature
+ | i | | t | | Phase
+ | S | | a | | (DM Enabled)
+ | C | | m | v
+ | S | | o |<--Transport
+ | I | Datamover-specific resources | v | Connection
+ | | deallocated, both connection | e | Terminated (e.g.
+ | L | level & task level | r | unexpected
+ | a | | | FIN/RESET)
+ | y | | L |
+ | e | Connection_Terminate_Notify | a |
+ | r |<-------------------------------| y |
+ | | | e |
+ | | | r |
+ | | | |
+
+ Figure 8. An Abnormal iSCSI Connection Termination
+
+
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 40]
+
+RFC 5047 DA October 2007
+
+
+ <-----Initiator-----> <-------Target------->
+
+ | | | | DM Msg holding | | | |
+ SCSI | | | | SCSI Cmd PDU & | | | |SCSI
+ Cmd | | Snd_Ctrl | |Unsol Imm Data | |Ctrl_Notify | |Cmd
+ ---->| |--------->| |--------------->| |----------->| |--->
+ | | | | | | | |
+ | | | | DM Msg holding | | | |
+ | | Snd_Ctrl | |SCSI Dataout PDU| |Ctrl_Notify | |
+ | |--------->| |--------------->| |----------->| |
+ | | . | | . | | . | |Unsol
+ | | . | D| . | D| . | |Data
+ | | . | a| DM Msg holding | a| . | |Trans
+ | i| Snd_Ctrl | t|SCSI Dataout PDU| t|Ctrl_Notify | i|
+ | S|--------->| a|--------------->| a|----------->| S|
+ | C| | m| | m| | C|Buf
+ | S| | o| | o| | S|Avail
+ | I| | v| | v| Get_Data | I|(R2T)
+ | | | e|----------------| e|<-----------| |<----
+ | L| | r||Solicited Data | r| | L| .
+ | a| | || Transfer | | | a| .
+ | y| | L|--------------->| L| . | y|Buf
+ | e| | a| . | a| . | e|Avail
+ | r| | y| . | y| Get_Data | r|(R2T)
+ | | | e|----------------| e|<-----------| |<----
+ | | | r||Solicited Data | r| | |
+ | | | || Transfer | | | |
+ | | | |--------------->| |Data_Cmp_Nfy| |Data
+ | | | | | |----------->| |Trans
+ | | | | | | | |Compl
+ | | | | DM Msg holding | | | |
+ SCSI | | | |SCSI Resp PDU & | | | |SCSI
+ Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp
+ <----| |<---------| |<---------------| |<-----------| |<----
+ | | | | | | | |
+
+ Figure 9. A SCSI Write Data Transfer
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 41]
+
+RFC 5047 DA October 2007
+
+
+ <-----Initiator-----> <-------Target------->
+
+ | | | | | | | |
+ SCSI | | | | DM Msg holding | | | |SCSI
+ Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd
+ ---->| |--------->| |--------------->| |----------->| |--->
+ | | | | | | | |
+ | | | D| SCSI Read | D| | |Buf
+ | | | a| Data Transfer | a| Put_Data | |Avail
+ | i| | t|<---------------| t|<-----------| i|<----
+ | S| | a| . | a| . | S| .
+ | C| | m| . | m| . | C| .
+ | S| | o| . | o| . | S| .
+ | I| | v| SCSI Read | v| . | I|Buf
+ | | | e| Data Transfer | e| Put_Data | |Avail
+ | L| | r|<---------------| r|<-----------| L|<----
+ | a| | | | | | a|
+ | y| | L| | L| | y|
+ | e| | a| | a|Data_Cmp_Nfy| e|Data
+ | r| | y| | y|----------->| r|Trans
+ | | | e| | e| | |Compl
+ | | | r| DM Msg holding | r| | |
+ SCSI | | | |SCSI Resp PDU & | | | |SCSI
+ Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp
+ <----| |<---------| |<---------------| |<-----------| |<----
+ | | | | | | | |
+
+ Figure 10. A SCSI Read Data Transfer
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 42]
+
+RFC 5047 DA October 2007
+
+
+ <-----Initiator-----> <-------Target------->
+
+ | | | | | | | |
+ SCSI | | | | DM Msg holding | | | |SCSI
+ Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd
+ ---->| |--------->| |--------------->| |----------->| |---->
+ | | | | | | | |
+ | | | D| SCSI Read | D| Put_Data | |Buf
+ | | | a| Data Transfer | a|Data_in.A=1 | |Avail
+ | i| | t|<---------------| t|<-----------| i|<----
+ | S| | a| . | a| . | S| .
+ | C| | m| . | m|Data_ACK_Nfy| C| .
+ | S| | o| | o|----------->| S| .
+ | I| | v| | v| . | I|
+ | | | e| | e| . | |
+ | L| | r| | r| | L|
+ | a| | | | | | a|
+ | y| | L| | L| | y|
+ | e| | a| | a| | e|Data
+ | r| | y| | y| | r|Trans
+ | | | e| | e| | |Compl
+ | | | r| DM Msg holding | r| | |
+ SCSI | | | |SCSI Resp PDU & | | | |SCSI
+ Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp
+ <----| |<---------| |<---------------| |<-----------| |<----
+ | | | | | | | |
+
+ Figure 11. A SCSI Read Data Acknowledgement
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 43]
+
+RFC 5047 DA October 2007
+
+
+ <-----Initiator-----> <-------Target------->
+
+ | | | | | | | |
+ SCSI | | | | DM Msg holding | | | |SCSI
+ Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd
+ ---->| |--------->| |--------------->| |----------->| |---->
+ | | | | | | | |
+ | | | D| SCSI Read | D| | |Buf
+ | | | a| Data Transfer | a| Put_Data | |Avail
+ | i| | t|<---------------| t|<-----------| i|<----
+ | S| | a| . | a| . | S| .
+ Abort| C| | m| DM Msg holding | m| . | C|Abort
+ Task | S| Snd_Ctrl | o| Abort TMF Req | o|Ctrl_Notify | S|Task
+ ---->| I|--------->| v|--------------->| v|----------->| I|---->
+ | | | e| . | e| . | |
+ Abort| L| | r| DM Msg holding| r| | L| .
+ Done | a|Ctrl_Ntfy | | Abort TMF Res| | Snd_Ctrl | |Abted
+ <----| y|<---------| L|<---------------| L|<-----------| y|<----
+ | e| | a| | a| | e|
+ | r| | y| | y| | r|
+ | | | e| | e| | |
+ | | | r| | r| | |
+ | | | | | | | |
+ | |Dal_Tk_Res| | | |Dal_Tk_Res | |
+ | |--------->| | | |<-----------| |
+ | | | | | | | |
+
+ Figure 12. Task Resource Cleanup on Abort
+
+Acknowledgements
+
+ The IP Storage (IPS) Working Group in the Transport Area of
+ IETF has been responsible for defining the iSCSI protocol
+ (apart from a host of other relevant IP Storage protocols).
+ The authors are grateful to the entire working group, whose
+ work allowed this document to build on the concepts and
+ details of the iSCSI protocol.
+
+ In addition, the following individuals reviewed and
+ contributed to the improvement of this document. The authors
+ are grateful for their contribution.
+
+ John Carrier
+ Adaptec, Inc.
+ 691 S. Milpitas Blvd., Milpitas, CA 95035, USA
+ Phone: +1 (360) 378-8526
+ EMail: john_carrier@adaptec.com
+
+
+
+
+Chadalapaka, et al. Informational [Page 44]
+
+RFC 5047 DA October 2007
+
+
+ Hari Ghadia
+ Adaptec, Inc.
+ 691 S. Milpitas Blvd., Milpitas, CA 95035, USA
+ Phone: +1 (408) 957-5608
+ EMail: hari_ghadia@adaptec.com
+
+ Hari Mudaliar
+ Adaptec, Inc.
+ 691 S. Milpitas Blvd., Milpitas, CA 95035, USA
+ Phone: +1 (408) 957-6012
+ EMail: hari_mudaliar@adaptec.com
+
+ Patricia Thaler
+ Agilent Technologies, Inc.
+ 1101 Creekside Ridge Drive, #100, M/S-RG10,
+ Roseville, CA 95678, USA
+ Phone: +1-916-788-5662
+ EMail: pat_thaler@agilent.com
+
+ Uri Elzur
+ Broadcom Corporation
+ 16215 Alton Parkway, Irvine, CA 92619-7013, USA
+ Phone: +1 (949) 585-6432
+ EMail: Uri@Broadcom.com
+
+ Mike Penna
+ Broadcom Corporation
+ 16215 Alton Parkway,Irvine, CA 92619-7013, USA
+ Phone: +1 (949) 926-7149
+ EMail: MPenna@Broadcom.com
+
+ David Black
+ EMC Corporation
+ 176 South St., Hopkinton, MA 01748, USA
+ Phone: +1 (508) 293-7953
+ EMail: black_david@emc.com
+
+ Ted Compton
+ EMC Corporation
+ Research Triangle Park, NC 27709, USA
+ Phone: +1-919-248-6075
+ EMail: compton_ted@emc.com
+
+ Dwight Barron
+ Hewlett-Packard Company
+ 20555 SH 249, Houston, TX 77070-2698, USA
+ Phone: +1 (281) 514-2769
+ EMail: Dwight.Barron@Hp.com
+
+
+
+Chadalapaka, et al. Informational [Page 45]
+
+RFC 5047 DA October 2007
+
+
+ Paul R. Culley
+ Hewlett-Packard Company
+ 20555 SH 249, Houston, TX 77070-2698, USA
+ Phone: +1 (281) 514-5543
+ EMail: paul.culley@hp.com
+
+ Dave Garcia
+ Hewlett-Packard Company
+ 19333 Vallco Parkway, Cupertino, CA 95014, USA
+ Phone: +1 (408) 285-6116
+ EMail: dave.garcia@hp.com
+
+ Randy Haagens
+ Hewlett-Packard Company
+ 8000 Foothills Blvd, MS 5668, Roseville CA, USA
+ Phone: +1-916-785-4578
+ EMail: randy_haagens@hp.com
+
+ Jeff Hilland
+ Hewlett-Packard Company
+ 20555 SH 249, Houston, TX 77070-2698, USA
+ Phone: +1 (281) 514-9489
+ EMail: jeff.hilland@hp.com
+
+ Mike Krause
+ Hewlett-Packard Company, 43LN
+ 19410 Homestead Road, Cupertino, CA 95014, USA
+ Phone: +1 (408) 447-3191
+ EMail: krause@cup.hp.com
+
+ Jim Wendt
+ Hewlett-Packard Company
+ 8000 Foothills Blvd, MS 5668, Roseville CA, USA
+ Phone: +1-916-785-5198
+ EMail: jim_wendt@hp.com
+
+ Mike Ko
+ IBM
+ 650 Harry Rd, San Jose, CA 95120, USA
+ Phone: +1 (408) 927-2085
+ EMail: mako@us.ibm.com
+
+ Renato Recio
+ IBM Corporation
+ 11501 Burnett Road, Austin, TX 78758, USA
+ Phone: +1 (512) 838-1365
+ EMail: recio@us.ibm.com
+
+
+
+
+Chadalapaka, et al. Informational [Page 46]
+
+RFC 5047 DA October 2007
+
+
+ Howard C. Herbert
+ Intel Corporation
+ MS CH7-404,5000 West Chandler Blvd., Chandler, AZ 85226, USA
+ Phone: +1 (480) 554-3116
+ EMail: howard.c.herbert@intel.com
+
+ Dave Minturn
+ Intel Corporation
+ MS JF1-210, 5200 North East Elam Young Parkway
+ Hillsboro, OR 97124, USA
+ Phone: +1 (503) 712-4106
+ EMail: dave.b.minturn@intel.com
+
+ James Pinkerton
+ Microsoft Corporation
+ One Microsoft Way, Redmond, WA 98052, USA
+ Phone: +1 (425) 705-5442
+ EMail: jpink@microsoft.com
+
+ Tom Talpey
+ Network Appliance
+ 375 Totten Pond Road, Waltham, MA 02451, USA
+ Phone: +1 (781) 768-5329
+ EMail: thomas.talpey@netapp.com
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 47]
+
+RFC 5047 DA October 2007
+
+
+Authors' Addresses
+
+ Mallikarjun Chadalapaka
+ Hewlett-Packard Company
+ 8000 Foothills Blvd.
+ Roseville, CA 95747-5668, USA
+
+ Phone: +1-916-785-5621
+ EMail: cbm@rose.hp.com
+
+
+ John L. Hufferd
+ Brocade, Inc.
+ 1745 Technology Drive
+ San Jose, CA 95110, USA
+
+ Phone: +1-408-333-5244
+ EMail: jhufferd@brocade.com
+
+
+ Julian Satran
+ IBM, Haifa Research Lab
+ Haifa University Campus - Mount Carmel
+ Haifa 31905, Israel
+
+ Phone +972-4-829-6264
+ EMail: Julian_Satran@il.ibm.com
+
+
+ Hemal Shah
+ Broadcom Corporation
+ 5300 California Avenue
+ Irvine, California 92617, USA
+
+ Phone: +1-949-926-6941
+ EMail: hemal@broadcom.com
+
+ Comments may be sent to Mallikarjun Chadalapaka.
+
+
+
+
+
+
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 48]
+
+RFC 5047 DA October 2007
+
+
+Full Copyright Statement
+
+ Copyright (C) The IETF Trust (2007).
+
+ This document is subject to the rights, licenses and restrictions
+ contained in BCP 78, and except as set forth therein, the authors
+ retain all their rights.
+
+ This document and the information contained herein are provided on an
+ "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
+ OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
+ THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
+ OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
+ THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
+ WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
+
+Intellectual Property
+
+ The IETF takes no position regarding the validity or scope of any
+ Intellectual Property Rights or other rights that might be claimed to
+ pertain to the implementation or use of the technology described in
+ this document or the extent to which any license under such rights
+ might or might not be available; nor does it represent that it has
+ made any independent effort to identify any such rights. Information
+ on the procedures with respect to rights in RFC documents can be
+ found in BCP 78 and BCP 79.
+
+ Copies of IPR disclosures made to the IETF Secretariat and any
+ assurances of licenses to be made available, or the result of an
+ attempt made to obtain a general license or permission for the use of
+ such proprietary rights by implementers or users of this
+ specification can be obtained from the IETF on-line IPR repository at
+ http://www.ietf.org/ipr.
+
+ The IETF invites any interested party to bring to its attention any
+ copyrights, patents or patent applications, or other proprietary
+ rights that may cover technology that may be required to implement
+ this standard. Please address the information to the IETF at
+ ietf-ipr@ietf.org.
+
+
+
+
+
+
+
+
+
+
+
+
+Chadalapaka, et al. Informational [Page 49]
+