doc: Add RFC documents

1 files changed, 1237 insertions, 0 deletions

diff --git a/doc/rfc/rfc998.txt b/doc/rfc/rfc998.txt
new file mode 100644
index 0000000..a572d6c
--- /dev/null
+++ b/doc/rfc/rfc998.txt
@@ -0,0 +1,1237 @@
+
+Network Working Group                                     David D. Clark
+Request for Comments:  998                               Mark L. Lambert
+Obsoletes:  RFC 969                                          Lixia Zhang
+                                                                     MIT
+                                                              March 1987
+
+
+                 NETBLT: A Bulk Data Transfer Protocol
+
+
+1. Status
+
+   This document is a description of, and a specification for, the
+   NETBLT protocol.  It is a revision of the specification published in
+   NIC RFC-969.  The protocol has been revised after extensive research
+   into NETBLT's performance over long-delay, high-bandwidth satellite
+   channels.  Most of the changes in the protocol specification have to
+   do with the computation and use of data timers in a multiple
+   buffering data transfer model.
+
+   This document is published for discussion and comment, and does not
+   constitute a standard.  The proposal may change and certain parts of
+   the protocol have not yet been specified; implementation of this
+   document is therefore not advised.
+
+2. Introduction
+
+   NETBLT (NETwork BLock Transfer) is a transport level protocol
+   intended for the rapid transfer of a large quantity of data between
+   computers.  It provides a transfer that is reliable and flow
+   controlled, and is designed to provide maximum throughput over a wide
+   variety of networks.  Although NETBLT currently runs on top of the
+   Internet Protocol (IP), it should be able to operate on top of any
+   datagram protocol similar in function to IP.
+
+   NETBLT's motivation is to achieve higher throughput than other
+   protocols might offer.  The protocol achieves this goal by trying to
+   minimize the effect of several network-related problems: network
+   congestion, delays over satellite links, and packet loss.
+
+   Its transmission rate-control algorithms deal well with network
+   congestion; its multiple-buffering capability allows high throughput
+   over long-delay satellite channels, and its various
+   timeout/retransmit algorithms minimize the effect of packet loss
+   during a transfer.  Most importantly, NETBLT's features give it good
+   performance over long-delay channels without impairing performance
+   over high-speed LANs.
+
+
+
+
+
+
+
+Clark, Lambert, & Zhang                                         [Page 1]
+
+RFC 998                                                       March 1987
+
+
+   The protocol works by opening a connection between two "clients" (the
+   "sender" and the "receiver"), transferring the data in a series of
+   large data aggregates called "buffers", and then closing the
+   connection.  Because the amount of data to be transferred can be very
+   large, the client is not required to provide at once all the data to
+   the protocol module.  Instead, the data is provided by the client in
+   buffers.  The NETBLT layer transfers each buffer as a sequence of
+   packets; since each buffer is composed of a large number of packets,
+   the per-buffer interaction between NETBLT and its client is far more
+   efficient than a per-packet interaction would be.
+
+   In its simplest form, a NETBLT transfer works as follows:  the
+   sending client loads a buffer of data and calls down to the NETBLT
+   layer to transfer it.  The NETBLT layer breaks the buffer up into
+   packets and sends these packets across the network in Internet
+   datagrams.  The receiving NETBLT layer loads these packets into a
+   matching buffer provided by the receiving client.  When the last
+   packet in the buffer has arrived, the receiving NETBLT checks to see
+   that all packets in that buffer have been correctly received.  If
+   some packets are missing, the receiving NETBLT requests that they be
+   resent.  When the buffer has been completely transmitted, the
+   receiving client is notified by its NETBLT layer.  The receiving
+   client disposes of the buffer and provides a new buffer to receive
+   more data.  The receiving NETBLT notifies the sender that the new
+   buffer is ready, and the sender prepares and sends the next buffer in
+   the same manner.  This continues until all the data has been sent; at
+   that time the sender notifies the receiver that the transmission has
+   been completed.  The connection is then closed.
+
+   As described above, the NETBLT protocol is "lock-step".  Action halts
+   after a buffer is transmitted, and begins again after confirmation is
+   received from the receiver of data.  NETBLT provides for multiple
+   buffering, a transfer model in which the sending NETBLT can transmit
+   new buffers while earlier buffers are waiting for confirmation from
+   the receiving NETBLT.  Multiple buffering makes packet flow
+   essentially continuous and markedly improves performance.
+
+   The remainder of this document describes NETBLT in detail.  The next
+   sections describe the philosophy behind a number of protocol
+   features:  packetization, flow control, transfer reliability, and
+   connection management. The final sections describe NETBLT's packet
+   formats.
+
+3. Buffers and Packets
+
+   NETBLT is designed to permit transfer of a very large amounts of data
+   between two clients.  During connection setup the sending NETBLT can
+   inform the receiving NETBLT of the transfer size; the maximum
+   transfer length is 2**32 bytes.  This limit should permit any
+   practical application.  The transfer size parameter is for the use of
+   the receiving client; the receiving NETBLT makes no use of it.  A
+
+
+
+Clark, Lambert, & Zhang                                         [Page 2]
+
+RFC 998                                                       March 1987
+
+
+   NETBLT receiver accepts data until told by the sender that the
+   transfer is complete.
+
+   The data to be sent must be broken up into buffers by the client.
+   Each buffer must be the same size, save for the last buffer.  During
+   connection setup, the sending and receiving NETBLTs negotiate the
+   buffer size, based on limits provided by the clients.  Buffer sizes
+   are in bytes only; the client is responsible for placing data in
+   buffers on byte boundaries.
+
+   NETBLT has been designed and should be implemented to work with
+   buffers of any size.  The only fundamental limitation on buffer size
+   should be the amount of memory available to the client.  Buffers
+   should be as large as possible since this minimizes the number of
+   buffer transmissions and therefore improves performance.
+
+   NETBLT is designed to require a minimum amount of memory, allowing
+   the client to allocate as much memory as possible for buffer storage.
+   In particular, NETBLT does not keep buffer copies for retransmission
+   purposes.  Instead, data to be retransmitted is recopied directly
+   from the client buffer.  This means that the client cannot release
+   buffer storage piece by piece as the buffer is sent, but this has not
+   been a problem in preliminary NETBLT implementations.
+
+   Buffers are broken down by the NETBLT layer into sequences of DATA
+   packets.  As with the buffer size, the DATA packet size is negotiated
+   between the sending and receiving NETBLTs during connection setup.
+   Unlike buffer size, DATA packet size is visible only to the NETBLT
+   layer.
+
+   All DATA packets save the last packet in a buffer must be the same
+   size.  Packets should be as large as possible, since NETBLT's
+   performance is directly related to packet size.  At the same time,
+   the packets should not be so large as to cause internetwork
+   fragmentation, since this normally causes performance degradation.
+
+   All buffers save the last buffer must be the same size; the last
+   buffer can be any size required to complete the transfer.  Since the
+   receiving NETBLT does not know the transfer size in advance, it needs
+   some way of identifying the last packet in each buffer.  For this
+   reason, the last packet of every buffer is not a DATA packet but
+   rather an LDATA packet.  DATA and LDATA packets are identical save
+   for the packet type.
+
+4. Flow Control
+
+   NETBLT uses two strategies for flow control, one internal and one at
+   the client level.
+
+   The sending and receiving NETBLTs transmit data in buffers; client
+   flow control is therefore at a buffer level.  Before a buffer can be
+
+
+
+Clark, Lambert, & Zhang                                         [Page 3]
+
+RFC 998                                                       March 1987
+
+
+   transmitted, NETBLT confirms that both clients have set up matching
+   buffers, that one is ready to send data, and that the other is ready
+   to receive data.  Either client can therefore control the flow of
+   data by not providing a new buffer.  Clients cannot stop a buffer
+   transfer once it is in progress.
+
+   Since buffers can be quite large, there has to be another method for
+   flow control that is used during a buffer transfer.  The NETBLT layer
+   provides this form of flow control.
+
+   There are several flow control problems that could arise while a
+   buffer is being transmitted.  If the sending NETBLT is transferring
+   data faster than the receiving NETBLT can process it, the receiver's
+   ability to buffer unprocessed packets could be overflowed, causing
+   packet loss.  Similarly, a slow gateway or intermediate network could
+   cause packets to collect and overflow network packet buffer space.
+   Packets will then be lost within the network.  This problem is
+   particularly acute for NETBLT because NETBLT buffers will generally
+   be quite large, and therefore composed of many packets.
+
+   A traditional solution to packet flow control is a window system, in
+   which the sending end is permitted to send only a certain number of
+   packets at a time.  Unfortunately, flow control using windows tends
+   to result in low throughput.  Windows must be kept small in order to
+   avoid overflowing hosts and gateways, and cannot easily be updated,
+   since an end-to-end exchange is required for each window change.
+
+   To permit high throughput over a variety of networks and gateways,
+   NETBLT uses a novel flow control method: rate control.  The
+   transmission rate is negotiated by the sending and receiving NETBLTs
+   during connection setup and after each buffer transmission.  The
+   sender uses timers, rather than messages from the receiver, to
+   maintain the negotiated rate.
+
+   In its simplest form, rate control specifies a minimum time period
+   per packet transmission.  This can cause performance problems for
+   several reasons.  First, the transmission time for a single packet is
+   very small, frequently smaller than the granularity of the timing
+   mechanism.  Also, the overhead required to maintain timing mechanisms
+   on a per packet basis is relatively high and lowers performance.
+
+   The solution is to control the transmission rate of groups of
+   packets, rather than single packets.  The sender transmits a burst of
+   packets over a negotiated time interval, then sends another burst.
+   In this way, the overhead decreases by a factor of the burst size,
+   and the per-burst transmission time is long enough that timing
+   mechanisms will work properly.  NETBLT's rate control therefore has
+   two parts, a burst size and a burst rate, with (burst size)/(burst
+   rate) equal to the average transmission time per packet.
+
+
+
+
+
+Clark, Lambert, & Zhang                                         [Page 4]
+
+RFC 998                                                       March 1987
+
+
+   The burst size and burst rate should be based not only on the packet
+   transmission and processing speed which each end can handle, but also
+   on the capacities of any intermediate gateways or networks.
+   Following are some intuitive values for packet size, buffer size,
+   burst size, and burst rate.
+
+   Packet sizes can be as small as 128 bytes.  Performance with packets
+   this small is almost always bad, because of the high per-packet
+   processing overhead.  Even the default Internet Protocol packet size
+   of 576 bytes is barely big enough for adequate performance.  Most
+   networks do not support packet sizes much larger than one or two
+   thousand bytes, and packets of this size can also get fragmented when
+   traveling over intermediate networks, lowering performance.
+
+   The size of a NETBLT buffer is limited only by the amount of memory
+   available to a client.  Theoretically, buffers of 100 Kbytes or more
+   are possible.  This would mean the transmission of 50 to 100 packets
+   per buffer.
+
+   The burst size and burst rate are obviously very machine dependent.
+   There is a certain amount of transmission overhead in the sending and
+   receiving machines associated with maintaining timers and scheduling
+   processes.  This overhead can be minimized by sending packets in
+   large bursts.  There are also limitations imposed on the burst size
+   by the number of available packet buffers in the operating system
+   kernel. On most modern operating systems, a burst size of between
+   five and ten packets should reduce the overhead to an acceptable
+   level.  A preliminary NETBLT implementation for the IBM PC/AT sends
+   packets in bursts of five.  It could send more, but is limited by the
+   available memory.
+
+   The burst rate is in part determined by the granularity of the
+   sender's timing mechanism, and in part by the processing speed of the
+   receiver and any intermediate gateways.  It is also directly related
+   to the burst size.  Burst rates from 20 to 45 milliseconds per 5-
+   packet burst have been tried on the IBM PC/AT and Symbolics 3600
+   NETBLT implementations with good results within a single local-area
+   network.  This value clearly depends on the network bandwidth and
+   packet buffering available.
+
+   All NETBLT flow control parameters (packet size, buffer size, burst
+   size, and burst rate) are negotiated during connection setup.  The
+   negotiation process is the same for all parameters.  The client
+   initiating the connection (the active end) proposes and sends a set
+   of values for each parameter in its connection request.  The other
+   client (the passive end) compares these values with the highest-
+   performance values it can support.  The passive end can then modify
+   any of the parameters, but only by making them more restrictive.  The
+   modified parameters are then sent back to the active end in its
+   response message.
+
+
+
+
+Clark, Lambert, & Zhang                                         [Page 5]
+
+RFC 998                                                       March 1987
+
+
+   The burst size and burst rate can also be re-negotiated after each
+   buffer transmission to adjust the transfer rate according to the
+   performance observed from transferring the previous buffer.  The
+   receiving end sends burst size and burst rate values in its OK
+   messages (described later).  The sender compares these values with
+   the values it can support.  Again, it may then modify any of the
+   parameters, but only by making them more restrictive.  The modified
+   parameters are then communicated to the receiver in a NULL-ACK
+   packet, described later.
+
+   Obviously each of the parameters depend on many factors -- gateway
+   and host processing speeds, available memory, timer granularity --
+   some of which cannot be checked by either client.  Each client must
+   therefore try to make as best a guess as it can, tuning for
+   performance on subsequent transfers.
+
+5. The NETBLT Transfer Model
+
+   Each NETBLT transfer has three stages, connection setup, data
+   transfer, and connection close.  The stages are described in detail
+   below, along with methods for insuring that each stage completes
+   reliably.
+
+5.1. Connection Setup
+
+   A NETBLT connection is set up by an exchange of two packets between
+   the active NETBLT and the passive NETBLT.  Note that either NETBLT
+   can send or receive data; the words "active" and "passive" are only
+   used to differentiate the end making the connection request from the
+   end responding to the connection request.  The active end sends an
+   OPEN packet; the passive end acknowledges the OPEN packet in one of
+   two ways.  It can either send a REFUSED packet, indicating that the
+   connection cannot be completed for some reason, or it can complete
+   the connection setup by sending a RESPONSE packet.  At this point the
+   transfer can begin.
+
+   As discussed in the previous section, the OPEN and RESPONSE packets
+   are used to negotiate flow control parameters.  Other parameters used
+   in the data transfer are also negotiated.  These parameters are (1)
+   the maximum number of buffers that can be sending at any one time,
+   and (2) whether or not DATA packet data will be checksummed.  NETBLT
+   automatically checksums all non-DATA/LDATA packets.  If the
+   negotiated checksum flag is set to TRUE (1), both the header and the
+   data of a DATA/LDATA packet are checksummed; if set to FALSE (0),
+   only the header is checksummed.  The checksum value is the bitwise
+   negation of the ones-complement sum of the 16-bit words being
+   checksummed.
+
+   Finally, each end transmits its death-timeout value in seconds in
+   either the OPEN or the RESPONSE packet.  The death-timeout value will
+   be used to determine the frequency with which to send KEEPALIVE
+
+
+
+Clark, Lambert, & Zhang                                         [Page 6]
+
+RFC 998                                                       March 1987
+
+
+   packets during idle periods of an opened connection (death timers and
+   KEEPALIVE packets are described in the following section).
+
+   The active end specifies a passive client through a client-specific
+   "well-known" 16 bit port number on which the passive end listens.
+   The active end identifies itself through a 32 bit Internet address
+   and a unique 16 bit port number.
+
+   In order to allow the active and passive ends to communicate
+   miscellaneous useful information, an unstructured, variable-length
+   field is provided in OPEN and RESPONSE packets for any client-
+   specific information that may be required.  In addition, a "reason
+   for refusal" field is provided in REFUSED packets.
+
+   Recovery for lost OPEN and RESPONSE packets is provided by the use of
+   timers.  The active end sets a timer when it sends an OPEN packet.
+   When the timer expires, another OPEN packet is sent, until some
+   predetermined maximum number of OPEN packets have been sent.  The
+   timer is cleared upon receipt of a RESPONSE packet.
+
+   To prevent duplication of OPEN and RESPONSE packets, the OPEN packet
+   contains a 32 bit connection unique ID that must be returned in the
+   RESPONSE packet.  This prevents the initiator from confusing the
+   response to the current request with the response to an earlier
+   connection request (there can only be one connection between any two
+   ports).  Any OPEN or RESPONSE packet with a destination port matching
+   that of an open connection has its unique ID checked.  If the unique
+   ID of the packet matches the unique ID of the connection, then the
+   packet type is checked.  If it is a RESPONSE packet, it is treated as
+   a duplicate and ignored.  If it is an OPEN packet, the passive NETBLT
+   sends another RESPONSE (assuming that a previous RESPONSE packet was
+   sent and lost, causing the initiating NETBLT to retransmit its OPEN
+   packet).  A non-matching unique ID must be treated as an attempt to
+   open a second connection between the same port pair and is rejected
+   by sending an ABORT message.
+
+5.2. Data Transfer
+
+   The simplest model of data transfer proceeds as follows.  The sending
+   client sets up a buffer full of data.  The receiving NETBLT sends a
+   GO message inside a CONTROL packet to the sender, signifying that it
+   too has set up a buffer and is ready to receive data.  Once the GO
+   message is received, the sender transmits the buffer as a series of
+   DATA packets followed by an LDATA packet.  When the last packet in
+   the buffer has been received, the receiver sends a RESEND message
+   inside a CONTROL packet containing a list of packets that were not
+   received.  The sender resends these packets.  This process continues
+   until there are no missing packets.  At that time the receiver sends
+   an OK message inside a CONTROL packet, sets up another buffer to
+   receive data, and sends another GO message.  The sender, having
+   received the OK message, sets up another buffer, waits for the GO
+
+
+
+Clark, Lambert, & Zhang                                         [Page 7]
+
+RFC 998                                                       March 1987
+
+
+   message, and repeats the process.
+
+   The above data transfer model is effectively a lock-step protocol,
+   and causes time to be wasted while the sending NETBLT waits for
+   permission to send a new buffer.  A more efficient transfer model
+   uses multiple buffering to increase performance.  Multiple buffering
+   is a technique in which the sender and receiver allocate and transmit
+   buffers in a manner that allows error recovery or successful
+   transmission confirmation of previous buffers to be concurrent with
+   transmission of the current buffer.
+
+   During the connection setup phase, one of the negotiated parameters
+   is the number of concurrent buffers permitted during the transfer.
+   If there is more than one buffer available, transfer of the next
+   buffer may start right after the current buffer finishes.  This is
+   illustrated in the following example:
+
+   Assume two buffers A and B in a multiple-buffer transfer, with A
+   preceding B. When A has been transferred and the sending NETBLT is
+   waiting for either an OK or a RESEND message for it, the sending
+   NETBLT can start sending B immediately, keeping data flowing at a
+   stable rate.  If the receiver of data sends an OK for A, all is well;
+   if it receives a RESEND, the missing packets specified in the RESEND
+   message are retransmitted.
+
+   In the multiple-buffer transfer model, all packets to be sent are
+   re-ordered by buffer number (lowest number first), with the transfer
+   rate specified by the burst size and burst rate.  Since buffer
+   numbers increase monotonically, packets from an earlier buffer will
+   always precede packets from a later buffer.
+
+   Having several buffers transmitting concurrently is actually not that
+   much more complicated than transmitting a single buffer at a time.
+   The key is to visualize each buffer as a finite state machine;
+   several buffers are merely a group of finite state machines, each in
+   one of several states.  The transfer process consists of moving
+   buffers through various states until the entire transmission has
+   completed.
+
+   There are several obvious flaws in the data transfer model as
+   described above.  First, what if the GO, OK, or RESEND messages are
+   lost?  The sender cannot act on a packet it has not received, so the
+   protocol will hang.  Second, if an LDATA packet is lost, how does the
+   receiver know when the buffer has been transmitted?  Solutions for
+   each of these problems are presented below.
+
+5.2.1. Recovering from Lost Control Messages
+
+   NETBLT solves the problem of lost OK, GO, and RESEND messages in two
+   ways.  First, it makes use of a control timer.  The receiver can send
+   one or more control messages (OK, GO, or RESEND) within a single
+
+
+
+Clark, Lambert, & Zhang                                         [Page 8]
+
+RFC 998                                                       March 1987
+
+
+   CONTROL packet.  Whenever the receiver sends a control packet, it
+   sets a control timer.  This timer is either "reset" (set again) or
+   "cleared" (deactivated), under the following conditions:
+
+   When the control timer expires, the receiving NETBLT resends the
+   control packet and resets the timer.  The receiving NETBLT continues
+   to resend control packets in response to control timer's expiration
+   until either the control timer is cleared or the receiving NETBLT's
+   death timer (described later) expires (at which time it shuts down
+   the connection).
+
+   Each control message includes a sequence number which starts at one
+   and increases by one for each control message sent.  The sending
+   NETBLT checks the sequence number of every incoming control message
+   against all other sequence numbers it has received.  It stores the
+   highest sequence number below which all other received sequence
+   numbers are consecutive (in following paragraphs this is called the
+   high-acknowledged-sequence-number) and returns this number in every
+   packet flowing back to the receiver.  The receiver is permitted to
+   clear its control timer when it receives a packet from the sender
+   with a high-acknowledged-sequence-number greater than or equal to the
+   highest sequence number in the control packet just sent.
+
+   Ideally, a NETBLT implementation should be able to cope with out-of-
+   sequence control messages, perhaps collecting them for later
+   processing, or even processing them immediately.  If an incoming
+   control message "fills" a "hole" in a group of message sequence
+   numbers, the implementation could even be clever enough to detect
+   this and adjust its outgoing sequence value accordingly.
+
+   The sending NETBLT, upon receiving a CONTROL packet, should act on
+   the packet as quickly as possible.  It either sets up a new buffer
+   (upon receipt of an OK message for a previous buffer), marks data for
+   resending (upon receipt of a RESEND message), or prepares a buffer
+   for sending (upon receipt of a GO message).  If the sending NETBLT is
+   not in a position to send data, it should send a NULL-ACK packet,
+   which contains its high-acknowledged-sequence-number (this permits
+   the receiving NETBLT to acknowledge any outstanding control
+   messages), and wait until it can send more data.  In all of these
+   cases, the system overhead for a response to the incoming control
+   message should be small and relatively constant.
+
+   The small amount of message-processing overhead allows accurate
+   control timers to be set for all types of control messages with a
+   single, simple algorithm -- the network round-trip transit time, plus
+   a variance factor.  This is more efficient than schemes used by other
+   protocols, where timer value calculation has been a problem because
+   the processing time for a particular packet can vary greatly
+   depending on the packet type.
+
+   Control timer value estimation is extremely important in a high-
+
+
+
+Clark, Lambert, & Zhang                                         [Page 9]
+
+RFC 998                                                       March 1987
+
+
+   performance protocol like NETBLT.  A long control timer causes the
+   receiving NETBLT to wait for long periods of time before
+   retransmitting unacknowledged messages.  A short control timer value
+   causes the sending NETBLT to receive many duplicate control messages
+   (which it can reject, but which takes time).
+
+   In addition to the use of control timers, NETBLT reduces lost control
+   messages by using a single long-lived control packet; the packet is
+   treated like a FIFO queue, with new control messages added on at the
+   end and acknowledged control messages removed from the front.  The
+   implementation places control messages in the control packet and
+   transmits the entire control packet, consisting of any unacknowledged
+   control messages plus new messages just added.  The entire control
+   packet is also transmitted whenever the control timer expires.  Since
+   control packet transmissions are fairly frequent, unacknowledged
+   messages may be transmitted several times before they are finally
+   acknowledged.  This redundant transmission of control messages
+   provides automatic recovery for most control message losses over a
+   noisy channel.
+
+   This scheme places some burdens on the receiver of the control
+   messages.  It must be able to quickly reject duplicate control
+   messages, since a given message may be retransmitted several times
+   before its acknowledgement is received and it is removed from the
+   control packet.  Typically this is fairly easy to do; the sender of
+   data merely throws away any control messages with sequence numbers
+   lower than its high-acknowledged-sequence-number.
+
+   Another problem with this scheme is that the control packet may
+   become larger than the maximum allowable packet size if too many
+   control messages are placed into it.  This has not been a problem in
+   the current NETBLT implementations: a typical control packet size is
+   1000 bytes; RESEND control messages average about 20 bytes in length,
+   GO messages are 8 bytes long, and OK messages are 16 bytes long.
+   This allows 50-80 control messages to be placed in the control
+   packet, more than enough for reasonable transfers.  Other
+   implementations can provide for multiple control packets if a single
+   control packet may not be sufficient.
+
+   The control timer value must be carefully estimated.  It can have as
+   its initial value an arbitrary number.  Subsequent control packets
+   should have their timer values based on the network round-trip
+   transit time (i.e. the time between sending the control packet and
+   receiving the acknowledgment of all messages in the control packet)
+   plus a variance factor.  The timer value should be continually
+   updated, based on a smoothed average of collected round-trip transit
+   times.
+
+
+
+
+
+
+
+Clark, Lambert, & Zhang                                        [Page 10]
+
+RFC 998                                                       March 1987
+
+
+5.2.2. Recovering from Lost LDATA Packets
+
+   NETBLT solves the problem of LDATA packet loss by using a data timer
+   for each buffer at the receiving end.  The simplest data timer model
+   has a data timer set when a buffer is ready to be received; if the
+   data timer expires, the receiving NETBLT assumes a lost LDATA packet
+   and sends a RESEND message requesting all missing DATA packets in the
+   buffer.  When all packets have been received, the timer is cleared.
+
+   Data timer values are not based on network round-trip transit time;
+   instead they are based on the amount of time taken to transfer a
+   buffer (as determined by the number of DATA packet bursts in the
+   buffer times the burst rate) plus a variance factor <1>.
+
+   Obviously an accurate estimation of the data timer value is very
+   important.  A short data timer value causes the receiving NETBLT to
+   send unnecessary RESEND packets.  This causes serious performance
+   degradation since the sending NETBLT has to stop what it is doing and
+   resend a number of DATA packets.
+
+   Data timer setting and clearing turns out to be fairly complicated,
+   particularly in a multiple-buffering transfer model.  In
+   understanding how and when data timers are set and cleared, it is
+   helpful to visualize each buffer as a finite-state machine and take a
+   look at the various states.
+
+   The state sequence for a sending buffer is simple.  When a GO message
+   for the buffer is received, the buffer is created, filled with data,
+   and placed in a SENDING state.  When an OK for that buffer has been
+   received, it goes into a SENT state and is disposed of.
+
+   The state sequence for a receiving buffer is a little more
+   complicated.  Assume existence of a buffer A. When a control message
+   for A is sent, the buffer moves into state ACK-WAIT (it is waiting
+   for acknowledgement of the control message).
+
+   As soon as the control message has been acknowledged, buffer A moves
+   from the ACK-WAIT state into the ACKED state (it is now waiting for
+   DATA packets to arrive).  At this point, A's data timer is set and
+   the control message removed from the control packet.  Estimation of
+   the data timer value at this point is quite difficult.  In a
+   multiple-buffer transfer model, the receiving NETBLT can send several
+   GO messages at once.  A single DATA packet from the sending NETBLT
+   could acknowledge all the GO messages, causing several buffers to
+   start up data timers.  Clearly each of the data timers must be set in
+   a manner that takes into account each buffer's place in the order of
+   transmission.  Packets for a buffer A - 1 will always be transmitted
+   before packets in A, so A's data timer must take into account the
+   arrival of all of A - 1's DATA packets as well as arrival of its own
+   DATA packets.  This means that the timer values become increasingly
+   less accurate for higher-numbered buffers.  Because this data timer
+
+
+
+Clark, Lambert, & Zhang                                        [Page 11]
+
+RFC 998                                                       March 1987
+
+
+   value can be quite inaccurate, it is called a "loose" data timer.
+   The loose data timer value is recalculated later (using the same
+   algorithm, but with updated information), giving a "tight" timer, as
+   described below.
+
+   When the first DATA packet for A arrives, A moves from the ACKED
+   state to the RECEIVING state and its data timer is set to a new
+   "tight" value.  The tight timer value is calculated in the same
+   manner as the loose timer, but it is more accurate since we have
+   moved forward in time and those buffers numbered lower than A have
+   presumably been dealt with (or their packets would have arrived
+   before A's), leaving fewer packets to arrive between the setting of
+   the data timer and the arrival of the last DATA packet in A.
+
+   The receiving NETBLT also sets the tight data timers of any buffers
+   numbered lower than A that are also in the ACKED state.  This is done
+   as an optimization: we know that buffers are processed in order,
+   lowest number first.  If a buffer B numbered lower than A is in the
+   ACKED state, its DATA packets should arrive before A's.  Since A's
+   have arrived first, B's must have gotten lost.  Since B's loose data
+   timer has not expired (it would then have sent a RESEND message and
+   be in the ACK-WAIT state), we set the tight timer, allowing the
+   missing packets to be detected earlier.  An immediate RESEND is not
+   sent because it is possible that A's packet was re-ordered before B's
+   by the network, and that B's packets may arrive shortly.
+
+   When all DATA packets for A have been received, it moves from the
+   RECEIVING state to the RECEIVED state and is disposed of.  Had any
+   packets been missing, A's data timer would have expired and A would
+   have moved into the ACK-WAIT state after sending a RESEND message.
+   The state progression would then move as in the above example.
+
+   The control and data timer system can be summarized as follows:
+   normally, the receiving NETBLT is working under one of two types of
+   timers, a control timer or a data timer.  There is one data timer per
+   buffer transmission and one control timer per control packet.  The
+   data timer is active while its buffer is in either the ACKED (loose
+   data timer value is used) or the RECEIVING (tight data timer value is
+   used) states; a control timer is active whenever the receiving NETBLT
+   has any unacknowledged control messages in its control packet.
+
+5.2.3. Death Timers and Keepalive Packets
+
+   The above system still leaves a few problems.  If the sending NETBLT
+   is not ready to send, it sends a single NULL-ACK packet to clear any
+   outstanding control timers at the receiving end.  After this the
+   receiver will wait.  The sending NETBLT could die and the receiver,
+   with its control timer cleared, would hang.  Also, the above system
+   puts timers only on the receiving NETBLT.  The sending NETBLT has no
+   timers; if the receiving NETBLT dies, the sending NETBLT will hang
+   while waiting for control messages to arrive.
+
+
+
+Clark, Lambert, & Zhang                                        [Page 12]
+
+RFC 998                                                       March 1987
+
+
+   The solution to the above two problems is the use of a death timer
+   and a keepalive packet for both the sending and receiving NETBLTs.
+   As soon as the connection is opened, each end sets a death timer;
+   this timer is reset every time a packet is received.  When a NETBLT's
+   death timer expires, it can assume the other end has died and can
+   close the connection.
+
+   It is possible that the sending or receiving NETBLTs will have to
+   wait for long periods while their respective clients get buffer space
+   and load their buffers with data.  Since a NETBLT waiting for buffer
+   space is in a perfectly valid state, the protocol must have some
+   method for preventing the other end's death timer from expiring.  The
+   solution is to use a KEEPALIVE packet, which is sent repeatedly at
+   fixed intervals when a NETBLT cannot send other packets.  Since the
+   death timer is reset whenever a packet is received, it will never
+   expire as long as the other end sends packets.
+
+   The frequency with which KEEPALIVE packets are transmitted is
+   computed as follows:  At connection startup, each NETBLT chooses a
+   death-timer value and sends it to the other end in either the OPEN or
+   the RESPONSE packet.  The other end takes the death-timeout value and
+   uses it to compute a frequency with which to send KEEPALIVE packets.
+   The KEEPALIVE frequency should be high enough that several KEEPALIVE
+   packets can be lost before the other end's death timer expires (e.g.
+   death timer value divided by four).
+
+   The death timer value is relatively easy to estimate.  Since it is
+   continually reset, it need not be based on the transfer size.
+   Instead, it should be based at least in part on the type of
+   application using NETBLT.  User applications should have smaller
+   death timeout values to avoid forcing humans to wait long periods of
+   time for a death timeout to occur.  Machine applications can have
+   longer timeout values.
+
+5.3. Closing the Connection
+
+   There are three ways to close a connection: a connection close, a
+   "quit", or an "abort".
+
+5.3.1. Successful Transfer
+
+   After a successful data transfer, NETBLT closes the connection.  When
+   the sender is transmitting the last buffer of data, it sets a "last-
+   buffer" flag on every DATA packet in the buffer.  This means that no
+   NEW data will be transmitted.  The receiver knows the transfer has
+   completed successfully when all of the following are true: (1) it has
+   received DATA packets with a "last-buffer" flag set, (2) all its
+   control messages have been acknowledged, and (3) it has no
+   outstanding buffers with missing packets.  At that point, the
+   receiver is permitted to close its half of the connection.  The
+   sender knows the transfer has completed when the following are true:
+
+
+
+Clark, Lambert, & Zhang                                        [Page 13]
+
+RFC 998                                                       March 1987
+
+
+   (1) it has transmitted DATA packets with a "last-buffer" flag set and
+   (2) it has received OK messages for all its buffers.  At that point,
+   it "dallies" for a predetermined period of time before closing its
+   half of the connection.  If the NULL-ACK packet acknowledging the
+   receiver's last OK message was lost, the receiver has time to
+   retransmit the OK message, receive a new NULL-ACK, and recognize a
+   successful transfer.  The dally timer value MUST be based on the
+   receiver's control timer value; it must be long enough to allow the
+   receiver's control timer to expire so that the OK message can be re-
+   sent.  For this reason, all OK messages contain (in addition to new
+   burst size and burst rate values), the receiver's current control
+   timer value in milliseconds.  The sender uses this value to compute
+   its dally timer value.
+
+   Since the dally timer value may be quite large, the receiving NETBLT
+   is permitted to "short-circuit" the sending NETBLT's dally timer by
+   transmitting a DONE packet.  The DONE packet is transmitted when the
+   receiver knows the transfer has been successfully completed.  When
+   the sender receives a DONE packet, it is allowed to clear its dally
+   timer and close its half of the connection immediately.  The DONE
+   packet is not reliably transmitted, since failure to receive it only
+   means that the sending NETBLT will take longer time to close its half
+   of the connection (as it waits for its dally timer to clear)
+
+5.3.2. Client QUIT
+
+   During a NETBLT transfer, one client may send a QUIT packet to the
+   other if it thinks that the other client is malfunctioning.  Since
+   the QUIT occurs at a client level, the QUIT transmission can only
+   occur between buffer transmissions.  The NETBLT receiving the QUIT
+   packet can take no action other than immediately notifying its client
+   and transmitting a QUITACK packet.  The QUIT sender must time out and
+   retransmit until a QUITACK has been received or its death timer
+   expires.  The sender of the QUITACK dallies before quitting, so that
+   it can respond to a retransmitted QUIT.
+
+5.3.3. NETBLT ABORT
+
+   An ABORT takes place when a NETBLT layer thinks that it or its
+   opposite is malfunctioning.  Since the ABORT originates in the NETBLT
+   layer, it can be sent at any time.  The ABORT implies that the NETBLT
+   layer is malfunctioning, so no transmit reliability is expected, and
+   the sender can immediately close it connection.
+
+6. Protocol Layering Structure
+
+   NETBLT is implemented directly on top of the Internet Protocol (IP).
+   It has been assigned an official protocol number of 30 (decimal).
+
+
+
+
+
+
+Clark, Lambert, & Zhang                                        [Page 14]
+
+RFC 998                                                       March 1987
+
+
+7. Planned Enhancements
+
+   As currently specified, NETBLT has no algorithm for determining its
+   rate-control parameters (burst rate, burst size, etc.).  In initial
+   performance testing, these parameters have been set by the person
+   performing the test.  We are now exploring ways to have NETBLT set
+   and adjust its rate-control parameters automatically.
+
+8. Packet Formats
+
+   NETBLT packets are divided into three categories, all of which share
+   a common packet header.  First, there are those packets that travel
+   only from data sender to receiver; these contain the high-
+   acknowledged-sequence-numbers which the receiver uses for control
+   message transmission reliability.  These packets are the NULL-ACK,
+   DATA, and LDATA packets.  Second, there is a packet that travels only
+   from receiver to sender.  This is the CONTROL packet; each CONTROL
+   packet can contain an arbitrary number of control messages (GO, OK,
+   or RESEND), each with its own sequence number.  Finally, there are
+   those packets which either have special ways of insuring reliability,
+   or are not reliably transmitted.  These are the OPEN, RESPONSE,
+   REFUSED, QUIT, QUITACK, DONE, KEEPALIVE, and ABORT packets.  Of
+   these, all save the DONE packet can be sent by both sending and
+   receiving NETBLTs.
+
+   All packets are "longword-aligned", i.e. all packets are a multiple
+   of 4 bytes in length and all 4-byte fields start on a longword
+   boundary.  All arbitrary-length string fields are terminated with at
+   least one null byte, with extra null bytes added at the end to create
+   a field that is a multiple of 4 bytes long.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Clark, Lambert, & Zhang                                        [Page 15]
+
+RFC 998                                                       March 1987
+
+
+   Packet Formats for NETBLT
+
+   OPEN (type 0) and RESPONSE (type 1):
+
+                      1                   2                   3
+    1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+   +---------------+---------------+---------------+---------------+
+   |           Checksum            |    Version    |     Type      |
+   +---------------+---------------+---------------+---------------+
+   |           Length              |           Local Port          |
+   +---------------+---------------+---------------+---------------+
+   |        Foreign Port           | Longword Alignment Padding    |
+   +---------------+---------------+---------------+---------------+
+   |                       Connection Unique ID                    |
+   +---------------+---------------+---------------+---------------+
+   |                         Buffer Size                           |
+   +---------------+---------------+---------------+---------------+
+   |                       Transfer Size                           |
+   +---------------+---------------+---------------+---------------+
+   |        DATA packet size       |          Burst Size           |
+   +---------------+---------------+---------------+---------------+
+   |           Burst Rate          |       Death Timer Value       |
+   +---------------+---------------+---------------+---------------+
+   |       Reserved (MBZ)      |C|M| Maximum # Outstanding Buffers |
+   +---------------+---------------+---------------+---------------+
+   | Client String ...
+   +---------------+---------------+---------------
+                                     Longword Alignment Padding    |
+                    ---------------+-------------------------------+
+
+   Checksum: packet checksum (algorithm is described in the section
+   "Connection Setup")
+
+   Version: the NETBLT protocol version number
+
+   Type: the NETBLT packet type number (OPEN = 0, RESPONSE = 1,
+   etc.)
+
+   Length: the total length (NETBLT header plus data, if present)
+   of the NETBLT packet in bytes
+
+   Local Port: the local NETBLT's 16-bit port number
+
+   Foreign Port: the foreign NETBLT's 16-bit port number
+
+   Connection UID: the 32 bit connection UID specified in the
+   section "Connection Setup".
+
+   Buffer size: the size in bytes of each NETBLT buffer (save the
+   last)
+
+
+
+
+Clark, Lambert, & Zhang                                        [Page 16]
+
+RFC 998                                                       March 1987
+
+
+   Transfer size: (optional) the size in bytes of the transfer.
+
+   This is for client information only; the receiving NETBLT should
+   NOT make use of it.
+
+   Data packet size: length of each DATA packet in bytes
+
+   Burst Size: Number of DATA packets in a burst
+
+   Burst Rate: Transmit time in milliseconds of a single burst
+
+   Death timer: Packet sender's death timer value in seconds
+
+   "M": the transfer mode (0 = READ, 1 = WRITE)
+
+   "C": the DATA packet data checksum flag (0 = do not checksum
+   DATA packet data, 1 = do)
+
+   Maximum Outstanding Buffers: maximum number of buffers that can
+   be transferred before waiting for an OK message from the
+   receiving NETBLT.
+
+   Client string: an arbitrary, null-terminated, longword-aligned
+   string for use by NETBLT clients.
+
+   KEEPALIVE (type 2), QUITACK (type 4), and DONE (type 11)
+
+                      1                   2                   3
+    1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+   +---------------+---------------+---------------+---------------+
+   |           Checksum            |    Version    |     Type      |
+   +---------------+---------------+---------------+---------------+
+   |           Length              |           Local Port          |
+   +---------------+---------------+---------------+---------------+
+   |        Foreign Port           | Longword Alignment Padding    |
+   +---------------+---------------+---------------+---------------+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Clark, Lambert, & Zhang                                        [Page 17]
+
+RFC 998                                                       March 1987
+
+
+   QUIT (type 3), ABORT (type 5), and REFUSED (type 10)
+
+                      1                   2                   3
+    1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+   +---------------+---------------+---------------+---------------+
+   |           Checksum            |    Version    |     Type      |
+   +---------------+---------------+---------------+---------------+
+   |           Length              |           Local Port          |
+   +---------------+---------------+---------------+---------------+
+   |        Foreign Port           | Longword Alignment Padding    |
+   +---------------+---------------+---------------+---------------+
+   | Reason for QUIT/ABORT/REFUSE...
+   +---------------+---------------+---------------
+                                     Longword Alignment Padding    |
+                    ---------------+-------------------------------+
+
+   DATA (type 6) and LDATA (type 7):
+
+                      1                   2                   3
+    1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+   +---------------+---------------+---------------+---------------+
+   |           Checksum            |    Version    |     Type      |
+   +---------------+---------------+---------------+---------------+
+   |           Length              |           Local Port          |
+   +---------------+---------------+---------------+---------------+
+   |        Foreign Port           | Longword Alignment Padding    |
+   +---------------+---------------+---------------+---------------+
+   |                       Buffer Number                           |
+   +---------------+---------------+---------------+---------------+
+   | High Consecutive Seq Num Rcvd |         Packet Number         |
+   +---------------+---------------+---------------+---------------+
+   |    Data Area Checksum Value   |      Reserved (MBZ)         |L|
+   +---------------+---------------+---------------+---------------+
+
+   Buffer number: a 32 bit unique number assigned to every buffer.
+   Numbers are monotonically increasing.
+
+   High Consecutive Sequence Number Received: Highest control
+   message sequence number below which all sequence numbers received
+   are consecutive.
+
+   Packet number: monotonically increasing DATA packet identifier
+
+   Data Area Checksum Value: Checksum of the DATA packet's data.
+   Algorithm used is the same as that used to compute checksums of
+   other NETBLT packets.
+
+   "L" is a flag set when the buffer that this DATA packet belongs
+   to is the last buffer in the transfer.
+
+
+
+
+
+Clark, Lambert, & Zhang                                        [Page 18]
+
+RFC 998                                                       March 1987
+
+
+   NULL-ACK (type 8)
+
+                      1                   2                   3
+    1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+   +---------------+---------------+---------------+---------------+
+   |           Checksum            |    Version    |     Type      |
+   +---------------+---------------+---------------+---------------+
+   |           Length              |           Local Port          |
+   +---------------+---------------+---------------+---------------+
+   |        Foreign Port           | Longword Alignment Padding    |
+   +---------------+---------------+---------------+---------------+
+   | High Consecutive Seq Num Rcvd |        New Burst Size         |
+   +---------------+---------------+---------------+---------------+
+   |       New Burst Rate          |  Longword Alignment Padding   |
+   +---------------+---------------+---------------+---------------+
+
+   High Consecutive Sequence Number Received: same as in DATA/LDATA
+   packet
+
+   New Burst Size:  Burst size as negotiated from value given by
+   receiving NETBLT in OK message
+
+   New burst rate: Burst rate as negotiated from value given
+   by receiving NETBLT in OK message.  Value is in milliseconds.
+
+   CONTROL (type 9):
+
+                      1                   2                   3
+    1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+   +---------------+---------------+---------------+---------------+
+   |           Checksum            |    Version    |     Type      |
+   +---------------+---------------+---------------+---------------+
+   |           Length              |           Local Port          |
+   +---------------+---------------+---------------+---------------+
+   |        Foreign Port           | Longword Alignment Padding    |
+   +---------------+---------------+---------------+---------------+
+
+   Followed by any number of messages, each of which is longword
+   aligned, with the following formats:
+
+   GO message (type 0):
+
+                      1                   2                   3
+    1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+   +---------------+---------------+---------------+---------------+
+   |    Type       | Word Padding  |       Sequence Number         |
+   +---------------+---------------+---------------+---------------+
+   |                        Buffer Number                          |
+   +---------------+---------------+---------------+---------------+
+
+   Type: message type (GO = 0, OK = 1, RESEND = 2)
+
+
+
+Clark, Lambert, & Zhang                                        [Page 19]
+
+RFC 998                                                       March 1987
+
+
+   Sequence number: A 16 bit unique message number.  Sequence
+   numbers must be monotonically increasing, starting from 1.
+
+   Buffer number: as in DATA/LDATA packet
+
+   OK message (type 1):
+
+                      1                   2                   3
+    1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+   +---------------+---------------+---------------+---------------+
+   |    Type       | Word Padding  |       Sequence Number         |
+   +---------------+---------------+---------------+---------------+
+   |                        Buffer Number                          |
+   +---------------+---------------+---------------+---------------+
+   |    New Offered Burst Size     |   New Offered Burst Rate      |
+   +---------------+---------------+---------------+---------------+
+   | Current control timer value   | Longword Alignment Padding    |
+   +---------------+---------------+---------------+---------------+
+
+   New offered burst size: burst size for subsequent buffer
+   transfers, possibly based on performance information for previous
+   buffer transfers.
+
+   New offered burst rate: burst rate for subsequent buffer
+   transfers, possibly based on performance information for previous
+   buffer transfers.  Rate is in milliseconds.
+
+   Current control timer value: Receiving NETBLT's control timer
+   value in milliseconds.
+
+   RESEND Message (type 2):
+
+                      1                   2                   3
+    1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+   +---------------+---------------+---------------+---------------+
+   |    Type       | Word Padding  |       Sequence Number         |
+   +---------------+---------------+---------------+---------------+
+   |                        Buffer Number                          |
+   +---------------+---------------+---------------+---------------+
+   |  Number of Missing Packets    | Longword Alignment Padding    |
+   +---------------+---------------+---------------+---------------+
+   |       Packet Number (2 bytes) ...
+   +---------------+---------------+----------
+                                   |    Padding (if necessary)     |
+                        -----------+---------------+---------------+
+
+   Packet number:  the 16 bit data packet identifier found in each
+   DATA packet.
+
+
+
+
+
+
+Clark, Lambert, & Zhang                                        [Page 20]
+
+RFC 998                                                       March 1987
+
+
+NOTES:
+
+   <1>  When the buffer size is large, the variances in the round trip
+   delays of many packets may cancel each other out; this means the
+   variance value need not be very big.  This expectation will be
+   explored in further testing.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Clark, Lambert, & Zhang                                        [Page 21]
+