diff options
Diffstat (limited to 'doc/rfc/rfc8673.txt')
| -rw-r--r-- | doc/rfc/rfc8673.txt | 510 |
1 files changed, 510 insertions, 0 deletions
diff --git a/doc/rfc/rfc8673.txt b/doc/rfc/rfc8673.txt new file mode 100644 index 0000000..59a3d82 --- /dev/null +++ b/doc/rfc/rfc8673.txt @@ -0,0 +1,510 @@ + + + + +Internet Engineering Task Force (IETF) C. Pratt +Request for Comments: 8673 +Category: Experimental D. Thakore +ISSN: 2070-1721 CableLabs + B. Stark + AT&T + November 2019 + + + HTTP Random Access and Live Content + +Abstract + + To accommodate byte-range requests for content that has data appended + over time, this document defines semantics that allow an HTTP client + and a server to perform byte-range GET and HEAD requests that start + at an arbitrary byte offset within the representation and end at an + indeterminate offset. + +Status of This Memo + + This document is not an Internet Standards Track specification; it is + published for examination, experimental implementation, and + evaluation. + + This document defines an Experimental Protocol for the Internet + community. This document is a product of the Internet Engineering + Task Force (IETF). It represents the consensus of the IETF + community. It has received public review and has been approved for + publication by the Internet Engineering Steering Group (IESG). Not + all documents approved by the IESG are candidates for any level of + Internet Standard; see Section 2 of RFC 7841. + + Information about the current status of this document, any errata, + and how to provide feedback on it may be obtained at + https://www.rfc-editor.org/info/rfc8673. + +Copyright Notice + + Copyright (c) 2019 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (https://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the Simplified BSD License. + +Table of Contents + + 1. Introduction + 1.1. Notational Conventions + 2. Performing Range Requests on Random-Access Aggregating (Live) + Content + 2.1. Establishing the Randomly Accessible Byte Range + 2.2. Byte-Range Requests beyond the Randomly Accessible Byte + Range + 3. Other Applications of Random-Access Aggregating Content + 3.1. Requests Starting at the Aggregation/Live Point + 3.2. Shift-Buffer Representations + 4. Recommendations for Byte-Range Request last-byte-pos Values + 5. IANA Considerations + 6. Security Considerations + 7. References + 7.1. Normative References + 7.2. Informative References + Acknowledgements + Authors' Addresses + +1. Introduction + + Some Hypertext Transfer Protocol (HTTP) clients use byte-range + requests (range requests using the "bytes" range unit) to transfer + select portions of large representations [RFC7233]. In some cases, + large representations require content to be continuously or + periodically appended, such as representations consisting of live + audio or video sources, blockchain databases, and log files. Clients + cannot access the appended/live content using a range request with + the "bytes" range unit using the currently defined byte-range + semantics without accepting performance or behavior sacrifices that + are not acceptable for many applications. + + For instance, HTTP clients have the ability to access appended + content on an indeterminate-length resource by transferring the + entire representation from the beginning and continuing to read the + appended content as it's made available. Obviously, this is highly + inefficient for cases where the representation is large and only the + most recently appended content is needed by the client. + + Alternatively, clients can access appended content by sending + periodic, open-ended byte-range requests using the last known end + byte position as the range start. Performing low-frequency periodic + byte-range requests in this fashion (polling) introduces latency + since the client will necessarily be somewhat behind in transferring + the aggregated content, effectively resulting in the same kind of + latency issues with the segmented content transfer mechanisms in + "HTTP Live Streaming" (HLS) [RFC8216] and "Dynamic Adaptive Streaming + over HTTP" [MPEG-DASH]. While performing these range requests at + higher frequency can reduce this latency, it also incurs more + processing overhead and HTTP exchanges as many of the requests will + return no content, since content is usually aggregated in groups of + bytes (e.g., a video frame, audio sample, block, or log entry). + + This document describes a usage model for range requests that enables + efficient retrieval of representations that are appended to over time + by using large values and associated semantics for communicating + range end positions. This model allows representations to be + progressively delivered by servers as new content is added. It also + ensures compatibility with servers and intermediaries that don't + support this technique. + +1.1. Notational Conventions + + This document cites Augmented Backus-Naur Form (ABNF) productions + from [RFC7233], using the notation defined in [RFC5234]. + +2. Performing Range Requests on Random-Access Aggregating (Live) + Content + + This document recommends a two-step process for accessing resources + that have indeterminate-length representations. + + Two steps are necessary because of limitations with the range request + header fields and the Content-Range response header fields. A server + cannot know from a range request that a client wishes to receive a + response that does not have a definite end. More critically, the + header fields do not allow the server to signal that a resource has + indeterminate length without also providing a fixed portion of the + resource. + + A client first learns that the resource has a representation of + indeterminate length by requesting a range of the resource. The + server responds with the range that is available but indicates that + the length of the representation is unknown using the existing + Content-Range syntax. See Section 2.1 for details and examples. + + Once the client knows the resource has indeterminate length, it can + request a range with a very large end position from the resource. + The client chooses an explicit end value larger than can be + transferred in the foreseeable term. A server that supports range + requests of indeterminate length signals its understanding of the + client's indeterminate range request by indicating that the range it + is providing has a range end that exactly matches the client's + requested range end rather than a range that is bounded by what is + currently available. See Section 2.2 for details. + +2.1. Establishing the Randomly Accessible Byte Range + + Determining if a representation is continuously aggregating ("live") + and determining the randomly accessible byte range can both be + performed using the existing definition for an open-ended byte-range + request. Specifically, Section 2.1 of [RFC7233] defines a byte-range + request of the form: + + byte-range-spec = first-byte-pos "-" [ last-byte-pos ] + + which allows a client to send a HEAD request with a first-byte-pos + and leave last-byte-pos absent. A server that receives a satisfiable + byte-range request (with first-byte-pos smaller than the current + representation length) may respond with a 206 status code (Partial + Content) with a Content-Range header field indicating the currently + satisfiable byte range. For example: + + HEAD /resource HTTP/1.1 + Host: example.com + Range: bytes=0- + + returns a response of the form: + + HTTP/1.1 206 Partial Content + Content-Range: bytes 0-1234567/* + + from the server indicating that (1) the complete representation + length is unknown (via the "*" in place of the complete-length field) + and (2) only bytes 0-1234567 were accessible at the time the request + was processed by the server. The client can infer from this response + that bytes 0-1234567 of the representation can be requested and + transfer can be performed immediately. + +2.2. Byte-Range Requests beyond the Randomly Accessible Byte Range + + Once a client has determined that a representation has an + indeterminate length and established the byte range that can be + accessed, it may want to perform a request with a start position + within the randomly accessible content range and an end position at + an indefinite/live point -- a point where the byte-range GET request + is fulfilled on-demand as the content is aggregated. + + For example, for a large video asset, a client may wish to start a + content transfer from the video "key" frame immediately before the + point of aggregation and continue the content transfer indefinitely + as content is aggregated, in order to support low-latency startup of + a live video stream. + + Unlike a byte-range request header field, a byte-content-range + response header field cannot be "open-ended", per Section 4.2 of + [RFC7233]: + + byte-content-range = bytes-unit SP + ( byte-range-resp / unsatisfied-range ) + + byte-range-resp = byte-range "/" ( complete-length / "*" ) + byte-range = first-byte-pos "-" last-byte-pos + unsatisfied-range = "*/" complete-length + + complete-length = 1*DIGIT + + Specifically, last-byte-pos is required in byte-range. So, in order + to preserve interoperability with existing HTTP clients, servers, + proxies, and caches, this document proposes a mechanism for a client + to indicate support for handling an indeterminate-length byte-range + response and a mechanism for a server to indicate if/when it's + providing an indeterminate-length response. + + A client can indicate support for handling indeterminate-length byte- + range responses by providing a very large value for the last-byte-pos + in the byte-range request. For example, a client can perform a byte- + range GET request of the form: + + GET /resource HTTP/1.1 + Host: example.com + Range: bytes=1230000-999999999999 + + where the last-byte-pos in the request is much larger than the last- + byte-pos returned in response to an open-ended byte-range HEAD + request, as described above, and much larger than the expected + maximum size of the representation. See Section 6 for range value + considerations. + + In response, a server may indicate that it is supplying a + continuously aggregating/live response by supplying the client + request's last-byte-pos in the Content-Range response header field. + + For example: + + GET /resource HTTP/1.1 + Host: example.com + Range: bytes=1230000-999999999999 + + returns + + HTTP/1.1 206 Partial Content + Content-Range: bytes 1230000-999999999999/* + + from the server to indicate that the response will start at byte + 1230000 and continue indefinitely to include all aggregated content, + as it becomes available. + + A server that doesn't support or supply a continuously aggregating/ + live response will supply the currently satisfiable byte range, as it + would with an open-ended byte request. + + For example: + + GET /resource HTTP/1.1 + Host: example.com + Range: bytes=1230000-999999999999 + + returns + + HTTP/1.1 206 Partial Content + Content-Range: bytes 1230000-1234567/* + + from the server to indicate that the response will start at byte + 1230000, end at byte 1234567, and not include any aggregated content. + This is the response expected from a typical HTTP server -- one that + doesn't support byte-range requests on aggregating content. + + A client that doesn't receive a response indicating it is + continuously aggregating must use other means to access aggregated + content (e.g., periodic byte-range polling). + + A server that does return a continuously aggregating/live response + should return data using chunked transfer coding and not provide a + Content-Length header field. A 0-length chunk indicates the end of + the transfer, per Section 4.1 of [RFC7230]. + +3. Other Applications of Random-Access Aggregating Content + +3.1. Requests Starting at the Aggregation/Live Point + + A client that wishes to only receive newly aggregated portions of a + resource (i.e., start at the live point) can use a HEAD request to + learn what range the server has currently available and initiate an + indeterminate-length transfer. For example: + + HEAD /resource HTTP/1.1 + Host: example.com + Range: bytes=0- + + with the Content-Range response header field indicating the range (or + ranges) available. For example: + + 206 Partial Content + Content-Range: bytes 0-1234567/* + + The client can then issue a request for a range starting at the end + value (using a very large value for the end of a range) and receive + only new content. + + For example: + + GET /resource HTTP/1.1 + Host: example.com + Range: bytes=1234567-999999999999 + + with a server returning a Content-Range response indicating that an + indeterminate-length response body will be provided: + + 206 Partial Content + Content-Range: bytes 1234567-999999999999/* + +3.2. Shift-Buffer Representations + + Some representations lend themselves to front-end content removal in + addition to aggregation. While still supporting random access, + representations of this type have a portion at the beginning (the "0" + end) of the randomly accessible region that becomes inaccessible over + time. Examples of this kind of representation would be an audio- + video time-shift buffer or a rolling log file. + + For example, a range request containing: + + HEAD /resource HTTP/1.1 + Host: example.com + Range: bytes=0- + + returns + + 206 Partial Content + Content-Range: bytes 1000000-1234567/* + + indicating that the first 1000000 bytes were not accessible at the + time the HEAD request was processed. Subsequent HEAD requests could + return: + + Content-Range: bytes 1000000-1234567/* + + Content-Range: bytes 1010000-1244567/* + + Content-Range: bytes 1020000-1254567/* + + Note though that the difference between the first-byte-pos and last- + byte-pos need not be constant. + + The client could then follow up with a GET range request containing: + + GET /resource HTTP/1.1 + Host: example.com + Range: bytes=1020000-999999999999 + + with the server returning + + 206 Partial Content + Content-Range: bytes 1020000-999999999999/* + + with the response body returning bytes 1020000-1254567 immediately + and aggregated/live data being returned as the content is aggregated. + + A server that doesn't support or supply a continuously aggregating/ + live response will supply the currently satisfiable byte range, as it + would with an open-ended byte request. For example: + + GET /resource HTTP/1.1 + Host: example.com + Range: bytes=0-999999999999 + + returns + + HTTP/1.1 206 Partial Content + Content-Range: bytes 1020000-1254567/* + + from the server to indicate that the response will start at byte + 1020000, end at byte 1254567, and not include any aggregated content. + This is the response expected from a typical HTTP server -- one that + doesn't support byte-range requests on aggregating content. + + Note that responses to GET requests performed on shift-buffer + representations using Range headers can be cached by intermediaries, + since the Content-Range response header indicates which portion of + the representation is being returned in the response body. However, + GET requests without a Range header cannot be cached since the first + byte of the response body can vary from request to request. To + ensure GET requests without Range headers on shift-buffer + representations are not cached, servers hosting a shift-buffer + representation should either not return a 200-level response (e.g., + send a 300-level redirect response with a URI that represents the + current start of the shift buffer) or indicate the response is non- + cacheable. See [RFC7234] for details on HTTP cache control. + +4. Recommendations for Byte-Range Request last-byte-pos Values + + While it would be ideal to define a single large last-byte-pos value + for byte-range requests, there's no single value that would work for + all applications and platforms. For example, JavaScript numbers + cannot represent all integer values above 2^^53, so a JavaScript + application may want to use 2^^53-1 for last-byte-pos. This value, + however, would not be sufficient for all applications, such as long- + duration high-bitrate streams. So 2^^53-1 (9007199254740991) is + recommended as a last-byte-pos unless an application has a good + justification to use a smaller or larger value. For example, if it + is always known that the resource won't exceed a value smaller than + the recommended last-byte-pos for an application, a smaller value can + be used. If it's likely that an application will utilize resources + larger than the recommended last-byte-pos (such as a continuously + aggregating high-bitrate media stream), a larger value should be + used. + + Note that, in accordance with the semantics defined above, servers + that support random-access live content will need to return the last- + byte-pos provided in the byte-range request in some cases -- even if + the last-byte-pos cannot be represented as a numerical value + internally by the server. As is the case with any continuously + aggregating/live resource, the server should terminate the content + transfer when the end of the resource is reached -- whether the end + is due to termination of the content source or the content length + exceeds the server's maximum representation length. + +5. IANA Considerations + + This document has no IANA actions. + +6. Security Considerations + + As described above, servers need to be prepared to receive last-byte- + pos values in range requests that are numerically larger than the + server implementation supports and return these values in Content- + Range response header fields. Servers should check the last-byte-pos + value before converting and storing them into numeric form to ensure + the value doesn't cause an overflow or index incorrect data. The + simplest way to satisfy the live-range semantics defined in this + document without potential overflow issues is to store the last-byte- + pos as a string value and return it in the byte-range Content-Range + response header's last-byte-pos field. + +7. References + +7.1. Normative References + + [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax + Specifications: ABNF", STD 68, RFC 5234, + DOI 10.17487/RFC5234, January 2008, + <https://www.rfc-editor.org/info/rfc5234>. + + [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer + Protocol (HTTP/1.1): Message Syntax and Routing", + RFC 7230, DOI 10.17487/RFC7230, June 2014, + <https://www.rfc-editor.org/info/rfc7230>. + + [RFC7233] Fielding, R., Ed., Lafon, Y., Ed., and J. Reschke, Ed., + "Hypertext Transfer Protocol (HTTP/1.1): Range Requests", + RFC 7233, DOI 10.17487/RFC7233, June 2014, + <https://www.rfc-editor.org/info/rfc7233>. + + [RFC7234] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, + Ed., "Hypertext Transfer Protocol (HTTP/1.1): Caching", + RFC 7234, DOI 10.17487/RFC7234, June 2014, + <https://www.rfc-editor.org/info/rfc7234>. + +7.2. Informative References + + [MPEG-DASH] + ISO, "Information technology -- Dynamic adaptive streaming + over HTTP (DASH) -- Part 1: Media presentation description + and segment formats", ISO/IEC 23009-1, August 2019, + <https://www.iso.org/standard/75485.html>. + + [RFC8216] Pantos, R., Ed. and W. May, "HTTP Live Streaming", + RFC 8216, DOI 10.17487/RFC8216, August 2017, + <https://www.rfc-editor.org/info/rfc8216>. + +Acknowledgements + + The authors would like to thank Mark Nottingham, Patrick McManus, + Julian Reschke, Remy Lebeau, Rodger Combs, Thorsten Lohmar, Martin + Thompson, Adrien de Croy, K. Morgan, Roy T. Fielding, and Jeremy + Poulter. + +Authors' Addresses + + Craig Pratt + Portland, OR 97229 + United States of America + + Email: pratt@acm.org + + + Darshak Thakore + CableLabs + 858 Coal Creek Circle + Louisville, CO 80027 + United States of America + + Email: d.thakore@cablelabs.com + + + Barbara Stark + AT&T + Atlanta, GA + United States of America + + Email: barbara.stark@att.com |