summaryrefslogtreecommitdiff
path: root/doc/rfc/rfc9309.txt
diff options
context:
space:
mode:
Diffstat (limited to 'doc/rfc/rfc9309.txt')
-rw-r--r--doc/rfc/rfc9309.txt607
1 files changed, 607 insertions, 0 deletions
diff --git a/doc/rfc/rfc9309.txt b/doc/rfc/rfc9309.txt
new file mode 100644
index 0000000..45a579a
--- /dev/null
+++ b/doc/rfc/rfc9309.txt
@@ -0,0 +1,607 @@
+
+
+
+
+Internet Engineering Task Force (IETF) M. Koster
+Request for Comments: 9309
+Category: Standards Track G. Illyes
+ISSN: 2070-1721 H. Zeller
+ L. Sassman
+ Google LLC
+ September 2022
+
+
+ Robots Exclusion Protocol
+
+Abstract
+
+ This document specifies and extends the "Robots Exclusion Protocol"
+ method originally defined by Martijn Koster in 1994 for service
+ owners to control how content served by their services may be
+ accessed, if at all, by automatic clients known as crawlers.
+ Specifically, it adds definition language for the protocol,
+ instructions for handling errors, and instructions for caching.
+
+Status of This Memo
+
+ This is an Internet Standards Track document.
+
+ This document is a product of the Internet Engineering Task Force
+ (IETF). It represents the consensus of the IETF community. It has
+ received public review and has been approved for publication by the
+ Internet Engineering Steering Group (IESG). Further information on
+ Internet Standards is available in Section 2 of RFC 7841.
+
+ Information about the current status of this document, any errata,
+ and how to provide feedback on it may be obtained at
+ https://www.rfc-editor.org/info/rfc9309.
+
+Copyright Notice
+
+ Copyright (c) 2022 IETF Trust and the persons identified as the
+ document authors. All rights reserved.
+
+ This document is subject to BCP 78 and the IETF Trust's Legal
+ Provisions Relating to IETF Documents
+ (https://trustee.ietf.org/license-info) in effect on the date of
+ publication of this document. Please review these documents
+ carefully, as they describe your rights and restrictions with respect
+ to this document. Code Components extracted from this document must
+ include Revised BSD License text as described in Section 4.e of the
+ Trust Legal Provisions and are provided without warranty as described
+ in the Revised BSD License.
+
+Table of Contents
+
+ 1. Introduction
+ 1.1. Requirements Language
+ 2. Specification
+ 2.1. Protocol Definition
+ 2.2. Formal Syntax
+ 2.2.1. The User-Agent Line
+ 2.2.2. The "Allow" and "Disallow" Lines
+ 2.2.3. Special Characters
+ 2.2.4. Other Records
+ 2.3. Access Method
+ 2.3.1. Access Results
+ 2.3.1.1. Successful Access
+ 2.3.1.2. Redirects
+ 2.3.1.3. "Unavailable" Status
+ 2.3.1.4. "Unreachable" Status
+ 2.3.1.5. Parsing Errors
+ 2.4. Caching
+ 2.5. Limits
+ 3. Security Considerations
+ 4. IANA Considerations
+ 5. Examples
+ 5.1. Simple Example
+ 5.2. Longest Match
+ 6. References
+ 6.1. Normative References
+ 6.2. Informative References
+ Authors' Addresses
+
+1. Introduction
+
+ This document applies to services that provide resources that clients
+ can access through URIs as defined in [RFC3986]. For example, in the
+ context of HTTP, a browser is a client that displays the content of a
+ web page.
+
+ Crawlers are automated clients. Search engines, for instance, have
+ crawlers to recursively traverse links for indexing as defined in
+ [RFC8288].
+
+ It may be inconvenient for service owners if crawlers visit the
+ entirety of their URI space. This document specifies the rules
+ originally defined by the "Robots Exclusion Protocol" [ROBOTSTXT]
+ that crawlers are requested to honor when accessing URIs.
+
+ These rules are not a form of access authorization.
+
+1.1. Requirements Language
+
+ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+ "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
+ "OPTIONAL" in this document are to be interpreted as described in
+ BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
+ capitals, as shown here.
+
+2. Specification
+
+2.1. Protocol Definition
+
+ The protocol language consists of rule(s) and group(s) that the
+ service makes available in a file named "robots.txt" as described in
+ Section 2.3:
+
+ Rule: A line with a key-value pair that defines how a crawler may
+ access URIs. See Section 2.2.2.
+
+ Group: One or more user-agent lines that are followed by one or more
+ rules. The group is terminated by a user-agent line or end of
+ file. See Section 2.2.1. The last group may have no rules, which
+ means it implicitly allows everything.
+
+2.2. Formal Syntax
+
+ Below is an Augmented Backus-Naur Form (ABNF) description, as
+ described in [RFC5234].
+
+ robotstxt = *(group / emptyline)
+ group = startgroupline ; We start with a user-agent
+ ; line
+ *(startgroupline / emptyline) ; ... and possibly more
+ ; user-agent lines
+ *(rule / emptyline) ; followed by rules relevant
+ ; for the preceding
+ ; user-agent lines
+
+ startgroupline = *WS "user-agent" *WS ":" *WS product-token EOL
+
+ rule = *WS ("allow" / "disallow") *WS ":"
+ *WS (path-pattern / empty-pattern) EOL
+
+ ; parser implementors: define additional lines you need (for
+ ; example, Sitemaps).
+
+ product-token = identifier / "*"
+ path-pattern = "/" *UTF8-char-noctl ; valid URI path pattern
+ empty-pattern = *WS
+
+ identifier = 1*(%x2D / %x41-5A / %x5F / %x61-7A)
+ comment = "#" *(UTF8-char-noctl / WS / "#")
+ emptyline = EOL
+ EOL = *WS [comment] NL ; end-of-line may have
+ ; optional trailing comment
+ NL = %x0D / %x0A / %x0D.0A
+ WS = %x20 / %x09
+
+ ; UTF8 derived from RFC 3629, but excluding control characters
+
+ UTF8-char-noctl = UTF8-1-noctl / UTF8-2 / UTF8-3 / UTF8-4
+ UTF8-1-noctl = %x21 / %x22 / %x24-7F ; excluding control, space, "#"
+ UTF8-2 = %xC2-DF UTF8-tail
+ UTF8-3 = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2UTF8-tail /
+ %xED %x80-9F UTF8-tail / %xEE-EF 2UTF8-tail
+ UTF8-4 = %xF0 %x90-BF 2UTF8-tail / %xF1-F3 3UTF8-tail /
+ %xF4 %x80-8F 2UTF8-tail
+
+ UTF8-tail = %x80-BF
+
+2.2.1. The User-Agent Line
+
+ Crawlers set their own name, which is called a product token, to find
+ relevant groups. The product token MUST contain only uppercase and
+ lowercase letters ("a-z" and "A-Z"), underscores ("_"), and hyphens
+ ("-"). The product token SHOULD be a substring of the identification
+ string that the crawler sends to the service. For example, in the
+ case of HTTP [RFC9110], the product token SHOULD be a substring in
+ the User-Agent header. The identification string SHOULD describe the
+ purpose of the crawler. Here's an example of a User-Agent HTTP
+ request header with a link pointing to a page describing the purpose
+ of the ExampleBot crawler, which appears as a substring in the User-
+ Agent HTTP header and as a product token in the robots.txt user-agent
+ line:
+
+ +==========================================+========================+
+ | User-Agent HTTP header | robots.txt user-agent |
+ | | line |
+ +==========================================+========================+
+ | User-Agent: Mozilla/5.0 (compatible; | user-agent: ExampleBot |
+ | ExampleBot/0.1; | |
+ | https://www.example.com/bot.html) | |
+ +------------------------------------------+------------------------+
+
+ Figure 1: Example of a User-Agent HTTP header and robots.txt
+ user-agent line for the ExampleBot product token
+
+ Note that the product token (ExampleBot) is a substring of the User-
+ Agent HTTP header.
+
+ Crawlers MUST use case-insensitive matching to find the group that
+ matches the product token and then obey the rules of the group. If
+ there is more than one group matching the user-agent, the matching
+ groups' rules MUST be combined into one group and parsed according to
+ Section 2.2.2.
+
+ +========================================+========================+
+ | Two groups that match the same product | Merged group |
+ | token exactly | |
+ +========================================+========================+
+ | user-agent: ExampleBot | user-agent: ExampleBot |
+ | disallow: /foo | disallow: /foo |
+ | disallow: /bar | disallow: /bar |
+ | | disallow: /baz |
+ | user-agent: ExampleBot | |
+ | disallow: /baz | |
+ +----------------------------------------+------------------------+
+
+ Figure 2: Example of how to merge two robots.txt groups that
+ match the same product token
+
+ If no matching group exists, crawlers MUST obey the group with a
+ user-agent line with the "*" value, if present.
+
+ +==================================+======================+
+ | Two groups that don't explicitly | Applicable group for |
+ | match ExampleBot | ExampleBot |
+ +==================================+======================+
+ | user-agent: * | user-agent: * |
+ | disallow: /foo | disallow: /foo |
+ | disallow: /bar | disallow: /bar |
+ | | |
+ | user-agent: BazBot | |
+ | disallow: /baz | |
+ +----------------------------------+----------------------+
+
+ Figure 3: Example of no matching groups other than the "*" for
+ the ExampleBot product token
+
+ If no group matches the product token and there is no group with a
+ user-agent line with the "*" value, or no groups are present at all,
+ no rules apply.
+
+2.2.2. The "Allow" and "Disallow" Lines
+
+ These lines indicate whether accessing a URI that matches the
+ corresponding path is allowed or disallowed.
+
+ To evaluate if access to a URI is allowed, a crawler MUST match the
+ paths in "allow" and "disallow" rules against the URI. The matching
+ SHOULD be case sensitive. The matching MUST start with the first
+ octet of the path. The most specific match found MUST be used. The
+ most specific match is the match that has the most octets. Duplicate
+ rules in a group MAY be deduplicated. If an "allow" rule and a
+ "disallow" rule are equivalent, then the "allow" rule SHOULD be used.
+ If no match is found amongst the rules in a group for a matching
+ user-agent or there are no rules in the group, the URI is allowed.
+ The /robots.txt URI is implicitly allowed.
+
+ Octets in the URI and robots.txt paths outside the range of the ASCII
+ coded character set, and those in the reserved range defined by
+ [RFC3986], MUST be percent-encoded as defined by [RFC3986] prior to
+ comparison.
+
+ If a percent-encoded ASCII octet is encountered in the URI, it MUST
+ be unencoded prior to comparison, unless it is a reserved character
+ in the URI as defined by [RFC3986] or the character is outside the
+ unreserved character range. The match evaluates positively if and
+ only if the end of the path from the rule is reached before a
+ difference in octets is encountered.
+
+ For example:
+
+ +==================+=======================+=======================+
+ | Path | Encoded Path | Path to Match |
+ +==================+=======================+=======================+
+ | /foo/bar?baz=quz | /foo/bar?baz=quz | /foo/bar?baz=quz |
+ +------------------+-----------------------+-----------------------+
+ | /foo/bar?baz= | /foo/bar?baz= | /foo/bar?baz= |
+ | https://foo.bar | https%3A%2F%2Ffoo.bar | https%3A%2F%2Ffoo.bar |
+ +------------------+-----------------------+-----------------------+
+ | /foo/bar/ | /foo/bar/%E3%83%84 | /foo/bar/%E3%83%84 |
+ | U+E38384 | | |
+ +------------------+-----------------------+-----------------------+
+ | /foo/ | /foo/bar/%E3%83%84 | /foo/bar/%E3%83%84 |
+ | bar/%E3%83%84 | | |
+ +------------------+-----------------------+-----------------------+
+ | /foo/ | /foo/bar/%62%61%7A | /foo/bar/baz |
+ | bar/%62%61%7A | | |
+ +------------------+-----------------------+-----------------------+
+
+ Figure 4: Examples of matching percent-encoded URI components
+
+ The crawler SHOULD ignore "disallow" and "allow" rules that are not
+ in any group (for example, any rule that precedes the first user-
+ agent line).
+
+ Implementors MAY bridge encoding mismatches if they detect that the
+ robots.txt file is not UTF-8 encoded.
+
+2.2.3. Special Characters
+
+ Crawlers MUST support the following special characters:
+
+ +===========+===================+==============================+
+ | Character | Description | Example |
+ +===========+===================+==============================+
+ | # | Designates a line | allow: / # comment in line |
+ | | comment. | |
+ | | | # comment on its own line |
+ +-----------+-------------------+------------------------------+
+ | $ | Designates the | allow: /this/path/exactly$ |
+ | | end of the match | |
+ | | pattern. | |
+ +-----------+-------------------+------------------------------+
+ | * | Designates 0 or | allow: /this/*/exactly |
+ | | more instances of | |
+ | | any character. | |
+ +-----------+-------------------+------------------------------+
+
+ Figure 5: List of special characters in robots.txt files
+
+ If crawlers match special characters verbatim in the URI, crawlers
+ SHOULD use "%" encoding. For example:
+
+ +============================+====================================+
+ | Percent-encoded Pattern | URI |
+ +============================+====================================+
+ | /path/file-with-a-%2A.html | https://www.example.com/path/ |
+ | | file-with-a-*.html |
+ +----------------------------+------------------------------------+
+ | /path/foo-%24 | https://www.example.com/path/foo-$ |
+ +----------------------------+------------------------------------+
+
+ Figure 6: Example of percent-encoding
+
+2.2.4. Other Records
+
+ Crawlers MAY interpret other records that are not part of the
+ robots.txt protocol -- for example, "Sitemaps" [SITEMAPS]. Crawlers
+ MAY be lenient when interpreting other records. For example,
+ crawlers may accept common misspellings of the record.
+
+ Parsing of other records MUST NOT interfere with the parsing of
+ explicitly defined records in Section 2. For example, a "Sitemaps"
+ record MUST NOT terminate a group.
+
+2.3. Access Method
+
+ The rules MUST be accessible in a file named "/robots.txt" (all
+ lowercase) in the top-level path of the service. The file MUST be
+ UTF-8 encoded (as defined in [RFC3629]) and Internet Media Type
+ "text/plain" (as defined in [RFC2046]).
+
+ As per [RFC3986], the URI of the robots.txt file is:
+
+ "scheme:[//authority]/robots.txt"
+
+ For example, in the context of HTTP or FTP, the URI is:
+
+ https://www.example.com/robots.txt
+
+ ftp://ftp.example.com/robots.txt
+
+2.3.1. Access Results
+
+2.3.1.1. Successful Access
+
+ If the crawler successfully downloads the robots.txt file, the
+ crawler MUST follow the parseable rules.
+
+2.3.1.2. Redirects
+
+ It's possible that a server responds to a robots.txt fetch request
+ with a redirect, such as HTTP 301 or HTTP 302 in the case of HTTP.
+ The crawlers SHOULD follow at least five consecutive redirects, even
+ across authorities (for example, hosts in the case of HTTP).
+
+ If a robots.txt file is reached within five consecutive redirects,
+ the robots.txt file MUST be fetched, parsed, and its rules followed
+ in the context of the initial authority.
+
+ If there are more than five consecutive redirects, crawlers MAY
+ assume that the robots.txt file is unavailable.
+
+2.3.1.3. "Unavailable" Status
+
+ "Unavailable" means the crawler tries to fetch the robots.txt file
+ and the server responds with status codes indicating that the
+ resource in question is unavailable. For example, in the context of
+ HTTP, such status codes are in the 400-499 range.
+
+ If a server status code indicates that the robots.txt file is
+ unavailable to the crawler, then the crawler MAY access any resources
+ on the server.
+
+2.3.1.4. "Unreachable" Status
+
+ If the robots.txt file is unreachable due to server or network
+ errors, this means the robots.txt file is undefined and the crawler
+ MUST assume complete disallow. For example, in the context of HTTP,
+ server errors are identified by status codes in the 500-599 range.
+
+ If the robots.txt file is undefined for a reasonably long period of
+ time (for example, 30 days), crawlers MAY assume that the robots.txt
+ file is unavailable as defined in Section 2.3.1.3 or continue to use
+ a cached copy.
+
+2.3.1.5. Parsing Errors
+
+ Crawlers MUST try to parse each line of the robots.txt file.
+ Crawlers MUST use the parseable rules.
+
+2.4. Caching
+
+ Crawlers MAY cache the fetched robots.txt file's contents. Crawlers
+ MAY use standard cache control as defined in [RFC9111]. Crawlers
+ SHOULD NOT use the cached version for more than 24 hours, unless the
+ robots.txt file is unreachable.
+
+2.5. Limits
+
+ Crawlers SHOULD impose a parsing limit to protect their systems; see
+ Section 3. The parsing limit MUST be at least 500 kibibytes [KiB].
+
+3. Security Considerations
+
+ The Robots Exclusion Protocol is not a substitute for valid content
+ security measures. Listing paths in the robots.txt file exposes them
+ publicly and thus makes the paths discoverable. To control access to
+ the URI paths in a robots.txt file, users of the protocol should
+ employ a valid security measure relevant to the application layer on
+ which the robots.txt file is served -- for example, in the case of
+ HTTP, HTTP Authentication as defined in [RFC9110].
+
+ To protect against attacks against their system, implementors of
+ robots.txt parsing and matching logic should take the following
+ considerations into account:
+
+ Memory management: Section 2.5 defines the lower limit of bytes that
+ must be processed, which inherently also protects the parser from
+ out-of-memory scenarios.
+
+ Invalid characters: Section 2.2 defines a set of characters that
+ parsers and matchers can expect in robots.txt files. Out-of-bound
+ characters should be rejected as invalid, which limits the
+ available attack vectors that attempt to compromise the system.
+
+ Untrusted content: Implementors should treat the content of a
+ robots.txt file as untrusted content, as defined by the
+ specification of the application layer used. For example, in the
+ context of HTTP, implementors should follow the Security
+ Considerations section of [RFC9110].
+
+4. IANA Considerations
+
+ This document has no IANA actions.
+
+5. Examples
+
+5.1. Simple Example
+
+ The following example shows:
+
+ *: A group that's relevant to all user agents that don't have an
+ explicitly defined matching group. It allows access to the URLs
+ with the /publications/ path prefix, and it restricts access to
+ the URLs with the /example/ path prefix and to all URLs with a
+ .gif suffix. The "*" character designates any character,
+ including the otherwise-required forward slash; see Section 2.2.
+
+ foobot: A regular case. A single user agent followed by rules. The
+ crawler only has access to two URL path prefixes on the site --
+ /example/page.html and /example/allowed.gif. The rules of the
+ group are missing the optional space character, which is
+ acceptable as defined in Section 2.2.
+
+ barbot and bazbot: A group that's relevant for more than one user
+ agent. The crawlers are not allowed to access the URLs with the
+ /example/page.html path prefix but otherwise have unrestricted
+ access to the rest of the URLs on the site.
+
+ quxbot: An empty group at the end of the file. The crawler has
+ unrestricted access to the URLs on the site.
+
+ User-Agent: *
+ Disallow: *.gif$
+ Disallow: /example/
+ Allow: /publications/
+
+ User-Agent: foobot
+ Disallow:/
+ Allow:/example/page.html
+ Allow:/example/allowed.gif
+
+ User-Agent: barbot
+ User-Agent: bazbot
+ Disallow: /example/page.html
+
+ User-Agent: quxbot
+
+ EOF
+
+5.2. Longest Match
+
+ The following example shows that in the case of two rules, the
+ longest one is used for matching. In the following case,
+ /example/page/disallowed.gif MUST be used for the URI
+ example.com/example/page/disallow.gif.
+
+ User-Agent: foobot
+ Allow: /example/page/
+ Disallow: /example/page/disallowed.gif
+
+6. References
+
+6.1. Normative References
+
+ [RFC2046] Freed, N. and N. Borenstein, "Multipurpose Internet Mail
+ Extensions (MIME) Part Two: Media Types", RFC 2046,
+ DOI 10.17487/RFC2046, November 1996,
+ <https://www.rfc-editor.org/info/rfc2046>.
+
+ [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
+ Requirement Levels", BCP 14, RFC 2119,
+ DOI 10.17487/RFC2119, March 1997,
+ <https://www.rfc-editor.org/info/rfc2119>.
+
+ [RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO
+ 10646", STD 63, RFC 3629, DOI 10.17487/RFC3629, November
+ 2003, <https://www.rfc-editor.org/info/rfc3629>.
+
+ [RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform
+ Resource Identifier (URI): Generic Syntax", STD 66,
+ RFC 3986, DOI 10.17487/RFC3986, January 2005,
+ <https://www.rfc-editor.org/info/rfc3986>.
+
+ [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
+ Specifications: ABNF", STD 68, RFC 5234,
+ DOI 10.17487/RFC5234, January 2008,
+ <https://www.rfc-editor.org/info/rfc5234>.
+
+ [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
+ 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
+ May 2017, <https://www.rfc-editor.org/info/rfc8174>.
+
+ [RFC8288] Nottingham, M., "Web Linking", RFC 8288,
+ DOI 10.17487/RFC8288, October 2017,
+ <https://www.rfc-editor.org/info/rfc8288>.
+
+ [RFC9110] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke,
+ Ed., "HTTP Semantics", STD 97, RFC 9110,
+ DOI 10.17487/RFC9110, June 2022,
+ <https://www.rfc-editor.org/info/rfc9110>.
+
+ [RFC9111] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke,
+ Ed., "HTTP Caching", STD 98, RFC 9111,
+ DOI 10.17487/RFC9111, June 2022,
+ <https://www.rfc-editor.org/info/rfc9111>.
+
+6.2. Informative References
+
+ [KiB] "Kibibyte", Simple English Wikipedia, the free
+ encyclopedia, 17 September 2020,
+ <https://simple.wikipedia.org/wiki/Kibibyte>.
+
+ [ROBOTSTXT]
+ "The Web Robots Pages (including /robots.txt)", 2007,
+ <https://www.robotstxt.org/>.
+
+ [SITEMAPS] "What are Sitemaps? (Sitemap protocol)", April 2020,
+ <https://www.sitemaps.org/index.html>.
+
+Authors' Addresses
+
+ Martijn Koster
+ Stalworthy Manor Farm
+ Suton Lane
+ Wymondham, Norfolk
+ NR18 9JG
+ United Kingdom
+ Email: m.koster@greenhills.co.uk
+
+
+ Gary Illyes
+ Google LLC
+ Brandschenkestrasse 110
+ CH-8002 Zürich
+ Switzerland
+ Email: garyillyes@google.com
+
+
+ Henner Zeller
+ Google LLC
+ 1600 Amphitheatre Pkwy
+ Mountain View, CA 94043
+ United States of America
+ Email: henner@google.com
+
+
+ Lizzi Sassman
+ Google LLC
+ Brandschenkestrasse 110
+ CH-8002 Zürich
+ Switzerland
+ Email: lizzi@google.com