diff options
Diffstat (limited to 'doc/rfc/rfc5537.txt')
| -rw-r--r-- | doc/rfc/rfc5537.txt | 2691 |
1 files changed, 2691 insertions, 0 deletions
diff --git a/doc/rfc/rfc5537.txt b/doc/rfc/rfc5537.txt new file mode 100644 index 0000000..741fa3f --- /dev/null +++ b/doc/rfc/rfc5537.txt @@ -0,0 +1,2691 @@ + + + + + + +Network Working Group R. Allbery, Ed. +Request for Comments: 5537 Stanford University +Obsoletes: 1036 C. Lindsey +Category: Standards Track November 2009 + + + Netnews Architecture and Protocols + +Abstract + + This document defines the architecture of Netnews systems and + specifies the correct manipulation and interpretation of Netnews + articles by software that originates, distributes, stores, and + displays them. It also specifies the requirements that must be met + by any protocol used to transport and serve Netnews articles. + +Status of This Memo + + This document specifies an Internet standards track protocol for the + Internet community, and requests discussion and suggestions for + improvements. Please refer to the current edition of the "Internet + Official Protocol Standards" (STD 1) for the standardization state + and status of this protocol. Distribution of this memo is unlimited. + +Copyright Notice + + Copyright (c) 2009 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (http://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the BSD License. + +Table of Contents + + 1. Introduction ....................................................3 + 1.1. Basic Concepts .............................................3 + 1.2. Scope ......................................................3 + 1.3. Requirements Notation ......................................3 + 1.4. Syntax Notation ............................................3 + 1.5. Definitions ................................................4 + 2. Transport .......................................................5 + + + +Allbery & Lindsey Standards Track [Page 1] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + 3. Duties of Agents ................................................6 + 3.1. General Principles .........................................6 + 3.2. The Path Header Field ......................................7 + 3.2.1. Constructing the Path Header Field ..................8 + 3.2.2. Path Header Field Example ...........................9 + 3.3. Article History and Duplicate Suppression .................10 + 3.4. Duties of a Posting Agent .................................11 + 3.4.1. Proto-Articles .....................................12 + 3.4.2. Multiple Injection of Articles .....................13 + 3.4.3. Followups ..........................................14 + 3.4.4. Construction of the References Header Field ........15 + 3.5. Duties of an Injecting Agent ..............................15 + 3.5.1. Forwarding Messages to a Moderator .................18 + 3.6. Duties of a Relaying Agent ................................19 + 3.7. Duties of a Serving Agent .................................21 + 3.8. Duties of a Reading Agent .................................22 + 3.9. Duties of a Moderator .....................................22 + 3.10. Duties of a Gateway ......................................24 + 3.10.1. Duties of an Outgoing Gateway .....................25 + 3.10.2. Duties of an Incoming Gateway .....................25 + 3.10.3. Original-Sender Header Field ......................27 + 3.10.4. Gateway Example ...................................28 + 4. Media Types ....................................................29 + 4.1. application/news-transmission .............................30 + 4.2. application/news-groupinfo ................................31 + 4.3. application/news-checkgroups ..............................33 + 5. Control Messages ...............................................35 + 5.1. Authentication and Authorization ..........................35 + 5.2. Group Control Messages ....................................36 + 5.2.1. The newgroup Control Message .......................36 + 5.2.1.1. newgroup Control Message Example ..........37 + 5.2.2. The rmgroup Control Message ........................38 + 5.2.3. The checkgroups Control Message ....................38 + 5.3. The cancel Control Message ................................40 + 5.4. The Supersedes Header Field ...............................40 + 5.5. The ihave and sendme Control Messages .....................41 + 5.6. Obsolete Control Messages .................................42 + 6. Security Considerations ........................................42 + 6.1. Compromise of System Integrity ............................42 + 6.2. Denial of Service .........................................44 + 6.3. Leakage ...................................................44 + 7. IANA Considerations ............................................45 + 8. References .....................................................45 + 8.1. Normative References ......................................45 + 8.2. Informative References ....................................46 + Appendix A. Changes to the Existing Protocols ....................47 + Appendix B. Acknowledgements .....................................48 + + + + +Allbery & Lindsey Standards Track [Page 2] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + +1. Introduction + +1.1. Basic Concepts + + "Netnews" is a set of protocols for generating, storing, and + retrieving news "articles" whose format is defined in [RFC5536], and + for exchanging them amongst a readership that is potentially widely + distributed. It is organized around "newsgroups", with the + expectation that each reader will be able to see all articles posted + to each newsgroup in which he participates. These protocols most + commonly use a flooding algorithm that propagates copies throughout a + network of participating servers. Typically, only one copy is stored + per server, and each server makes it available on demand to readers + able to access that server. + + "Usenet" is a particular worldwide, publicly accessible network based + on the Netnews protocols. It is only one such possible network; + there are deployments of the Netnews protocols other than Usenet + (such as ones internal to particular organizations). This document + discusses the more general Netnews architecture and protocols. + +1.2. Scope + + This document defines the architecture of Netnews systems and + specifies the correct manipulation and interpretation of Netnews + articles by software that originates, distributes, stores, and + displays them. It addresses protocol issues that are independent of + transport protocols such as the Network News Transfer Protocol (NNTP) + [RFC3977], and specifies the requirements Netnews places on those + underlying transport protocols. It also specifies the handling of + control messages. + + The format and syntax of Netnews articles are specified in [RFC5536], + which should be read in conjunction with this document. + +1.3. Requirements Notation + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in [RFC2119]. + +1.4. Syntax Notation + + Syntax defined in this document uses the Augmented Backus-Naur Form + (ABNF) notation (including the Core Rules) defined in [RFC5234] and + constructs defined in [RFC5536] and [RFC5322]. + + + + + +Allbery & Lindsey Standards Track [Page 3] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + The ABNF rules defined elsewhere and used in this document are: + + CRLF = <see [RFC5234] Appendix B.1> + DIGIT = <see [RFC5234] Appendix B.1> + HTAB = <see [RFC5234] Appendix B.1> + SP = <see [RFC5234] Appendix B.1> + WSP = <see [RFC5234] Appendix B.1> + VCHAR = <see [RFC5234] Appendix B.1> + + argument = <see [RFC5536] Section 3.2.3> + article-locator = <see [RFC5536] Section 3.2.14> + component = <see [RFC5536] Section 3.1.4> + control-command = <see [RFC5536] Section 3.2.3> + diag-keyword = <see [RFC5536] Section 3.1.5> + diag-match = <see [RFC5536] Section 3.1.5> + diag-other = <see [RFC5536] Section 3.1.5> + dist-name = <see [RFC5536] Section 3.2.4> + msg-id = <see [RFC5536] Section 3.1.3> + newsgroup-name = <see [RFC5536] Section 3.1.4> + path-diagnostic = <see [RFC5536] Section 3.1.5> + path-identity = <see [RFC5536] Section 3.1.5> + path-nodot = <see [RFC5536] Section 3.1.5> + tail-entry = <see [RFC5536] Section 3.1.5> + verb = <see [RFC5536] Section 3.2.3> + + display-name = <see [RFC5322] Section 3.4> + local-part = <see [RFC5322] Section 3.4.1> + mailbox = <see [RFC5322] Section 3.4> + +1.5. Definitions + + Any term used in this document that is defined in Section 1.5 of + [RFC5536] is used with the definition given there. In addition, the + following terms will be used: + + A "hierarchy" is the set of all newsgroups whose names share a first + <component> (as defined in Section 3.1.4 of [RFC5536]). A "sub- + hierarchy" is the set of all newsgroups whose names share several + initial components. + + A "news server" is further distinguished into the roles of "injecting + agent", "relaying agent", and "serving agent". An "injecting agent" + accepts a proto-article with the goal of distributing it to relaying + and serving agents and hence to readers. A "relaying agent" accepts + articles from other relaying agents or injecting agents and + distributes them to other relaying agents or serving agents. A + "serving agent" receives an article from a relaying agent or + injecting agent and makes it available to readers. + + + +Allbery & Lindsey Standards Track [Page 4] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + A "user agent" is further distinguished into the roles of "posting + agent" and "reading agent". A "posting agent" is software that + assists in the preparation of a proto-article and then passes it to + an injecting agent. A "reading agent" is software that retrieves + articles from a serving agent for presentation to a reader. + + "Injecting" an article is the processing of a proto-article by an + injecting agent. Normally, this action is done once and only once + for a given article. "Multiple injection" is passing the same + article to multiple injecting agents, either serially or in parallel, + by one or several posting agents. + + A "gateway" is software that receives news articles and converts them + to messages of some other kind (such as [RFC5322] mail messages), + receives messages of some other kind and converts them to news + articles, or conveys articles between two separate Netnews networks. + +2. Transport + + The exact means used to transmit articles from one agent to another + is not specified. NNTP [RFC3977] is the most common transport + mechanism for Netnews networks. Other methods in use include the + Unix-to-Unix Copy Protocol [UUCP] (extensively used in the early days + of Usenet) and physically delivered magnetic and optical media. Any + mechanism may be used in conjunction with this protocol provided that + it can meet the requirements specified here. + + Transports for Netnews articles MUST treat news articles as + uninterpreted sequences of octets, excluding the values %d00 (which + may not occur in Netnews articles), %d13, and %d10 (which MUST only + appear in Netnews articles as a pair in that order and which, + together, denote a line separator). These octets are the US-ASCII + [ASCII] characters NUL, CR, and LF respectively. + + NOTE: This corresponds to the range of octets permitted in MIME + 8bit data [RFC2045]. Transports for Netnews are not required to + support transmission of MIME binary data. + + In particular, transports MUST convey all header fields unmodified + (including header fields within message/rfc822 objects in article + bodies), even if they contain octets in the range of 128 to 255. + Furthermore, transports for relaying and serving agents MUST, and + transports for other agents SHOULD, convey lines even if they exceed + 998 characters in length, especially in article bodies. (This + requirement is stricter than MIME 8bit data.) These requirements + include the transport paths between posting agents, injecting agents, + serving agents, and reading agents. + + + + +Allbery & Lindsey Standards Track [Page 5] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + +3. Duties of Agents + + The following section specifies the duties of the agents involved in + the creation, relaying, and serving of Netnews articles. This + protocol is described by following the life of a typical Usenet + article: it is prepared by a posting agent, given to an injecting + agent, transferred through one or more relaying agents, accepted by a + serving agent, and finally retrieved by a reading agent. Articles + submitted to moderated groups go through an additional process, which + is described separately (see Section 3.5.1 and Step 7 of + Section 3.5). Finally, the additional duties and requirements of a + gateway are discussed. + + At each step, each agent has a set of checks and transformations of + the article that it is required to perform. These are described as + sequences of steps to be followed, but it should be understood that + it is the effect of these sequences that is important, and + implementations may use any method that produces the same effect. + + Many news servers combine the functions of injecting agent, relaying + agent, and serving agent in a single software package. For the + purposes of this specification, such combined agents should + conceptually be treated as an injecting agent that sends articles to + a serving agent and, optionally, to a relaying agent. The + requirements of all three agents MUST still be met when the news + server is performing the functions of those agents. + + On news servers that accept them, control messages may have + additional effects than those described below. Those effects are + described in Section 5. + +3.1. General Principles + + There are two important principles that news implementors and + administrators need to keep in mind. The first is the well-known + Internet Robustness Principle: + + Be liberal in what you accept, and conservative in what you send. + + As applied to Netnews, this primarily means that unwanted or non- + compliant articles SHOULD be rejected as early as possible, but once + they are in general circulation, relaying and serving agents may wish + to accept them where possible rather than lose information. Posting + agents and injecting agents SHOULD therefore be maximally strict in + their application of both this protocol and [RFC5536], and reading + agents SHOULD be robust in the presence of violations of the Netnews + article format where possible. + + + + +Allbery & Lindsey Standards Track [Page 6] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + In the case of Netnews, there is an even more important principle, + derived from a much older code of practice, the Hippocratic Oath (we + may thus call this the Hippocratic Principle): + + First, do no harm. + + It is vital to realize that decisions that might be merely suboptimal + in a smaller context can become devastating mistakes when amplified + by the actions of thousands of hosts within a few minutes. + + No Netnews agent is ever required to accept any article. It is + common for injecting, relaying, and serving agents to reject well- + formed articles for reasons of local policy (such as not wishing to + carry a particular newsgroup or attempting to filter out unwanted + articles). This document specifies how articles are to be treated if + they are accepted and specifies some cases where they must be + rejected, but an agent MAY always reject any article for other + reasons than those stated here. + + A primary goal of the Netnews protocol is to ensure that all readers + receiving a particular article (as uniquely identified by the content + of its Message-ID header field) see the identical article, apart from + allowable divergence in trace headers and local metadata. + Accordingly, agents (other than moderators) MUST NOT modify articles + in ways other than described here. Unacceptable articles MUST be + rejected rather than corrected. + +3.2. The Path Header Field + + All news server components (injecting agents, relaying agents, and + serving agents) MUST identify themselves, when processing an article, + by prepending their <path-identity> (defined in Section 3.1.5 of + [RFC5536]) to the Path header field. Injecting agents MUST also use + the same identity in Injection-Info header fields that they add, and + serving and relaying agents SHOULD use the same identity in any Xref + header fields they add. + + The <path-identity> used by an agent may be chosen via one of the + following methods (in decreasing order of preference): + + 1. The fully qualified domain name (FQDN) of the system on which the + agent is running. + + 2. A fully qualified domain name (FQDN) within a domain affiliated + with the administrators of the agent and guaranteed to be unique + by the administrators of that domain. For example, the + + + + + +Allbery & Lindsey Standards Track [Page 7] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + uniqueness of server.example.org could be guaranteed by the + administrator of example.org even if there is no DNS record for + server.example.org itself. + + 3. Some other (arbitrary) name in the form of a <path-nodot>, + believed to be unique and registered at least with all the other + news servers to which that relaying agent or injecting agent + sends articles. This option SHOULD NOT be used unless the + earlier options are unavailable or unless the name is of + longstanding usage. + + Some existing implementations treat <path-identity> as case- + sensitive, some as case-insensitive. The <path-identity> therefore + SHOULD be all lowercase and implementations SHOULD compare identities + case-insensitively. + +3.2.1. Constructing the Path Header Field + + If a relaying or serving agent receives an article from an injecting + or serving agent that is part of the same news server, it MAY leave + the Path header field of the article unchanged. Otherwise, every + injecting, relaying, or serving agent that accepts an article MUST + update the Path header field as follows. Note that the Path header + field content is constructed from right to left by prepending + elements. + + 1. The agent MUST prepend "!" to the Path header field content. + + 2. An injecting agent SHOULD prepend the <path-diagnostic> + "!.POSTED", optionally followed by "." and the FQDN or IP address + of the source, to the Path header field content. + + 3. A relaying or serving agent SHOULD prepend a <path-diagnostic> to + the Path header field content, where the <path-diagnostic> is + chosen as follows: + + * If the expected <path-identity> of the source of the article + matches the leftmost <path-identity> of the Path header + field's content, use "!" (<diag-match>), resulting in two + consecutive "!"s. + + * If the expected <path-identity> of the source of the article + does not match, use "!.MISMATCH." followed by the expected + <path-identity> of the source or its IP address. + + * If the relaying or serving agent is not willing or able to + check the <path-identity>, use "!.SEEN." followed by the FQDN, + IP address, or expected <path-identity> of the source. + + + +Allbery & Lindsey Standards Track [Page 8] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + The "expected <path-identity> of the source of the article" is a + <path-identity> for the injecting or relaying agent that passed + the article to this relaying or serving agent, determined by + properties of the connection via which the article was received + (for example, an authentication identity or a peer IP address). + Be aware that [RFC1036] did not include <path-diagnostic>. + Implementations that predate this specification will add only + single "!" characters between <path-identity> strings. + + 4. The agent MAY then prepend to the Path header field content "!" + or "!!" followed by an additional <path-identity> for itself + other than its primary one. Using "!!", and thereby adding a + <diag-match> since the <path-identity> clearly is verified, is + RECOMMENDED. This step may be repeated any number of times. + This is permitted for agents that have multiple <path-identity>s + (such as during a transition from one to another). Each of these + <path-identity>s MUST meet the requirements set out in + Section 3.2. + + 5. Finally, the agent MUST prepend its primary <path-identity> to + the Path header field content. The primary <path-identity> is + the <path-identity> it normally advertises to its peers for their + use in generating <path-diagnostic>s as described above. + + Any agent that modifies the Path header field MAY fold it by + inserting FWS (folding white space) immediately after any <path- + identity> or <diag-other> it added (see Section 3.1.5 of [RFC5536] + for allowable locations for FWS). + +3.2.2. Path Header Field Example + + Here is an example of a Path header field created by following the + rules for injecting and relaying agents. + + Path: foo.isp.example!.SEEN.isp.example!foo-news + !.MISMATCH.2001:DB8:0:0:8:800:200C:417A!bar.isp.example + !!old.site.example!barbaz!!baz.isp.example + !.POSTED.dialup123.baz.isp.example!not-for-mail + + This article was injected by baz.isp.example as indicated by the + <diag-keyword> "POSTED". The injector has recorded that it received + the article from dialup123.baz.isp.example. "not-for-mail" is a + common <tail-entry>. + + + + + + + + +Allbery & Lindsey Standards Track [Page 9] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + The article was relayed to the relaying agent known, at least to + old.site.example, as "barbaz". That relaying agent confirmed to its + satisfaction that "baz.isp.example" was an expected <path-identity> + for the source of the article and therefore used <diag-match> ("!") + for its <path-diagnostic>. + + barbaz relayed it to old.site.example, which does not support <diag- + keyword> and therefore used the old "!" delimiter. This indicates + that the identity of "barbaz" was not verified and may have been + forged. + + old.site.example relayed it to a news server using the <path- + identity> of bar.isp.example and claiming (by using the "!" <path- + diagnostic>) to have verified that it came from old.site.example. + + bar.isp.example relayed it to foo-news, which, not being convinced + that it truly came from bar.isp.example, inserted the <diag-keyword> + "MISMATCH" and then stated that it received the article from the IPv6 + address [2001:DB8:0:0:8:800:200C:417A]. (This is not to say that + bar.isp.example was not a correct <path-identity> for that source but + simply that the identity did not match the expectations of foo-news.) + + foo-news then passed the article to foo.isp.example, which declined + to validate its <path-identity> and instead appended the <diag- + keyword> "SEEN" to indicate it knows the source of the article as + isp.example. This may be either an expected <path-identity> or the + FQDN of the system from which it received the article. Presumably, + foo.isp.example is a serving agent that then delivered the article to + a reading agent. + + baz.isp.example, bar.isp.example, and foo-news folded the Path header + field. + +3.3. Article History and Duplicate Suppression + + Netnews normally uses a flood-fill algorithm for propagation of + articles in which each news server offers the articles it accepts to + multiple peers, and each news server may be offered the same article + from multiple other news servers. Accordingly, duplicate suppression + is key; if a news server accepted every article it was offered, it + may needlessly accept (and then potentially retransmit) dozens of + copies of every article. + + Relaying and serving agents therefore MUST keep a record of articles + they have already seen and use that record to reject additional + offers of the same article. This record is called the "history" file + or database. + + + + +Allbery & Lindsey Standards Track [Page 10] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + Each article is uniquely identified by its message identifier, so a + relaying or serving agent could satisfy this requirement by storing a + record of every message identifier that agent has ever seen. Such a + history database would grow without bound, however, so it is common + and permitted to optimize based on the Injection-Date or Date header + field of an article as follows. (In the following discussion, the + "date" of an article is defined to be the date represented by its + Injection-Date header field, if present; otherwise, by its Date + header field.) + + o Agents MAY select a cutoff interval and reject any article with a + date farther in the past than that cutoff interval. If this + interval is shorter than the time it takes for an article to + propagate through the network, the agent might reject an article + it had not yet seen, so it ought not to be aggressively short. + For Usenet, for example, a cutoff interval of no less than seven + days is conventional. + + o Agents that enforce such a cutoff MAY then drop records of + articles that had dates older than the cutoff from their history + databases. If such an article were offered to the agent again, it + would be rejected due to the cutoff date, so the history record is + no longer required to suppress the duplicate. + + o Alternatively, agents MAY drop history records according to the + date when the article was first seen by that agent rather than the + date of the article. In this case, the history retention interval + MUST be at least 24 hours longer than the cutoff interval to allow + for articles dated in the future. This interval matches the + allowable error in the date of the article (see Section 3.5). + + These are just two implementation strategies for article history, + albeit the most common ones. Relaying and serving agents are not + required to use these strategies, only to meet the requirement of not + accepting an article more than once. However, these strategies are + safe and widely deployed, and implementors are encouraged to use one + of them, especially if they do not have extensive experience with + Netnews and the subtle effects of its flood-fill algorithm. + +3.4. Duties of a Posting Agent + + A posting agent is the component of a user agent that assists a + poster in creating a valid proto-article and forwarding it to an + injecting agent. + + + + + + + +Allbery & Lindsey Standards Track [Page 11] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + Posting agents SHOULD ensure that proto-articles they create are + valid according to [RFC5536] and any other applicable policies. They + MUST NOT create any Injection-Info header field; this header field + may only be added by the injecting agent. + + If the proto-article already contains both Message-ID and Date header + fields, posting agents MAY add an Injection-Date header field to that + proto-article immediately before passing that proto-article to an + injection agent. They SHOULD do so if the Date header field + (representing the composition time of the proto-article) is more than + a day in the past at the time of injection. They MUST do so if the + proto-article is being submitted to more than one injecting agent; + see Section 3.4.2. + + The Injection-Date header field is new in this revision of the + Netnews protocol and is designed to allow the Date header field to + hold the composition date (as recommended in Section 3.6.1 of + [RFC5322]), even if the proto-article is not to be injected for some + time after its composition. However, note that all implementations + predating this specification ignore the Injection-Date header field + and use the Date header field in its stead for rejecting articles + older than their cutoff (see Section 3.3), and injecting agents + predating this specification do not add an Injection-Date header. + Articles with a Date header field substantially in the past will + still be rejected by implementations predating this specification, + regardless of the Injection-Date header field, and hence may suffer + poorer propagation. + + Contrary to [RFC5322], which implies that the mailbox or mailboxes in + the From header field should be that of the poster or posters, a + poster who does not, for whatever reason, wish to use his own mailbox + MAY use any mailbox ending in the top-level domain ".invalid" + [RFC2606]. + + Posting agents meant for use by ordinary posters SHOULD reject any + attempt to post an article that cancels or supersedes (via the + Supersedes header field) another article of which the poster is not + the author or sender. + +3.4.1. Proto-Articles + + A proto-article is an article in the format used by a posting agent + when offering that article to an injecting agent. It may omit + certain header fields that can be better supplied by the injecting + agent and will not contain header fields that are added by the + injecting agent. A proto-article is only for transmission to an + injecting agent and SHOULD NOT be transmitted to any other agent. + + + + +Allbery & Lindsey Standards Track [Page 12] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + A proto-article has the same format as a normal article except that + the Injection-Info and Xref header fields MUST NOT be present, the + Path header field SHOULD NOT contain a "POSTED" <diag-keyword>, and + any of the following mandatory header fields MAY be omitted: + Message-ID, Date, and Path. In all other respects, a proto-article + MUST be a valid Netnews article. In particular, the header fields + that may be omitted MUST NOT be present with invalid content. + + If a posting agent intends to offer the same proto-article to + multiple injecting agents, the header fields Message-ID, Date, and + Injection-Date MUST be present and identical in all copies of the + proto-article. See Section 3.4.2. + +3.4.2. Multiple Injection of Articles + + Under some circumstances (for example, when posting to multiple, + supposedly disjoint, networks, when using injecting agents with + spotty connectivity, or when desiring additional redundancy), a + posting agent may wish to offer the same article to multiple + injecting agents. In this unusual case, the goal is not to create + multiple independent articles but rather to inject the same article + at multiple points and let the normal duplicate suppression facility + of Netnews (see Section 3.3) ensure that any given agent accepts the + article only once, even if supposedly disjoint networks have + unexpected links. + + Whenever possible, multiple injection SHOULD be done by offering the + same proto-article to multiple injecting agents. The posting agent + MUST supply the Message-ID, Date, and Injection-Date header fields, + and the proto-article as offered to each injecting agent MUST be + identical. + + In some cases, offering the same proto-article to all injecting + agents may not be possible (such as when gatewaying, after injection, + articles found on one Netnews network to another supposedly + unconnected one). In this case, the posting agent MUST remove any + Xref header field and rename or remove any Injection-Info, Path, and + other trace header fields before passing it to another injecting + agent. (This converts the article back into a proto-article.) It + MUST retain unmodified the Message-ID, Date, and Injection-Date + header fields. It MUST NOT add an Injection-Date header field if it + is missing from the existing article. + + NOTE: Multiple injection inherently risks duplicating articles. + Multiple injection after injection, by converting an article back + to a proto-article and injecting it again, additionally risks + loops, loss of trace information, unintended repeat injection into + the same network, and other problems. It should be done with care + + + +Allbery & Lindsey Standards Track [Page 13] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + and only when there is no alternative. The requirement to retain + Message-ID, Date, and Injection-Date header fields minimizes the + possibility of a loop and ensures that the newly injected article + is not treated as a new, separate article. + + Multiple injection of an article that lists one or more moderated + newsgroups in its Newsgroups header field SHOULD only be done by a + moderator and MUST only be done after the proto-article has been + approved for all moderated groups to which it is to be posted and + after an Approved header field has been added (see Section 3.9). + Multiple injection of an unapproved article intended for moderated + newsgroups will normally only result in the moderator receiving + multiple copies, and if the newsgroup status is not consistent across + all injecting agents, may result in duplication of the article or + other problems. + +3.4.3. Followups + + A followup is an article that contains a response to the contents of + an earlier article, its precursor. In addition to its normal duties, + a posting agent preparing a followup is also subject to the following + requirements. Wherever in the following it is stated that, by + default, a header field is said to be inherited from one of those + header fields in the precursor, it means that its initial content is + to be a copy of the content of that precursor header field (with + changes in folding permitted). However, posters MAY then override + that default before posting. + + Despite the historic practice of some posting agents, the Keywords + header field SHOULD NOT be inherited by default from the precursor + article. + + 1. If the Followup-To header field of the precursor article consists + of "poster", the followup MUST NOT be posted by default but, by + default, is to be emailed to the address given in the precursor's + Reply-To or From header field following the rules for an email + reply [RFC5322]. This action MAY be overridden by the poster, in + which case the posting agent should continue as if the + Followup-To header field in the precursor did not exist. + + 2. The Newsgroups header field SHOULD, by default, be inherited from + the precursor's Followup-To header field if present; otherwise, + it is inherited from the precursor's Newsgroups header field. + + + + + + + + +Allbery & Lindsey Standards Track [Page 14] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + 3. The Subject header field SHOULD, by default, be inherited from + that of the precursor. The case-sensitive string "Re: " + (including the space after the colon) MAY be prepended to the + content of its Subject header field unless it already begins with + that string. + + NOTE: Prepending "Re: " serves no protocol function and hence + is not required, but it is widely expected and not doing so + would be surprising. + + 4. The Distribution header field SHOULD, by default, be inherited + from the precursor's Distribution header field, if present. + + 5. The followup MUST have a References header field referring to its + precursor, constructed in accordance with Section 3.4.4. + +3.4.4. Construction of the References Header Field + + The following procedure is to be used whenever some previous article + (the "parent") is to be referred to in the References header field of + a new article, whether because the new article is a followup and the + parent is its precursor or for some other reason. + + The content of the new article's References header field MUST be + formed from the content of the parent's References header field if + present, followed by the content of the Message-ID header field of + the parent. If the parent had a References header, FWS as defined in + [RFC5536] MUST be added between its content and the Message-ID header + field content. + + If the resulting References header field would, after unfolding, + exceed 998 characters in length (including its field name but not the + final CRLF), it MUST be trimmed (and otherwise MAY be trimmed). + Trimming means removing any number of message identifiers from its + content, except that the first message identifier and the last two + MUST NOT be removed. + + An essential property of the References header field, guaranteed by + the above procedure and REQUIRED to be maintained by any extensions + to this protocol, is that an article MUST NOT precede one of its + parents. + +3.5. Duties of an Injecting Agent + + An injecting agent takes a proto-article from a posting agent and + either forwards it to a moderator or passes it to a relaying or + serving agent or agents. An injecting agent bears the primary + responsibility for ensuring that any article it injects conforms with + + + +Allbery & Lindsey Standards Track [Page 15] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + the rules of the Netnews standards. The administrator of an + injecting agent is also expected to bear some responsibility towards + the rest of the Netnews network to which it is connected for the + articles the injecting agent accepts. + + Injecting agents, when rejecting articles, are encouraged to + communicate the reason for rejection to the posting agent by using + whatever facility is provided by the underlying transport. The + injecting agent is in a unique position to communicate the reason for + rejection; relaying agents and serving agents normally have to reject + messages silently. The injecting agent therefore bears much of the + burden of diagnosing broken posting agents or communicating policy + violations to posters. + + An injecting agent MUST have available a list (possibly empty) of + moderated groups for which it accepts articles and the corresponding + submission addresses. It SHOULD have available a list of valid + newsgroups to catch articles not posted to a valid newsgroup and + therefore likely to be silently discarded by relaying and serving + agents. Usually, an injecting agent is deployed in conjunction with + a serving agent and maintains these lists based on control messages + received by the serving agent. + + An injecting agent processes proto-articles as follows: + + 1. It SHOULD verify that the article is from a trusted source (for + example, by relying on the authorization capability of the + underlying transport used to talk to the posting agent). + + 2. It MUST reject any proto-article that does not have the proper + mandatory header fields for a proto-article, that has Injection- + Info or Xref header fields, that has a Path header field + containing the "POSTED" <diag-keyword>, or that is not + syntactically valid as defined by [RFC5536]. It SHOULD reject + any proto-article that contains a header field deprecated for + Netnews (see, for example, [RFC3798]). It MAY reject any proto- + article that contains trace header fields (e.g., NNTP-Posting- + Host) indicating that it was already injected by an injecting + agent that did not add Injection-Info or Injection-Date. + + 3. It SHOULD reject any article whose Injection-Date or Date header + field is more than 24 hours into the future (and MAY use a + margin less than 24 hours). It SHOULD reject any article whose + Injection-Date header field is too far in the past (older than + the cutoff interval of a relaying agent that the injecting agent + is using, for example). It SHOULD similarly reject any article + whose Date header field is too far in the past, since not all + news servers support Injection-Date and only the injecting agent + + + +Allbery & Lindsey Standards Track [Page 16] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + can provide a useful error message to the posting agent. In + either case, this interval SHOULD NOT be any shorter than 72 + hours into the past. + + 4. It SHOULD reject any proto-article whose Newsgroups header field + does not contain at least one <newsgroup-name> for a valid + group, or that contains a <newsgroup-name> reserved for specific + purposes by Section 3.1.4 of [RFC5536] unless that specific + purpose or local agreement applies to the proto-article being + processed. Crossposting to unknown newsgroups is not precluded + provided that at least one of the newsgroups in the Newsgroups + header is valid. + + 5. The Message-ID and Date header fields with appropriate contents + MUST be added when not present in the proto-article. + + 6. The injecting agent MUST NOT alter the body of the article in + any way (including any change of Content-Transfer-Encoding). It + MAY add other header fields not already provided by the poster, + but injecting agents are encouraged to use the Injection-Info + header for such information and to minimize the addition of + other headers. It SHOULD NOT alter, delete, or reorder any + existing header field except the Path header field. It MUST NOT + alter or delete any existing Message-ID header field. + + 7. If the Newsgroups header contains one or more moderated groups + and the proto-article does not contain an Approved header field, + the injecting agent MUST either forward it to a moderator as + specified in Section 3.5.1 or, if that is not possible, reject + it. This forwarding MUST be done after adding the Message-ID + and Date headers if required, and before adding the Injection- + Info and Injection-Date headers. + + 8. Otherwise, a Path header field with a <tail-entry> MUST be added + if not already present. + + 9. The injecting agent MUST then update the Path header field as + described in Section 3.2.1. + + 10. An Injection-Info header field SHOULD be added that identifies + the source of the article and possibly other trace information + as described in Section 3.2.8 of [RFC5536]. + + 11. If the proto-article already had an Injection-Date header field, + it MUST NOT be modified or replaced. If the proto-article had + both a Message-ID header field and a Date header field, an + Injection-Date header field MUST NOT be added, since the proto- + article may have been multiply injected by a posting agent that + + + +Allbery & Lindsey Standards Track [Page 17] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + predates this standard. Otherwise, the injecting agent MUST add + an Injection-Date header field containing the current date and + time. + + 12. Finally, the injecting agent forwards the article to one or more + relaying agents, and the injection process is complete. + +3.5.1. Forwarding Messages to a Moderator + + An injecting agent MUST forward the proto-article to the moderator of + the leftmost moderated group listed in the Newsgroups header field, + customarily via email. There are two standard ways in which it may + do this: + + 1. The complete proto-article is encapsulated, header fields and + all, within the email. This SHOULD be done by creating an email + message with a Content-Type of application/news-transmission with + the usage parameter set to "moderate". The body SHOULD NOT + contain any content other than the message. This method has the + advantage of removing any possible conflict between Netnews and + email header fields and any changes to those fields during + transport through email. + + 2. The proto-article is sent as an email with the addition of any + header fields required for an email as defined in [RFC5322], and + possibly with the addition of other header fields conventional in + email, such as To and Received. The existing Message-ID header + field SHOULD be retained. + + Although both of these methods have been used in the past and the + first has clear technical advantages, the second is in more common + use and many moderators are not prepared to deal with messages in the + first format. Accordingly, the first method SHOULD NOT be used + unless the moderator to which it is being forwarded is known to be + able to handle this method. + + NOTE: Deriving the email address of the moderator of a group is + outside the scope of this document. It is worth mentioning, + however, that a common method is to use a forwarding service that + handles submissions for many moderated groups. For maximum + compatibility with existing news servers, such forwarding services + generally form the submission address for a moderated group by + replacing each "." in the <newsgroup-name> with "-" and then using + that value as the <local-part> of a <mailbox> formed by appending + a set domain. + + + + + + +Allbery & Lindsey Standards Track [Page 18] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + Forwarding proto-articles to moderators via email is the most general + method and the most common in large Netnews networks such as Usenet, + but any means of forwarding the article that preserves it without + injecting it MAY be used. For example, if the injecting agent has + access to a database used by the moderator to store proto-articles + awaiting processing, it may place the proto-article directly into + that database. Such methods may be more appropriate for smaller + Netnews networks. + +3.6. Duties of a Relaying Agent + + A relaying agent accepts injected articles from injecting and other + relaying agents and passes them on to relaying or serving agents. To + avoid bypass of injecting agent policies and forgery of Path and + Injection-Info headers, relaying agents SHOULD accept articles only + from trusted agents. + + An article SHOULD NOT be relayed unless the sending agent has been + configured to supply, and the receiving agent to receive, at least + one of the <newsgroup-name>s in its Newsgroups header field and at + least one of the <dist-name>s in its Distribution header field (if + present). Exceptionally, control messages creating or removing + newsgroups (newgroup or rmgroup control messages, for example) SHOULD + be relayed if the affected group appears in its Newsgroups header + field and both the sending and receiving relaying agents are + configured to relay a newsgroup of that name (whether or not such a + newsgroup exists). + + In order to avoid unnecessary relaying attempts, an article SHOULD + NOT be relayed if the <path-identity> of the receiving agent (or some + known alias thereof) appears as a <path-identity> (excluding within + the <tail-entry> or following a "POSTED" <diag-keyword>) in its Path + header field. + + A relaying agent processes an article as follows: + + 1. It MUST reject any article without a Newsgroups header field or + Message-ID header field, or without either an Injection-Date or + Date header field. + + 2. It MUST examine the Injection-Date header field or, if absent, + the Date header field, and reject the article if that date is + more than 24 hours into the future. It MAY reject articles with + dates in the future with a smaller margin than 24 hours. + + + + + + + +Allbery & Lindsey Standards Track [Page 19] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + 3. It MUST reject any article that has already been accepted. If it + implements one of the mechanisms described in Section 3.3, this + means that it MUST reject any article whose date falls outside + the cutoff interval since it won't know whether or not such + articles had been accepted previously. + + 4. It SHOULD reject any article that does not include all the + mandatory header fields. It MAY reject any article that contains + header fields that do not have valid contents. + + 5. It SHOULD reject any article that matches an already-received + cancel control message or the contents of the Supersedes header + field of an accepted article, provided that the relaying agent + has chosen (on the basis of local site policy) to honor that + cancel control message or Supersedes header field. + + 6. It MAY reject any article without an Approved header field posted + to a newsgroup known to be moderated. This practice is strongly + encouraged, but the information necessary to do so is not + required to be maintained by a relaying agent. + + 7. It MUST update the Path header field as described in + Section 3.2.1. + + 8. It MAY delete any Xref header field already present. It MAY add + a new Xref header field for its own use (but recall that + [RFC5536] permits at most one such header field). + + 9. Finally, it passes the article on to other relaying and serving + agents to which it is configured to send articles. + + Relaying agents SHOULD, where possible in the underlying transport, + inform the agent that passed the article to the relaying agent if the + article was rejected. Relaying agents MUST NOT inform any other + external entity of the rejection of an article unless that external + entity has explicitly requested that it be informed of such errors. + + Relaying agents MUST NOT alter, delete, or rearrange any part of an + article except for the Path and Xref header fields. They MUST NOT + modify the body of articles in any way. If an article is not + acceptable as is, the article MUST be rejected rather than modified. + + + + + + + + + + +Allbery & Lindsey Standards Track [Page 20] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + +3.7. Duties of a Serving Agent + + A serving agent accepts articles from a relaying agent or injecting + agent, stores them, and makes them available to reading agents. + Articles are normally indexed by newsgroup and <article-locator> + (Section 3.2.14 of [RFC5536]), usually in the form of a decimal + number. + + If the serving agent stores articles by newsgroup, control messages + SHOULD NOT be stored in the newsgroups listed in the control + message's Newsgroups header field. Instead, they SHOULD be stored in + a newsgroup in the hierarchy "control", which is reserved for this + purpose. Conventionally, control messages are stored in newsgroups + named for the type of control message (such as "control.cancel" for + cancel control messages). + + A serving agent MUST have available a list (possibly empty) of + moderated groups for which it accepts articles so that it can reject + unapproved articles posted to moderated groups. Frequently, a + serving agent is deployed in combination with an injecting agent and + can use the same list as the injecting agent. + + A serving agent processes articles as follows: + + 1. It MUST reject any article that does not include all the + mandatory header fields or any article that contains header + fields that do not have valid contents. + + 2. It MUST examine the Injection-Date header field or, if absent, + the Date header field, and reject the article if that date is + more than 24 hours into the future. It MAY reject articles with + dates in the future with a smaller margin than 24 hours. + + 3. It MUST reject any article that has already been accepted. If it + implements one of the mechanisms described in Section 3.3, this + means that it MUST reject any article whose date falls outside + the cutoff interval since it won't know whether or not such + articles had been accepted previously. + + 4. It SHOULD reject any article that matches an already-received and + honored cancel message or Supersedes header field, following the + same rules as a relaying agent (Section 3.6). + + 5. It MUST reject any article without an Approved header field + posted to any newsgroup listed as moderated. + + 6. It MUST update the Path header field as described in + Section 3.2.1. + + + +Allbery & Lindsey Standards Track [Page 21] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + 7. It MUST remove any Xref header field from each article (except + when specially configured to preserve the <article-locator>s set + by the sending site). It then MAY (and usually will) add a new + Xref header field (but recall that [RFC5536] permits at most one + such header field). + + 8. Finally, it stores the article and makes it available for reading + agents. + + Serving agents MUST NOT create new newsgroups simply because an + unrecognized <newsgroup-name> occurs in a Newsgroups header field. + Newsgroups are normally created via control messages (Section 5.2.1). + + Serving agents MUST NOT alter, delete, or rearrange any part of an + article except for the Path and Xref header fields. They MUST NOT + modify the body of the articles in any way. If an article is not + acceptable as is, the article MUST be rejected rather than modified. + +3.8. Duties of a Reading Agent + + Since a reading agent is only a passive participant in a Netnews + network, there are no specific protocol requirements placed on it. + See [USEAGE] for best-practice recommendations. + +3.9. Duties of a Moderator + + A moderator receives news articles, customarily by email, decides + whether to approve them and, if so, either passes them to an + injecting agent or forwards them to further moderators. + + Articles are normally received by the moderator in email, either + encapsulated as an object of Content-Type application/ + news-transmission (or possibly encapsulated but without an explicit + Content-Type header field) or else directly as an email already + containing all the header fields appropriate for a Netnews article + (see Section 3.5.1). Moderators who may receive articles via email + SHOULD be prepared to accept articles in either format. + + There are no protocol restrictions on what criteria are used for + accepting or rejecting messages or on what modifications a moderator + may make to a message (both header fields and body) before injecting + it. Recommended best practice, however, is to make the minimal + required changes. Moderators need to be aware that modifications + made to articles may invalidate signatures created by the poster or + previous moderators. See [USEAGE] for further best-practice + recommendations. + + + + + +Allbery & Lindsey Standards Track [Page 22] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + Moderators process articles as follows: + + 1. They decide whether to approve or reject a proto-article and, if + approved, prepare the proto-article for injection. If the proto- + article was received as an unencapsulated email message, this + will require converting it back to a valid Netnews proto-article. + If the article is rejected, it is normally rejected for all + newsgroups to which it was posted and nothing further is done. + If it is approved, the moderator proceeds with the following + steps. + + 2. If the Newsgroups header field contains further moderated + newsgroups for which approval has not already been given, they + may either reach some agreement with the other moderators on the + disposition of the article or, more generally, add an indication + (identifying both the moderator and the name of the newsgroup) + that they approve the article and then forward it to the + moderator of the leftmost unapproved newsgroup. This forwarding + SHOULD be done following the procedure in Section 3.5.1. It MAY + be done by rotating the <newsgroup-name>s in the Newsgroups + header field so that the leftmost unapproved newsgroup is the + leftmost moderated newsgroup in that field and then posting it, + letting the injecting agent do the forwarding. However, when + using this mechanism, they MUST first ensure that the article + contains no Approved header field. + + 3. If the Newsgroups header field contains no further unapproved + moderated groups, they add an Approved header field (see Section + 3.2.1 of [RFC5536]) identifying the moderator and, insofar as is + possible, all the other moderators who have approved the article. + The moderator who takes this step assumes responsibility for + ensuring that the article was approved by the moderators of all + moderated newsgroups to which it was posted. + + 4. Moderators are encouraged to retain the Message-ID header field + unless it is invalid or the article was significantly changed + from its original form. Moderators are also encouraged to retain + the Date header field unless it appears to be stale (72 hours or + more in the past) for reasons understood by the moderator (such + as delays in the moderation process), in which case they MAY + substitute the current date. Any Injection-Date, Injection-Info, + or Xref header fields already present MUST be removed. + + 5. Any Path header field MUST either be removed or truncated to only + those entries following its "POSTED" <diag-keyword>, if any. + + 6. The moderator then passes the article to an injecting agent, + having first observed all the duties of a posting agent. + + + +Allbery & Lindsey Standards Track [Page 23] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + +3.10. Duties of a Gateway + + A gateway transforms an article into the native message format of + another medium, or translates the messages of another medium into + news articles, or transforms articles into proto-articles for + injection into a separate Netnews network. Encapsulation of a news + article into a message of MIME type application/news-transmission, or + the subsequent undoing of that encapsulation, is not gatewaying since + it involves no transformation of the article. + + There are two basic types of gateway, the outgoing gateway that + transforms a news article into a different type of message, and the + incoming gateway that transforms a message from another network into + a news proto-article and injects it into a Netnews network. These + are handled separately below. + + Transformation of an article into another medium stands a very high + chance of discarding or interfering with the protection inherent in + the news system against duplicate articles. The most common problem + caused by gateways is loops that repeatedly reinject previously + posted articles. To prevent this, a gateway MUST take precautions + against loops, as detailed below. + + The transformations applied to the message SHOULD be as minimal as + possible while still accomplishing the gatewaying. Every change made + by a gateway potentially breaks a property of one of the media or + loses information, and therefore only those transformations made + necessary by the differences between the media should be applied. + + If bidirectional gatewaying (both an incoming and an outgoing + gateway) is being set up between Netnews and some other medium, the + incoming and outgoing gateways SHOULD be coordinated to avoid + unintended reinjection of gated articles. Circular gatewaying + (gatewaying a message into another medium and then back into Netnews) + SHOULD NOT be done; encapsulation of the article SHOULD be used + instead where this is necessary. + + Safe bidirectional gatewaying between a mailing list and a newsgroup + is far easier if the newsgroup is moderated. Posts to the moderated + group and submissions to the mailing list can then go through a + single point that does the necessary gatewaying and then sends the + message out to both the newsgroup and the mailing list at the same + time, eliminating most of the possibility of loops. Bidirectional + gatewaying between a mailing list and an unmoderated newsgroup, in + contrast, is difficult to do correctly and is far more fragile. + Newsgroups intended to be bidirectionally gated to a mailing list + SHOULD therefore be moderated where possible, even if the moderator + + + + +Allbery & Lindsey Standards Track [Page 24] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + is a simple gateway and injecting agent that correctly handles + crossposting to other moderated groups and otherwise passes all + traffic. + +3.10.1. Duties of an Outgoing Gateway + + From the perspective of Netnews, an outgoing gateway is just a + special type of reading agent. The exact nature of what the outgoing + gateway will need to do to articles depends on the medium to which + the articles are being gated. Because it raises the danger of loops + due to the possibility of one or more corresponding incoming gateways + back from that medium to Netnews, the operation of the outgoing + gateway is subject to additional constraints. + + The following practices are encouraged for all outgoing gateways, + regardless of whether there is known to be a related incoming + gateway, both as precautionary measures and as guidelines to quality + of implementation: + + 1. The message identifier of the news article should be preserved if + at all possible, preferably as or within the corresponding unique + identifier of the other medium. However, if it is not preserved + in this way, then at least it should be preserved as a comment in + the message. This helps greatly with preventing loops. + + 2. The Date and Injection-Date of the news article should also be + preserved if possible, for similar reasons. + + 3. The message should be tagged in some way so as to prevent its + reinjection into Netnews. This may be impossible to do without + knowledge of potential incoming gateways, but it is better to try + to provide some indication even if not successful; at the least, + a human-readable indication that the article should not be gated + back to Netnews can help locate a human problem. + + 4. Netnews control messages should not be gated to another medium + unless they would somehow be meaningful in that medium. + +3.10.2. Duties of an Incoming Gateway + + The incoming gateway has the responsibility of ensuring that all of + the requirements of this protocol are met by the articles that it + forms. In addition to its special duties as a gateway, it bears all + of the duties and responsibilities of a posting agent, and it has the + same responsibility of a relaying agent to reject articles that it + has already gatewayed. + + + + + +Allbery & Lindsey Standards Track [Page 25] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + An incoming gateway MUST NOT gate the same message twice. It may not + be possible to ensure this in the face of mangling or modification of + the message, but at the very least a gateway, when given a copy of a + message that it has already gated and that is identical except for + trace header fields (like Received in Email or Path in Netnews), MUST + NOT gate the message again. An incoming gateway SHOULD take + precautions against having this rule bypassed by modifications of the + message that can be anticipated. + + News articles prepared by gateways MUST be valid news proto-articles + (see Section 3.4.1). This often requires the gateway to synthesize a + conforming article from non-conforming input. The gateway MUST then + pass the article to an injecting agent, not directly to a relaying + agent. + + Incoming gateways MUST NOT pass control messages (articles containing + a Control or Supersedes header field) without removing or renaming + that header field. Gateways MAY, however, generate cancel control + messages for messages they have gatewayed. If a gateway receives a + message that it can determine is a valid equivalent of a cancel + control message in the medium it is gatewaying, it SHOULD discard + that message without gatewaying it, generate a corresponding cancel + control message of its own, and inject that cancel control message. + + NOTE: It is not unheard of for mail-to-news gateways to be used to + post control messages, but encapsulation should be used for these + cases instead. Gateways by their very nature are particularly + prone to loops. Spews of normal articles are bad enough; spews of + control messages with special significance to the news system, + possibly resulting in high processing load or even in emails being + sent for every message received, are catastrophic. It is far + preferable to construct a system specifically for posting control + messages that can do appropriate consistency checks and + authentication of the originator of the message. + + If there is a message identifier that fills a role similar to that of + the Message-ID header field in news, it SHOULD be used in the + formation of the message identifier of the news article, perhaps with + transformations required to meet the uniqueness requirement of + Netnews and with the removal of any comments so as to comply with the + syntax in Section 3.1.3 of [RFC5536]. Such transformations SHOULD be + designed so that two messages with the same identifier generate the + same Message-ID header field. + + NOTE: Message identifiers play a central role in the prevention of + duplicates, and their correct use by gateways will do much to + prevent loops. Netnews does, however, require that message + identifiers be unique, and therefore message identifiers from + + + +Allbery & Lindsey Standards Track [Page 26] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + other media may not be suitable for use without modification. A + balance must be struck by the gateway between preserving + information used to prevent loops and generating unique message + identifiers. + + Exceptionally, if there are multiple incoming gateways for a + particular set of messages, each to a different newsgroup(s), each + one SHOULD generate a message identifier unique to that gateway. + Each incoming gateway nonetheless MUST ensure that it does not gate + the same message twice. + + NOTE: Consider the example of two gateways of a given mailing list + into two separate Usenet newsgroups, both of which preserve the + email message identifier. Each newsgroup may then receive a + portion of the messages (different sites seeing different + portions). In these cases, where there is no one "official" + gateway, some other method of generating message identifiers has + to be used to avoid collisions. It would obviously be preferable + for there to be only one gateway that crossposts, but this may not + be possible to coordinate. + + If no date information is available, the gateway MAY supply a Date + header field with the gateway's current date. If only partial + information is available (such as date but not time), this SHOULD be + fleshed out to a full Date by adding default values rather than by + discarding this information. Only in very exceptional circumstances + should Date information be discarded, as it plays an important role + in preventing reinjection of old messages. + + An incoming gateway MUST add a Sender header field to the news + article it forms by containing the <mailbox> of the administrator of + the gateway. Problems with the gateway may be reported to this + <mailbox>. The <display-name> portion of this <mailbox> SHOULD + indicate that the entity responsible for injection of the message is + a gateway. If the original message already had a Sender header + field, it SHOULD be renamed to Original-Sender so that its contents + can be preserved. See Section 3.10.3 for the specification of that + header field. + +3.10.3. Original-Sender Header Field + + The Original-Sender header field holds the content of a Sender header + field in an original message received by an incoming gateway, + preserving it while the incoming gateway adds its own Sender header + field. The content syntax makes use of syntax defined in [RFC5536] + and [RFC5322]. + + + + + +Allbery & Lindsey Standards Track [Page 27] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + header =/ Original-Sender-header + Original-Sender-header + = "Original-Sender" ":" SP + Original-Sender-content + Original-Sender-content + = mailbox + + The Permanent Message Header Field Repository entry for this header + field is: + + Header field name: Original-Sender + Applicable protocol: Netnews + Status: standard + Author/Change controller: IETF + Specification document(s): RFC 5537 + +3.10.4. Gateway Example + + To illustrate the type of precautions that should be taken against + loops, here is an example of the measures taken by one particular + combination of mail-to-news and news-to-mail gateways designed to + handle bidirectional gatewaying between mailing lists and unmoderated + groups: + + 1. The news-to-mail gateway preserves the message identifier of the + news article in the generated email message. The mail-to-news + gateway likewise preserves the email message identifier, provided + that it is syntactically valid for Netnews. This allows the news + system's built-in suppression of duplicates to serve as the first + line of defense against loops. + + 2. The news-to-mail gateway adds an X-* header field to all messages + it generates. The mail-to-news gateway discards any incoming + messages containing this header field. This is robust against + mailing list managers that replace the message identifier and + against any number of email hops, provided that the other message + header fields are preserved. + + 3. The mail-to-news gateway prepends the host name from which it + received the email message to the content of the Path header + field. The news-to-mail gateway refuses to gateway any message + that contains the list server name in its Path header field + (including in the tail section). This is robust against any + amount of munging of the message header fields by the mailing + list, provided that the email only goes through one hop. + + + + + + +Allbery & Lindsey Standards Track [Page 28] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + 4. The mail-to-news gateway is designed never to generate bounces to + the envelope sender. Instead, articles that are rejected by the + news server (for reasons not warranting silent discarding of the + message) result in a bounce message sent to an errors address + that is known not to forward to any mailing lists. In this way, + they can be handled by the news administrators. + + These precautions have proven effective in practice at preventing + loops for this particular application (bidirectional gatewaying + between mailing lists and locally distributed newsgroups where both + gateways can be designed together). General gatewaying to world-wide + newsgroups poses additional difficulties; one must be very wary of + strange configurations, such as a newsgroup gated to a mailing list + that is in turn gated to a different newsgroup. + +4. Media Types + + This document defines several media types, which have been registered + with IANA as provided for in [RFC4288]. + + The media type message/news, as previously registered with IANA, is + hereby declared obsolete. The intent of this media type was to + define a standard way of transmitting news articles via mail for + human reading. However, it was never widely implemented, and its + default treatment as application/octet-stream by agents that did not + recognize it was counter-productive. The media type message/rfc822 + (defined in Section 5.2.1 of [RFC2046]) SHOULD be used in its place. + + The updated MIME media type definition of message/news is: + + MIME type name: message + + MIME subtype name: news + + Required parameters: none + + Optional parameters: none + + Encoding considerations: same as message/rfc822 + + Security considerations: News articles may constitute "control + messages", which can have effects on a + host's news system beyond just addition + of information. Since control messages + may occur in normal news flow, most hosts + are suitably defended against undesired + effects already, but transmission of news + articles via mail may bypass + + + +Allbery & Lindsey Standards Track [Page 29] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + firewall-type defenses. Reading a news + article transmitted by mail involves no + hazards beyond those of mail, but + submitting it to news software for + processing should be done with care. + + Interoperability considerations: + Rarely used, and therefore often + handled as application/octet-stream. + Disposition should by default be inline. + + Published specification: RFC 5537 + + Applications that use this media type: + Some old mail and news user agents. + + Intended usage: OBSOLETE + + Author: Henry Spencer + + Change controller: IETF + +4.1. application/news-transmission + + The media type application/news-transmission is intended for the + encapsulation of complete news articles where the intention is that + the recipient should then inject them into Netnews. This application + type provides one of the methods for mailing articles to moderators + (see Section 3.5.1) and may be used to convey messages to an + injecting agent. This encapsulation removes the need to transform an + email message into a Netnews proto-article and provides a way to send + a Netnews article using MIME through a transport medium that does not + support 8bit data. + + The MIME media type definition of application/news-transmission is: + + MIME type name: application + + MIME subtype name: news-transmission + + Required parameters: none + + Optional parameters: One and only one of "usage=moderate", + "usage=inject", or "usage=relay". + + + + + + + +Allbery & Lindsey Standards Track [Page 30] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + Encoding considerations: A transfer-encoding different from that + of the article transmitted MAY be + supplied to ensure correct transmission + over some 7bit transport medium. + + Security considerations: News articles may constitute "control + messages", which can have effects on a + host's news system beyond just addition + of information. Since control messages + may occur in normal news flow, most hosts + are suitably defended against undesired + effects already, but transmission of news + articles via mail may bypass + firewall-type defenses. + + Published specification: RFC 5537 + + Body part: A complete proto-article ready for + injection into Netnews or an article + being relayed to another agent. + + Applications that use this media type: + Injecting agents, Netnews moderators. + + Intended usage: COMMON + + Change controller: IETF + + usage=moderate indicates the article is intended for a moderator, + usage=inject for an injecting agent, and usage=relay for a relaying + agent. The entity receiving the article may only implement one type + of agent, in which case the parameter MAY be omitted. + + Contrary to the prior registration of this media type, article + batches are not permitted as a body part. Multiple messages or a + message with multiple application/news-transmission parts may be used + instead. + +4.2. application/news-groupinfo + + The application/news-groupinfo media type is used in conjunction with + the newgroup control message (see Section 5.2.1). Its body part + contains brief information about a newsgroup: the newsgroup name, its + description, and its moderation status. + + The MIME media type definition of application/news-groupinfo is: + + + + + +Allbery & Lindsey Standards Track [Page 31] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + MIME type name: application + + MIME subtype name: news-groupinfo + + Required parameters: none + + Optional parameters: charset, which MUST be a charset + registered for use with MIME text types. + It has the same syntax as the parameter + defined for text/plain [RFC2046]. + Specifies the charset of the body part. + If not given, the charset defaults to + US-ASCII [ASCII]. + + Encoding considerations: 7bit or 8bit encoding MUST be used to + maintain compatibility. + + Security considerations: None. + + Interoperability considerations: + Disposition should by default be inline. + + Applications that use this media type: + Control message issuers, relaying + agents, serving agents. + + Published specification: RFC 5537 + + Intended usage: COMMON + + Change controller: IETF + + The content of the application/news-groupinfo body part is defined + as: + + groupinfo-body = [ newsgroups-tag CRLF ] + newsgroups-line CRLF + newsgroups-tag = %x46.6F.72 SP %x79.6F.75.72 SP + %x6E.65.77.73.67.72.6F.75.70.73 SP + %x66.69.6C.65.3A + ; case sensitive + ; "For your newsgroups file:" + newsgroups-line = newsgroup-name + [ 1*HTAB newsgroup-description ] + [ *WSP moderation-flag ] + newsgroup-description + = eightbit-utext *( *WSP eightbit-utext ) + moderation-flag = SP "(" %x4D.6F.64.65.72.61.74.65.64 ")" + + + +Allbery & Lindsey Standards Track [Page 32] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + ; SPACE + case sensitive "(Moderated)" + eightbit-utext = VCHAR / %d127-255 + + This unusual format is backward-compatible with the scanning of the + body of newgroup control messages for descriptions done by Netnews + implementations that predate this specification. Although optional, + the <newsgroups-tag> SHOULD be included for backward compatibility. + + The <newsgroup-description> MUST NOT contain any occurrence of the + string "(Moderated)" within it. Moderated newsgroups MUST be marked + by appending the case-sensitive text " (Moderated)" at the end. + + While a charset parameter is defined for this media type, most + existing software does not understand MIME header fields or correctly + handle descriptions in a variety of charsets. Using a charset of US- + ASCII where possible is therefore RECOMMENDED; if not possible, UTF-8 + [RFC3629] SHOULD be used. Regardless of the charset used, the + constraints of the above grammar MUST be met and the <newsgroup-name> + MUST be represented in that charset using the same octets as would be + used with a charset of US-ASCII. + +4.3. application/news-checkgroups + + The application/news-checkgroups media type contains a list of + newsgroups within a hierarchy or hierarchies, including their + descriptions and moderation status. It is primarily for use with the + checkgroups control message (see Section 5.2.3). + + The MIME media type definition of application/news-checkgroups is: + + MIME type name: application + + MIME subtype name: news-checkgroups + + Required parameters: none + + Optional parameters: charset, which MUST be a charset + registered for use with MIME text types. + It has the same syntax as the parameter + defined for text/plain [RFC2046]. + Specifies the charset of the body part. + If not given, the charset defaults to + US-ASCII [ASCII]. + + Encoding considerations: 7bit or 8bit encoding MUST be used to + maintain compatibility. + + + + + +Allbery & Lindsey Standards Track [Page 33] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + Security considerations: This media type provides only a means + for conveying a list of newsgroups and + does not provide any information + indicating whether the sender is + authorized to state which newsgroups + should exist within a hierarchy. Such + authorization must be accomplished by + other means. + + Interoperability considerations: + Disposition should by default be inline. + + Applications that use this media type: + Control message issuers, relaying + agents, serving agents. + + Published specification: RFC 5537 + + Intended usage: COMMON + + Change controller: IETF + + The content of the application/news-checkgroups body part is defined + as: + + checkgroups-body = *( valid-group CRLF ) + valid-group = newsgroups-line ; see Section 4.2 + + The same restrictions on charset, <newsgroup-name>, and <newsgroup- + description> apply for this media type as for application/ + news-groupinfo. + + One application/news-checkgroups message may contain information for + one or more hierarchies and is considered complete for any hierarchy + for which it contains a <valid-group> unless accompanied by external + information limiting its scope (such as a <chkscope> parameter to a + checkgroups control message, as described in Section 5.2.3). In + other words, an application/news-checkgroups body part consisting of + + example.moderated A moderated newsgroup (Moderated) + example.test An unmoderated test group + + is a statement that the example.* hierarchy contains two newsgroups, + example.moderated and example.test, and no others. This media type + therefore MUST NOT be used for conveying partial information about a + hierarchy; if a group from a given hierarchy is present, all groups + + + + + +Allbery & Lindsey Standards Track [Page 34] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + that exist in that hierarchy MUST be listed unless its scope is + limited by external information, in which case all groups SHOULD be + listed. + + Spaces are used in this example for formatting reasons. In an actual + message, the newsgroup name and description MUST be separated by one + or more tabs (HTAB, ASCII %d09), not spaces. + +5. Control Messages + + A control message is an article that contains a Control header field + and thereby indicates that some action should be taken by an agent + other than distribution and display. Any article containing a + Control header field (defined in Section 3.2.3 of [RFC5536]) is a + control message. Additionally, the action of an article containing a + Supersedes header field is described here; while such an article is + not a control message, it specifies an action similar to the cancel + control message. + + The <control-command> of a Control header field comprises a <verb>, + which indicates the action to be taken, and one or more <argument> + values, which supply the details. For some control messages, the + body of the article is also significant. Each recognized <verb> (the + control message type) is described in a separate section below. + Agents MAY accept other control message types than those specified + below, and MUST either ignore or reject control messages with + unrecognized types. Syntactic definitions of valid <argument> values + and restrictions on control message bodies are given in the section + for each control message type. + + Contrary to [RFC1036], the presence of a Subject header field + starting with the string "cmsg " MUST NOT cause an article to be + interpreted as a control message. Agents MAY reject an article that + has such a Subject header field and no Control header field as + ambiguous. Likewise, the presence of a <newsgroup-name> ending in + ".ctl" in the Newsgroups header field or the presence of an Also- + Control header field MUST NOT cause the article to be interpreted as + a control message. + +5.1. Authentication and Authorization + + Control messages specify actions above and beyond the normal + processing of an article and are therefore potential vectors of abuse + and unauthorized action. There is, at present, no standardized means + of authenticating the sender of a control message or verifying that + the contents of a control message were sent by the claimed sender. + There are, however, some unstandardized authentication mechanisms in + common use, such as [PGPVERIFY]. + + + +Allbery & Lindsey Standards Track [Page 35] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + Agents acting on control messages SHOULD take steps to authenticate + control messages before acting on them, as determined by local + authorization policy. Whether this is done via the use of an + unstandardized authentication protocol, by comparison with + information obtained through another protocol, by human review, or by + some other means is left unspecified by this document. Further + extensions or revisions of this protocol are expected to standardize + a digital signature mechanism. + + Agents are expected to have their own local authorization policies + for which control messages will be honored. No Netnews agent is ever + required to act on any control message. The following descriptions + specify the actions that a control message requests, but an agent MAY + always decline to act on any given control message. + +5.2. Group Control Messages + + A group control message is any control message type that requests + some update to the list of newsgroups known to a news server. The + standard group control message types are "newgroup", "rmgroup", and + "checkgroups". + + Before honoring any group control message, an agent MUST check the + newsgroup or newsgroups affected by that control message and decline + to create any newsgroups not in conformance with the restrictions in + Section 3.1.4 of [RFC5536]. + + All of the group control messages MUST have an Approved header field + (Section 3.2.1 of [RFC5536]). Group control messages without an + Approved header field SHOULD NOT be honored. + + Group control messages affecting specific groups (newgroup and + rmgroup control messages, for example) SHOULD include the <newsgroup- + name> for the group or groups affected in their Newsgroups header + field. Other newsgroups MAY be included in the Newsgroups header + field so that the control message will reach more news servers, but + due to the special relaying rules for group control messages (see + Section 3.6) this is normally unnecessary and may be excessive. + +5.2.1. The newgroup Control Message + + The newgroup control message requests that the specified group be + created or, if already existing, that its moderation status or + description be changed. The syntax of its Control header field is: + + control-command =/ Newgroup-command + Newgroup-command = "newgroup" Newgroup-arguments + Newgroup-arguments = 1*WSP newsgroup-name + + + +Allbery & Lindsey Standards Track [Page 36] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + [ 1*WSP newgroup-flag ] + newgroup-flag = "moderated" + + If the request is honored, the moderation status of the group SHOULD + be set in accordance with the presence or absence of the <newgroup- + flag> "moderated". "moderated" is the only flag defined by this + protocol. Other flags MAY be defined by extensions to this protocol + and accepted by agents. If an agent does not recognize the + <newgroup-flag> of a newgroup control message, it SHOULD ignore that + control message. + + The body of a newgroup message SHOULD contain an entity of type + application/news-groupinfo specifying the description of the + newsgroup, either as the entire body or as an entity within a + multipart/mixed object [RFC2046]. If such an entity is present, the + moderation status specified therein MUST match the moderation status + specified by the <newgroup-flag>. The body of a newgroup message MAY + contain other entities (encapsulated in multipart/mixed) that provide + additional information about the newsgroup or the circumstances of + the control message. + + In the absence of an application/news-groupinfo entity, a news server + MAY search the body of the message for the line "For your newsgroups + file:" and take the following line as a <newsgroups-line>. Prior to + the standardization of application/news-groupinfo, this was the + convention for providing a newsgroup description. + + If the request is honored and contains a newsgroup description, and + if the news server honoring it stores newsgroup descriptions, the + stored newsgroup description SHOULD be updated to the description + specified in the control message, even if no other property of the + group has changed. + +5.2.1.1. newgroup Control Message Example + + A newgroup control message requesting creation of the moderated + newsgroup example.admin.info. + + From: "example.* Administrator" <admin@noc.example> + Newsgroups: example.admin.info + Date: 27 Feb 2002 12:50:22 +0200 + Subject: cmsg newgroup example.admin.info moderated + Approved: admin@noc.example + Control: newgroup example.admin.info moderated + Message-ID: <ng-example.admin.info-20020227@noc.example> + MIME-Version: 1.0 + Content-Type: multipart/mixed; boundary="nxtprt" + Content-Transfer-Encoding: 8bit + + + +Allbery & Lindsey Standards Track [Page 37] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + This is a MIME control message. + --nxtprt + Content-Type: application/news-groupinfo; charset=us-ascii + + For your newsgroups file: + example.admin.info About the example.* groups (Moderated) + + --nxtprt + Content-Type: text/plain; charset=us-ascii + + A moderated newsgroup for announcements about new newsgroups in + the example.* hierarchy. + + --nxtprt-- + + Spaces are used in this example for formatting reasons. In an actual + message, the newsgroup name and description MUST be separated by one + or more tabs (HTAB, ASCII %d09), not spaces. + +5.2.2. The rmgroup Control Message + + The rmgroup control message requests that the specified group be + removed from a news server's list of valid groups. The syntax of its + Control header field is: + + control-command =/ Rmgroup-command + Rmgroup-command = "rmgroup" Rmgroup-arguments + Rmgroup-arguments = 1*WSP newsgroup-name + + The body of the control message MAY contain anything, usually an + explanatory text. + +5.2.3. The checkgroups Control Message + + The checkgroups control message contains a list of all the valid + groups in a hierarchy with descriptions and moderation status. It + requests that a news server update its valid newsgroup list for that + hierarchy to include the groups specified, remove any groups not + specified, and update group descriptions and moderation status to + match those given in the checkgroups control message. The syntax of + its Control header field is: + + control-command =/ Checkgroup-command + Checkgroup-command = "checkgroups" Checkgroup-arguments + Checkgroup-arguments= [ chkscope ] [ chksernr ] + chkscope = 1*( 1*WSP ["!"] newsgroup-name ) + chksernr = 1*WSP "#" 1*DIGIT + + + + +Allbery & Lindsey Standards Track [Page 38] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + A checkgroups message is interpreted as an exhaustive list of the + valid groups in all hierarchies or sub-hierarchies with a prefix + listed in the <chkscope> argument, excluding any sub-hierarchy where + the <chkscope> argument is prefixed by "!". For complex cases with + multiple <chkscope> arguments, start from an empty list of groups, + include all groups in the checkgroups control message matching + <chkscope> arguments without a "!" prefix, and then exclude all + groups matching <chkscope> arguments with a "!" prefix. Follow this + method regardless of the order of the <chkscope> arguments in the + Control header field. + + If no <chkscope> argument is given, it applies to all hierarchies for + which group statements appear in the body of the message. + + Since much existing software does not honor the <chkscope> argument, + the body of the checkgroups control message MUST NOT contain group + statements for newsgroups outside the intended scope and SHOULD + contain a correct newsgroup list even for sub-hierarchies excluded + with "!" <chkscope> terms. News servers, however, MUST honor + <chkscope> as specified here. + + The <chksernr> argument may be any positive integer. If present, it + MUST increase with every change to the newsgroup list, MUST NOT ever + decrease, and MUST be included in all subsequent checkgroups control + messages with the same scope. If provided, news servers SHOULD + remember the <chksernr> value of the previous checkgroups control + message honored for a particular hierarchy or sub-hierarchy and + decline to honor any subsequent checkgroups control message for the + same hierarchy or sub-hierarchy with a smaller <chksernr> value or + with no <chksernr> value. + + There is no upper limit on the length of <chksernr>, other than the + limitation on the length of header fields. Implementations may + therefore want to do comparisons by zero-padding the shorter of two + <chksernr> values on the left and then doing a string comparison, + rather than assuming <chksernr> can be manipulated as a number. + + For example, the following Control header field + + Control: checkgroups de !de.alt #2009021301 + + indicates that the body of the message will list every newsgroup in + the de.* hierarchy, excepting the de.alt.* sub-hierarchy, and should + not be honored if a checkgroups control message with a serial number + greater than 2009021301 was previously honored. The serial number in + this example was formed from the date in YYYYMMDD format, followed by + a two-digit sequence number within that date. + + + + +Allbery & Lindsey Standards Track [Page 39] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + The body of the message is an entity of type application/ + news-checkgroups. It SHOULD be declared as such with appropriate + MIME headers, but news servers SHOULD interpret checkgroups messages + that lack the appropriate MIME headers as if the body were of type + application/news-checkgroups for backward compatibility. + +5.3. The cancel Control Message + + The cancel control message requests that a target article be + withdrawn from circulation and access. The syntax of its Control + header field is: + + control-command =/ Cancel-command + Cancel-command = "cancel" Cancel-arguments + Cancel-arguments = 1*WSP msg-id + + The argument identifies the article to be cancelled by its message + identifier. The body of the control message MAY contain anything, + usually an explanatory text. + + A serving agent that elects to honor a cancel message SHOULD make the + article unavailable to reading agents (perhaps by deleting it + completely). If the cancel control message arrives before the + article it targets, news servers choosing to honor it SHOULD remember + the message identifier that was cancelled and reject the cancelled + article when it arrives. + + To best ensure that it will be relayed to the same news servers as + the original message, a cancel control message SHOULD have the same + Newsgroups header field as the message it is cancelling. + + Cancel control messages listing moderated newsgroups in their + Newsgroups header field MUST contain an Approved header field like + any other article in a moderated newsgroup. This means that cancels + posted to a moderated newsgroup will normally be sent to the + moderator first for approval. Outside of moderated newsgroups, + cancel messages are not required to contain an Approved header field. + + Contrary to [RFC1036], cancel control messages are not required to + contain From and Sender header fields matching the target message. + This requirement only encouraged cancel issuers to conceal their + identity and provided no security. + +5.4. The Supersedes Header Field + + The presence of a Supersedes header field in an article requests that + the message identifier given in that header field be withdrawn in + exactly the same manner as if it were the target of a cancel control + + + +Allbery & Lindsey Standards Track [Page 40] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + message. Accordingly, news servers SHOULD apply to a Supersedes + header field the same authentication and authorization checks as they + would apply to cancel control messages. If the Supersedes header + field is honored, the news server SHOULD take the same actions as it + would take when honoring a cancel control message for the given + target article. + + The article containing the Supersedes header field, whether or not + the Supersedes header field is honored, SHOULD be handled as a normal + article and SHOULD NOT receive the special treatment of control + messages described in Section 3.7. + +5.5. The ihave and sendme Control Messages + + The ihave and sendme control messages implement a predecessor of the + NNTP [RFC3977] protocol. They are largely obsolete on the Internet + but still see use in conjunction with some transport protocols such + as UUCP [UUCP]. News servers are not required to support them. + + ihave and sendme control messages share similar syntax for their + Control header fields and bodies: + + control-command =/ Ihave-command + Ihave-command = "ihave" Ihave-arguments + Ihave-arguments = 1*WSP *( msg-id 1*WSP ) relayer-name + control-command =/ Sendme-command + Sendme-command = "sendme" Sendme-arguments + Sendme-arguments = Ihave-arguments + relayer-name = path-identity ; see 3.1.5 of [RFC5536] + ihave-body = *( msg-id CRLF ) + sendme-body = ihave-body + + The body of the article consists of a list of <msg-id>s, one per + line. The message identifiers SHOULD be put in the body of the + article, not in the Control header field, but news servers MAY + recognize and process message identifiers in the Control header field + for backward compatibility. Message identifiers MUST NOT be put in + the Control header field if they are present in the body of the + control message. + + The ihave message states that the named relaying agent has received + articles with the specified message identifiers, which may be of + interest to the relaying agents receiving the ihave message. The + sendme message requests that the agent receiving it send the articles + having the specified message identifiers to the named relaying agent. + Contrary to [RFC1036], the relayer-name MUST be given as the last + argument in the Control header field. + + + + +Allbery & Lindsey Standards Track [Page 41] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + Upon receipt of the sendme message (and a decision to honor it), the + receiving agent sends the article or articles requested. The + mechanism for this transmission is unspecified by this document and + is arranged between the sites involved. + + These control messages are normally sent as point-to-point articles + between two sites and not then sent on to other sites. Newsgroups + beginning with "to." are reserved for such point-to-point + communications and are formed by prepending "to." to a <relayer-name> + to form a <newsgroup-name>. Articles with such a group in their + Newsgroups header fields SHOULD NOT be sent to any news server other + than the one identified by <relayer-name>. + +5.6. Obsolete Control Messages + + The following control message types are declared obsolete by this + document and SHOULD NOT be sent or honored: + + sendsys + version + whogets + senduuname + +6. Security Considerations + + Netnews is designed for broad dissemination of public messages and + offers little in the way of security. What protection Netnews has + against abuse and impersonation is provided primarily by the + underlying transport layer. In large Netnews networks where news + servers cannot be relied upon to enforce authentication and + authorization requirements at the transport layer, articles may be + trivially forged and widely read, and the identities of article + senders and the privacy of articles cannot be assured. + + See Section 5 of [RFC5536] for further security considerations + related to the format of articles. + +6.1. Compromise of System Integrity + + Control messages pose a particular security concern since acting on + unauthorized control messages may cause newsgroups to be removed, + articles to be deleted, and unwanted newsgroups to be created. + Administrators of news servers SHOULD therefore take steps to verify + the authenticity of control messages as discussed in Section 5.1. + Articles containing Supersedes header fields are effectively cancel + control messages and SHOULD be subject to the same checks as + + + + + +Allbery & Lindsey Standards Track [Page 42] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + discussed in Section 5.4. Currently, many sites are ignoring all + cancel control messages and Supersedes header fields due to the + difficulty of authenticating them and their widespread abuse. + + Cancel control messages are not required to have the same Newsgroups + header field as the messages they are cancelling. Since they are + sometimes processed before the original message is received, it may + not be possible to check that the Newsgroup header fields match. + This allows a malicious poster to inject a cancel control message for + an article in a moderated newsgroup without adding an Approved header + field to the control message, and to hide malicious cancel control + messages from some reading agents by using a different Newsgroups + header field so that the cancel control message is not accepted by + all news servers that accepted the original message. + + All agents should be aware that all article content, most notably + including the content of the Control header field, is potentially + untrustworthy and malicious. Articles may be constructed in + syntactically invalid ways to attempt to overflow internal buffers, + violate hidden assumptions, or exploit implementation weaknesses. + For example, some news server implementations have been successfully + attacked via inclusion of Unix shell code in the Control header + field. All article contents, and particularly control message + contents, SHOULD be handled with care and rigorously verified before + any action is taken on the basis of the contents of the article. + + A malicious poster may add an Approved header field to bypass the + moderation process of a moderated newsgroup. Injecting agents SHOULD + verify that messages approved for a moderated newsgroup are being + injected by the moderator using authentication information from the + underlying transport or some other authentication mechanism arranged + with the moderator. There is, at present, no standardized method of + authenticating approval of messages to moderated groups, although + some unstandardized authentication methods such as [PGPMOOSE] are in + common use. + + A malicious news server participating in a Netnews network may bypass + checks performed by injecting agents, forge Path header fields and + other trace information (such as Injection-Info header fields), and + otherwise compromise the authorization requirements of a Netnews + network. News servers SHOULD use the facilities of the underlying + transport to authenticate their peers and reject articles from + injecting and relaying agents that do not follow the requirements of + this protocol or the Netnews network. + + + + + + + +Allbery & Lindsey Standards Track [Page 43] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + +6.2. Denial of Service + + The proper functioning of individual newsgroups can be disrupted by + the excessive posting of unwanted articles; by the repeated posting + of identical or near identical articles; by posting followups that + either are unrelated to their precursors or that quote their + precursors in full with the addition of minimal extra material + (especially if this process is iterated); by crossposting to, or + requesting followups to, totally unrelated newsgroups; and by abusing + control messages and the Supersedes header field to delete articles + or newsgroups. + + Such articles intended to deny service, or other articles of an + inflammatory nature, may also have their From or Reply-To addresses + set to valid but incorrect email addresses, thus causing large + volumes of email to descend on the true owners of those addresses. + Users and agents should always be aware that identifying information + in articles may be forged. + + A malicious poster may prevent an article from being seen at a + particular site by including in the Path header field of the proto- + article the <path-identity> of that site. Use of the <diag-keyword> + "POSTED" by injecting agents to mark the point of injection can + prevent this attack. + + Primary responsibility for preventing such attacks lies with + injecting agents, which can apply authentication and authorization + checks via the underlying transport and prevent those attempting + denial-of-service attacks from posting messages. If specific + injecting agents fail to live up to their responsibilities, they may + be excluded from the Netnews network by configuring relaying agents + to reject articles originating from them. + + A malicious complainer may submit a modified copy of an article (with + an altered Injection-Info header field, for instance) to the + administrator of an injecting agent in an attempt to discredit the + author of that article and even to have his posting privileges + removed. Administrators SHOULD therefore obtain a genuine copy of + the article from their own serving agent before taking action in + response to such a complaint. + +6.3. Leakage + + Articles that are intended to have restricted distribution are + dependent on the goodwill of every site receiving them. Restrictions + on dissemination and retention of articles may be requested via the + Archive and Distribution header fields, but such requests cannot be + enforced by the protocol. + + + +Allbery & Lindsey Standards Track [Page 44] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + The flooding algorithm used by Netnews transports such as NNTP + [RFC3977] is extremely good at finding any path by which articles can + leave a subnet with supposedly restrictive boundaries, and + substantial administrative effort is required to avoid this. + Organizations wishing to control such leakage are advised to + designate a small number of gateways to handle all news exchanges + with the outside world. + + The sendme control message (Section 5.5), insofar as it is still + used, can be used to request articles that the requester should not + have access to. + +7. IANA Considerations + + IANA has registered the following media types, described elsewhere in + this document, for use with the Content-Type header field, in the + IETF tree in accordance with the procedures set out in [RFC4288]. + + application/news-transmission (4.1) + application/news-groupinfo (4.2) + application/news-checkgroups (4.3) + + application/news-transmission is a change to a previous registration. + + IANA has registered the new header field, Original-Sender, in the + Permanent Message Header Field Repository, using the template in + Section 3.10.3. + + IANA has changed the status of the message/news media type to + "OBSOLETE". message/rfc822 should be used instead. An updated + template is included in Section 4. + +8. References + +8.1. Normative References + + [ASCII] American National Standard for Information Systems, + "Coded Character Sets - 7-Bit American National + Standard Code for Information Interchange (7-Bit + ASCII)", ANSI X3.4, 1986. + + [RFC2046] Freed, N. and N. Borenstein, "Multipurpose Internet + Mail Extensions (MIME) Part Two: Media Types", + RFC 2046, November 1996. + + [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate + Requirement Levels", BCP 14, RFC 2119, March 1997. + + + + +Allbery & Lindsey Standards Track [Page 45] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + + [RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO + 10646", STD 63, RFC 3629, November 2003. + + [RFC4288] Freed, N. and J. Klensin, "Media Type Specifications + and Registration Procedures", BCP 13, RFC 4288, + December 2005. + + [RFC5234] Crocker, D. and P. Overell, "Augmented BNF for Syntax + Specifications: ABNF", STD 68, RFC 5234, January 2008. + + [RFC5322] Resnick, P., Ed., "Internet Message Format", RFC 5322, + October 2008. + + [RFC5536] Murchison, K., Ed., Lindsey, C., and D. Kohn, "Netnews + Article Format", RFC 5536, November 2009. + +8.2. Informative References + + [PGPMOOSE] Rose, G., "PGP Moose", November 1998. + + [PGPVERIFY] Lawrence, D., "Signing Control Messages", August 2001, + <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>. + + [RFC1036] Horton, M. and R. Adams, "Standard for interchange of + USENET messages", RFC 1036, December 1987. + + [RFC2045] Freed, N. and N. Borenstein, "Multipurpose Internet + Mail Extensions (MIME) Part One: Format of Internet + Message Bodies", RFC 2045, November 1996. + + [RFC2606] Eastlake, D. and A. Panitz, "Reserved Top Level DNS + Names", BCP 32, RFC 2606, June 1999. + + [RFC3798] Hansen, T. and G. Vaudreuil, "Message Disposition + Notification", RFC 3798, May 2004. + + [RFC3977] Feather, C., "Network News Transfer Protocol (NNTP)", + RFC 3977, October 2006. + + [SON-OF-1036] Spencer, H., "News Article Format and Transmission", + Work in Progress, May 2009. + + [USEAGE] Lindsey, C., "Usenet Best Practice", Work in Progress, + March 2005. + + [UUCP] O'Reilly, T. and G. Todino, "Managing UUCP and + Usenet", O'Reilly & Associates Ltd., January 1992. + + + + +Allbery & Lindsey Standards Track [Page 46] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + +Appendix A. Changes to the Existing Protocols + + This document prescribes many changes, clarifications, and new + features since the protocol described in [RFC1036]. Most notably: + + o A new, backward-compatible Path header field format that permits + standardized embedding of additional trace and authentication + information is now RECOMMENDED. See Section 3.2. Folding of the + Path header is permitted. + + o Trimming of the References header field is REQUIRED, and a + mechanism for doing so is defined. + + o Addition of the new Injection-Date header field is required in + some circumstances for posting agents (Section 3.4.2) and + injecting agents (Section 3.5), and MUST be used by news servers + for date checks (Section 3.6). Injecting agents are also strongly + encouraged to put all local trace information in the new + Injection-Info header field. + + o A new media type is defined for transmitting Netnews articles + through other media (Section 4.1), and moderators SHOULD prepare + to receive submissions in that format (Section 3.5.1). + + o Certain control messages (Section 5.6) are declared obsolete, and + the special significance of "cmsg" at the start of a Subject + header field is removed. + + o Additional media types are defined for improved structuring, + specification, and automated processing of control messages + (Sections 4.2 and 4.3). + + o Two new optional parameters are added to the checkgroups control + message. + + o The message/news media type is declared obsolete. + + o Cancel control messages are no longer required to have From and + Sender header fields matching those of the target message, as this + requirement added no real security. + + o The relayer-name parameter in the Control header field of ihave + and sendme control messages is now required. + + In addition, many protocol steps and article verification + requirements that are unmentioned or left ambiguous by [RFC1036] but + are widely implemented by Netnews servers have been standardized and + specified in detail. + + + +Allbery & Lindsey Standards Track [Page 47] + +RFC 5537 Netnews Architecture and Protocols November 2009 + + +Appendix B. Acknowledgements + + This document is the result of a twelve-year effort and the number of + people that have contributed to its content are too numerous to + mention individually. Many thanks go out to all past and present + members of the USEFOR Working Group of the Internet Engineering Task + Force (IETF) and the accompanying mailing list. + + Special thanks are due to Henry Spencer, whose [SON-OF-1036] draft + served as the initial basis for this document. + +Authors' Addresses + + Russ Allbery (editor) + Stanford University + P.O. Box 20066 + Stanford, CA 94309 + US + + EMail: rra@stanford.edu + URI: http://www.eyrie.org/~eagle/ + + + Charles H. Lindsey + 5 Clerewood Avenue + Heald Green + Cheadle + Cheshire SK8 3JU + United Kingdom + + Phone: +44 161 436 6131 + EMail: chl@clerew.man.ac.uk + + + + + + + + + + + + + + + + + + + +Allbery & Lindsey Standards Track [Page 48] + |