From 4bfd864f10b68b71482b35c818559068ef8d5797 Mon Sep 17 00:00:00 2001 From: Thomas Voss Date: Wed, 27 Nov 2024 20:54:24 +0100 Subject: doc: Add RFC documents --- doc/rfc/rfc6686.txt | 675 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 675 insertions(+) create mode 100644 doc/rfc/rfc6686.txt (limited to 'doc/rfc/rfc6686.txt') diff --git a/doc/rfc/rfc6686.txt b/doc/rfc/rfc6686.txt new file mode 100644 index 0000000..9e47d7c --- /dev/null +++ b/doc/rfc/rfc6686.txt @@ -0,0 +1,675 @@ + + + + + + +Internet Engineering Task Force (IETF) M. Kucherawy +Request for Comments: 6686 Cloudmark +Category: Informational July 2012 +ISSN: 2070-1721 + + + Resolution of the Sender Policy Framework (SPF) + and Sender ID Experiments + +Abstract + + In 2006, the IETF published a suite of protocol documents comprising + the Sender Policy Framework (SPF) and Sender ID: two proposed email + authentication protocols. Both of these protocols enable one to + publish, via the Domain Name System, a policy declaring which mail + servers were authorized to send email on behalf of the domain name + being queried. There was concern that the two would conflict in some + significant operational situations, interfering with message + delivery. + + The IESG required all of these documents (RFC 4405, RFC 4406, RFC + 4407, and RFC 4408) to be published as Experimental RFCs and + requested that the community observe deployment and operation of the + protocols over a period of two years from the date of publication to + determine a reasonable path forward. + + After six years, sufficient experience and evidence have been + collected that the experiments thus created can be considered + concluded. This document presents those findings. + +Status of This Memo + + This document is not an Internet Standards Track specification; it is + published for informational purposes. + + This document is a product of the Internet Engineering Task Force + (IETF). It represents the consensus of the IETF community. It has + received public review and has been approved for publication by the + Internet Engineering Steering Group (IESG). Not all documents + approved by the IESG are a candidate for any level of Internet + Standard; see Section 2 of RFC 5741. + + Information about the current status of this document, any errata, + and how to provide feedback on it may be obtained at + http://www.rfc-editor.org/info/rfc6686. + + + + + + +Kucherawy Informational [Page 1] + +RFC 6686 SPF/Sender ID Experiments July 2012 + + +Copyright Notice + + Copyright (c) 2012 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (http://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the Simplified BSD License. + +Table of Contents + + 1. Introduction ....................................................2 + 2. Definitions .....................................................3 + 3. Evidence of Deployment ..........................................3 + 3.1. DNS Resource Record Types ..................................3 + 3.2. Implementations ............................................5 + 3.3. The SUBMITTER SMTP Extension ...............................6 + 4. Evidence of Differences .........................................7 + 5. Analysis ........................................................7 + 6. Conclusions .....................................................8 + 7. Security Considerations .........................................9 + 8. References ......................................................9 + 8.1. Normative References .......................................9 + 8.2. Informative References .....................................9 + Appendix A. Background on the RRTYPE Issue ........................10 + Appendix B. Acknowledgments .......................................11 + +1. Introduction + + In April 2006, the IETF published the [SPF] and Sender ID email + authentication protocols, the latter consisting of three documents + ([SUBMITTER], [SENDER-ID], and [PRA]). Both of these protocols + enable one to publish, via the Domain Name System, a policy declaring + which mail servers are authorized to send email on behalf of the + selected domain name. + + Consensus did not clearly support one protocol over the other, and + there was significant concern that the two would conflict in some + significant operational situations, interfering with message + delivery. The IESG required the publication of all of these + documents as Experimental, and requested that the community observe + + + + +Kucherawy Informational [Page 2] + +RFC 6686 SPF/Sender ID Experiments July 2012 + + + deployment and operation of the protocols over a period of two years + from the date of publication in order to determine a reasonable path + forward. + + In line with the IESG's request to evaluate after a period of time, + this document concludes the experiments by presenting evidence + regarding both deployment and comparative effect of the two + protocols. At the end, it presents conclusions based on the data + collected. + + It is important to note that this document makes no direct technical + comparison of the two protocols in terms of correctness, weaknesses, + or use case coverage. The email community at large has already done + that through its deployment choices. Rather, the analysis presented + here is merely an observation of what has been deployed and supported + in the time since the protocols were published and lists conclusions + based on those observations. + + The data collected and presented here are presumed to be a reasonable + representative view of the global deployment data, which could never + itself be fully surveyed within a reasonable period of time. + +2. Definitions + + The term "RRTYPE" is used to refer to a Domain Name System ([DNS]) + Resource Record (RR) type. These are always expressed internally in + software as numbers, assigned according to the procedures in + [DNS-IANA] Assigned RRTYPEs also have names. The two of interest in + this work are the TXT RRTYPE (16) and the SPF RRTYPE (99). + +3. Evidence of Deployment + + This section presents the collected research done to determine what + parts of the two protocol suites are in general use as well as + related issues like [DNS] support. + +3.1. DNS Resource Record Types + + Three large-scale DNS surveys were run that looked for the two + supported kinds of RRTYPEs that can contain SPF policy statements. + These surveys selected substantial sets of distinct domain names from + email headers and logs over long periods, regardless of whether the + DNS data for those domains included A, MX, or any other RRTYPEs. The + nameservers for these domains were queried, asking for both of the + RRTYPEs that could be used for SPF and/or Sender ID. + + + + + + +Kucherawy Informational [Page 3] + +RFC 6686 SPF/Sender ID Experiments July 2012 + + + In the tables below, replies were counted only if they included + prefixes that indicated the record was intended to be of a form + defined in either [SPF] or [SENDER-ID], though complete syntax + validation of the replies was not done. That is, the records started + either "v=spf1" or "spf2.0/", or they were not counted as replies. + + The tables are broken down into three parts: (a) the size of the + sample set, (b) a report about RRTYPE use independent of content, and + (c) a report about content independent of RRTYPE. + + "SPF+TXT" indicates the count of domains where both types were in + use. + + DNS Survey #1 (Cisco) + + +------------------+-----------+-------+ + | Domains queried | 1,000,000 | - | + +------------------+-----------+-------+ + | TXT replies | 397,511 | 39.8% | + | SPF replies | 6,627 | <1.0% | + | SPF+TXT replies | 6,603 | <1.0% | + +------------------+-----------+-------+ + | v=spf1 replies | 395,659 | 39.6% | + | spf2.0/* replies | 5,291 | <1.0% | + +------------------+-----------+-------+ + + Domains were selected as the top million domains as reported by + Alexa, which monitors browser activity. + + DNS Survey #2 (The Trusted Domain Project) + + +------------------+-----------+-------+ + | Domains queried | 278,353 | - | + +------------------+-----------+-------+ + | TXT replies | 156,894 | 56.4% | + | SPF replies | 2,876 | 1.0% | + | SPF+TXT replies | 2,689 | <1.0% | + +------------------+-----------+-------+ + | v=spf1 replies | 149,985 | 53.9% | + | spf2.0/* replies | 7,285 | 2.7% | + +------------------+-----------+-------+ + + This survey selected its domains from data observed in email headers + and previous SPF and Sender ID evaluations, collected from 23 + reporting hosts across a handful of unrelated operators over a period + of 22 months. + + + + + +Kucherawy Informational [Page 4] + +RFC 6686 SPF/Sender ID Experiments July 2012 + + + During this second survey, some domains were observed to provide + immediate answers for RRTYPE 16 queries, but would time out waiting + for replies to RRTYPE 99 queries. For example, it was observed that + 4,360 (over 1.6%) distinct domains in the survey returned a result of + some kind (a record or an error) for the TXT query in time N, while + the SPF query ultimately failed after at least time 4N. + + DNS Survey #3 (Hotmail) + + +------------------+-----------+-------+ + | Domains queried | 100,000 | - | + +------------------+-----------+-------+ + | TXT replies | 46,221 | 46.2% | + | SPF replies | 954 | <1.0% | + | SPF+TXT replies | 1,383 | 1.4% | + +------------------+-----------+-------+ + + Hotmail's domain set was selected from live email traffic at the time + the sample was extracted. Only the RRTYPE portion of the report is + available. + + A separate survey was done of queries for RRTYPE 16 and RRTYPE 99 + records by observing nameserver traffic records. Only a few queries + were ever received for RRTYPE 99 records, and those almost + exclusively came from one large email service provider that queried + for both RRTYPEs. The vast majority of other querying agents only + ever requested RRTYPE 16. + +3.2. Implementations + + It is likely impossible to determine from a survey which Mail + Transfer Agents (MTAs) have SPF and/or Sender ID checking enabled at + message ingress since it does not appear, for example, in the reply + to the EHLO command from extended [SMTP]. Therefore, we relied on + evidence found via web searches and observed the following: + + o A web site [SID-IMPL] dedicated to highlighting Sender ID + implementations, last updated in late 2007, listed 13 commercial + implementations, which we assume means they implement the + Purported Responsible Address (PRA) checks. At least one of them + is known no longer to be supported by its vendor. There were no + free open-source implementations listed. + + o The [OPENSPF] web site maintains a list of implementations of SPF. + At the time of this document's writing, it listed six libraries, + 22 MTAs with built-in SPF implementations, and numerous patches + for MTAs and mail clients. The set included a mix of commercial + and free open-source implementations. + + + +Kucherawy Informational [Page 5] + +RFC 6686 SPF/Sender ID Experiments July 2012 + + +3.3. The SUBMITTER SMTP Extension + + The PRA is the output of a heuristic that seeks to scan a message + header and extract from it the email address most likely to be the + one responsible for injection of that message into the mail stream. + The SUBMITTER extension to SMTP is a mechanism to provide an early + hint (i.e., as part of the MAIL command in an SMTP session) to the + receiving MTA of what the PRA would be on full receipt of the + message. + + In a review of numerous MTAs in current or recent use, two + (Santronics WinServer and McAfee MxLogic) were found to contain + implementations of the SMTP SUBMITTER extension as part of the MTA + service, which could act as an enabler to Sender ID. + + An unknown number of SMTP clients implement the SUBMITTER SMTP + extension. Although information from MTA logs indicates substantial + use of the SMTP extension, it is not possible to determine whether + the usage is from multiple instances of the same SMTP client or + different SMTP client implementations. + + An active survey of MTAs accessible over the Internet was performed. + The MTAs selected were found by querying for MX and A resource + records of a subset of all domains observed by The Trusted Domain + Project's data collection system in the preceding 20 months. The + results were as follows: + + SUBMITTER Survey (The Trusted Domain Project) + + +-------------------+-----------+-------+ + | MTAs selected | 484,980 | - | + | MTAs responding | 371,779 | 76.7% | + | SUBMITTER enabled | 17,425 | 4.7% | + | MXLogic banner | 16,914 | 4.6% | + +-------------------+-----------+-------+ + + Note: The bottom two rows indicate the percentage of responding MTAs + with the stated property, not the percentage of selected MTAs. + + Based on the SMTP banner presented upon connection, the entire set of + SUBMITTER-enabled MTAs consisted of the two found during the review + (above) and a third whose identity could not be positively + determined. + + Of those few responding MTAs advertising the SUBMITTER SMTP + extension, 97% were different instances of one MTA. The service + operating that MTA (MXLogic, a division of McAfee) reported that + + + + +Kucherawy Informational [Page 6] + +RFC 6686 SPF/Sender ID Experiments July 2012 + + + about 11% of all observed SMTP sessions involved SMTP clients that + make use of the SUBMITTER extension. Note that this represents about + 11% of the clients of 4.6% of the responding MTAs in the survey. + +4. Evidence of Differences + + Separate surveys from Hotmail and The Trusted Domain Project compared + the cases where the PRA (used by Sender ID) and the RFC5321.MailFrom + address (used by SPF) differed. The results of these tests showed + that, at least 50% of the time, the two addresses were the same, but, + beyond that, the percentage varied substantially from one sampling + location to the next due to the nature of the mail streams they each + receive. + + Further, The Trusted Domain Project analyzed approximately 150,000 + messages and found that in more than 95% of those cases, Sender ID + and SPF reach the same conclusion about a message, meaning either + both protocols return a "pass" result or both return a "fail" result. + Note that this does not include an evaluation of whether "fail" meant + spam or other abusive mail was thus detected or that "pass" mail is + good mail; it is merely a measure of how often the two protocols + concurred. The data set yielding this response could not further + characterize the cases in which the answers differed. + + A second analysis of the same nature by Hotmail found that the two + protocols yielded the same result approximately 80% of the time when + evaluated across billions of messages. + + Anecdotally, the differences in conclusions have not been noted as + causing significant operational problems by the email-receiving + community. + +5. Analysis + + Given the six years that have passed since the publication of the + Experimental RFCs, and the evidence reported in the earlier sections + of this document, the following analysis appears to be supported: + + 1. There has not been substantial adoption of the RRTYPE 99 (SPF) + DNS resource record. In all large-scale surveys performed for + this work, fewer than 2% of responding domains published RRTYPE + 99 records, and almost no clients requested them. + + 2. Of the DNS resource records retrieved, fewer than 3% included + specific requests for processing of messages using the PRA + algorithm, which is an essential part of Sender ID. + + + + + +Kucherawy Informational [Page 7] + +RFC 6686 SPF/Sender ID Experiments July 2012 + + + 3. Although the two protocols often used different email address + fields as the subject being evaluated, no data collected showed + any substantial operational benefit, in terms of improved + accuracy, to using one mechanism over the other. + + 4. A review of known implementations shows significant support for + both protocols, though there were more implementations in support + of SPF than of Sender ID. Further, the SPF implementations + showed better upkeep and current interest than the Sender ID + implementations. + + 5. A survey of running MTAs shows fewer than 5% of them advertised + the SUBMITTER extension, which is a Sender ID enabler. Only + three implementations of it were found. + + 6. There remain obstacles to deployment of protocols that use DNS + RRTYPEs other than the most common ones, including firewalls and + DNS servers that block or discard requests for unknown RRTYPEs. + Further, few if any web-based DNS configuration tools offer + support for RRTYPE 99 records. + +6. Conclusions + + In light of the analysis in the previous section, the following + conclusions are supported: + + 1. The experiments comprising the series of RFCs defining the + SUBMITTER SMTP extension (RFC4405), the Sender ID mechanism + (RFC4406), the Purported Responsible Address algorithm (RFC4407), + and SPF (RFC4408), should be considered concluded. + + 2. The absence of significant adoption of the RRTYPE 99 DNS Resource + Record suggests that it has not attracted enough support to be + useful. + + 3. Unavailability of software implementing the protocols was not a + gating factor in terms of the selection of which to use. + + 4. The absence of significant adoption of the [SUBMITTER] extension, + [SENDER-ID], and [PRA], indicates that there is not a strong + community deploying and using these protocols. + + 5. [SPF] has widespread implementation and deployment, comparable to + that of many Standards Track protocols. + + Appendix A is offered as a cautionary review of problems that + affected the process of developing SPF and Sender ID in terms of + their use of the DNS. + + + +Kucherawy Informational [Page 8] + +RFC 6686 SPF/Sender ID Experiments July 2012 + + +7. Security Considerations + + This document contains information for the community, akin to an + implementation report, and does not introduce any new security + concerns. + +8. References + +8.1. Normative References + + [DNS] Mockapetris, P., "Domain names - implementation and + specification", STD 13, RFC 1035, November 1987. + + [PRA] Lyon, J., "Purported Responsible Address in E-Mail + Messages", RFC 4407, April 2006. + + [SENDER-ID] Lyon, J. and M. Wong, "Sender ID: Authenticating + E-Mail", RFC 4406, April 2006. + + [SPF] Wong, M. and W. Schlitt, "Sender Policy Framework (SPF) + for Authorizing Use of Domains in E-Mail, Version 1", + RFC 4408, April 2006. + + [SUBMITTER] Allman, E. and H. Katz, "SMTP Service Extension for + Indicating the Responsible Submitter of an E-Mail + Message", RFC 4405, April 2006. + +8.2. Informative References + + [DNS-EXPAND] IAB, Faltstrom, P., Austein, R., and P. Koch, "Design + Choices When Expanding the DNS", RFC 5507, April 2009. + + [DNS-IANA] Eastlake 3rd, D., "Domain Name System (DNS) IANA + Considerations", BCP 42, RFC 6195, March 2011. + + [OPENSPF] "Sender Policy Framework: Project Overview", + . + + [SID-IMPL] "Sender ID Framework Industry Support and Solutions", + October 2007, . + + [SMTP] Klensin, J., "Simple Mail Transfer Protocol", RFC 5321, + October 2008. + + + + + + + +Kucherawy Informational [Page 9] + +RFC 6686 SPF/Sender ID Experiments July 2012 + + +Appendix A. Background on the RRTYPE Issue + + SPF was originally created by a community of interested developers + outside the IETF, with the intent of bringing it to the IETF for + standardization after it had become relatively mature and ready for + the IETF Standards process. + + At the time of SPF's initial development, the prospect of getting an + RRTYPE allocated for SPF was not seriously considered, partly because + doing so had high barriers to entry. As a result, at the time it was + brought to the IETF for development and publication, there was + already a substantial and growing installed base that had SPF running + using TXT RRs. Eventually, the application was made for the new + RRTYPE as a result of pressure from the DNS experts in the community, + who insisted upon doing so as the preferred path toward using the DNS + for storing such things as policy data. + + Later, after RRTYPE 99 was assigned (long after IESG approval of + [SPF], in fact), a plan was put into place to effect a gradual + transition to using RRTYPE 99 instead of using RRTYPE 16. This plan + failed to take effect for four primary reasons: + + 1. there was hesitation to make the transition because existing + nameservers (and, in fact, DNS-aware firewalls) would drop or + reject requests for unknown RRTYPEs (see Section 3 for evidence + of this), which means successful rollout of a new RRTYPE is + contingent upon widespread adoption of updated nameservers and + resolver functions; + + 2. many DNS provisioning tools (e.g., web interfaces to controlling + DNS zone data) were, and still are, typically lethargic about + adding support for new RRTYPEs; + + 3. the substantial deployed base was already using RRTYPE 16, and it + was working just fine, leading to inertia; + + 4. [SPF] itself included a faulty transition plan, likely because of + the late addition of a requirement to develop one -- it said: + + An SPF-compliant domain name SHOULD have SPF records of both RR + types. A compliant domain name MUST have a record of at least + one type. + + which means both can claim to be fully compliant while failing + utterly to interoperate. Publication occurred without proper + IETF review, so this was not detected prior to publication. + + + + + +Kucherawy Informational [Page 10] + +RFC 6686 SPF/Sender ID Experiments July 2012 + + + It is likely that this will happen again if the bar to creating new + RRTYPEs even for experimental development purposes is not lowered, + and handling of unknown RRTYPEs in software becomes generally more + graceful. Also, important in this regard is encouragement of support + for new RRTYPEs in DNS record provisioning tools. + + Fortunately, in the meantime, the requirements for new RRTYPE + assignments was changed to be less stringent (see [DNS-IANA]). Also, + the publication of [DNS-EXPAND] has provided some useful guidance in + this regard. However, there is still a common perception that adding + new types of data to the DNS will face resistance due to the lack of + appropriate software support. + + There are DNS experts within the community that will undoubtedly + point to DNS servers and firewalls that mistreat queries for unknown + RRTYPEs, and to overly simplistic provisioning tools, and claim they + are broken as a way of answering these concerns. This is undoubtedly + correct, but the reality is that they are among us and likely will be + for some time, and this needs to be considered as new protocols and + IETF procedures are developed. + +Appendix B. Acknowledgments + + The following provided operational data that contributed to the + evidence presented above: + + Cisco: contributed data about observed Sender ID and SPF records in + the DNS for a large number of domains (DNS survey #1) + + Hotmail: contributed data about the difference between + RFC5321.MailFrom and RFC5322.From domains across large mail + volumes, and a survey of DNS replies observed in response to + incoming mail traffic (DNS survey #3) + + John Levine: conducted a survey of DNS server logs to evaluate SPF- + related query traffic + + McAfee: provided details about their SUBMITTER implementation and + usage statistics + + Santronics: contributed data about the use of the SUBMITTER + extension in aggregate SMTP client traffic + + The Trusted Domain Project: contributed data about the difference + between Sender ID and SPF results, conducted one of the detailed + TXT/SPF RRTYPE surveys including collecting timing data (DNS + survey #2), and conducted the MTA SUBMITTER survey + + + + +Kucherawy Informational [Page 11] + +RFC 6686 SPF/Sender ID Experiments July 2012 + + + The author would also like to thank the following for their + contributions to the development of the text in this document: Dave + Crocker, Scott Kitterman, Barry Leiba, John Leslie, John Levine, + Hector Santos, and Alessandro Vesely. + +Author's Address + + Murray S. Kucherawy + Cloudmark + 128 King St., 2nd Floor + San Francisco, CA 94107 + USA + + Phone: +1 415 946 3800 + EMail: superuser@gmail.com + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Kucherawy Informational [Page 12] + -- cgit v1.2.3