summaryrefslogtreecommitdiff
path: root/doc/rfc/rfc6686.txt
diff options
context:
space:
mode:
Diffstat (limited to 'doc/rfc/rfc6686.txt')
-rw-r--r--doc/rfc/rfc6686.txt675
1 files changed, 675 insertions, 0 deletions
diff --git a/doc/rfc/rfc6686.txt b/doc/rfc/rfc6686.txt
new file mode 100644
index 0000000..9e47d7c
--- /dev/null
+++ b/doc/rfc/rfc6686.txt
@@ -0,0 +1,675 @@
+
+
+
+
+
+
+Internet Engineering Task Force (IETF) M. Kucherawy
+Request for Comments: 6686 Cloudmark
+Category: Informational July 2012
+ISSN: 2070-1721
+
+
+ Resolution of the Sender Policy Framework (SPF)
+ and Sender ID Experiments
+
+Abstract
+
+ In 2006, the IETF published a suite of protocol documents comprising
+ the Sender Policy Framework (SPF) and Sender ID: two proposed email
+ authentication protocols. Both of these protocols enable one to
+ publish, via the Domain Name System, a policy declaring which mail
+ servers were authorized to send email on behalf of the domain name
+ being queried. There was concern that the two would conflict in some
+ significant operational situations, interfering with message
+ delivery.
+
+ The IESG required all of these documents (RFC 4405, RFC 4406, RFC
+ 4407, and RFC 4408) to be published as Experimental RFCs and
+ requested that the community observe deployment and operation of the
+ protocols over a period of two years from the date of publication to
+ determine a reasonable path forward.
+
+ After six years, sufficient experience and evidence have been
+ collected that the experiments thus created can be considered
+ concluded. This document presents those findings.
+
+Status of This Memo
+
+ This document is not an Internet Standards Track specification; it is
+ published for informational purposes.
+
+ This document is a product of the Internet Engineering Task Force
+ (IETF). It represents the consensus of the IETF community. It has
+ received public review and has been approved for publication by the
+ Internet Engineering Steering Group (IESG). Not all documents
+ approved by the IESG are a candidate for any level of Internet
+ Standard; see Section 2 of RFC 5741.
+
+ Information about the current status of this document, any errata,
+ and how to provide feedback on it may be obtained at
+ http://www.rfc-editor.org/info/rfc6686.
+
+
+
+
+
+
+Kucherawy Informational [Page 1]
+
+RFC 6686 SPF/Sender ID Experiments July 2012
+
+
+Copyright Notice
+
+ Copyright (c) 2012 IETF Trust and the persons identified as the
+ document authors. All rights reserved.
+
+ This document is subject to BCP 78 and the IETF Trust's Legal
+ Provisions Relating to IETF Documents
+ (http://trustee.ietf.org/license-info) in effect on the date of
+ publication of this document. Please review these documents
+ carefully, as they describe your rights and restrictions with respect
+ to this document. Code Components extracted from this document must
+ include Simplified BSD License text as described in Section 4.e of
+ the Trust Legal Provisions and are provided without warranty as
+ described in the Simplified BSD License.
+
+Table of Contents
+
+ 1. Introduction ....................................................2
+ 2. Definitions .....................................................3
+ 3. Evidence of Deployment ..........................................3
+ 3.1. DNS Resource Record Types ..................................3
+ 3.2. Implementations ............................................5
+ 3.3. The SUBMITTER SMTP Extension ...............................6
+ 4. Evidence of Differences .........................................7
+ 5. Analysis ........................................................7
+ 6. Conclusions .....................................................8
+ 7. Security Considerations .........................................9
+ 8. References ......................................................9
+ 8.1. Normative References .......................................9
+ 8.2. Informative References .....................................9
+ Appendix A. Background on the RRTYPE Issue ........................10
+ Appendix B. Acknowledgments .......................................11
+
+1. Introduction
+
+ In April 2006, the IETF published the [SPF] and Sender ID email
+ authentication protocols, the latter consisting of three documents
+ ([SUBMITTER], [SENDER-ID], and [PRA]). Both of these protocols
+ enable one to publish, via the Domain Name System, a policy declaring
+ which mail servers are authorized to send email on behalf of the
+ selected domain name.
+
+ Consensus did not clearly support one protocol over the other, and
+ there was significant concern that the two would conflict in some
+ significant operational situations, interfering with message
+ delivery. The IESG required the publication of all of these
+ documents as Experimental, and requested that the community observe
+
+
+
+
+Kucherawy Informational [Page 2]
+
+RFC 6686 SPF/Sender ID Experiments July 2012
+
+
+ deployment and operation of the protocols over a period of two years
+ from the date of publication in order to determine a reasonable path
+ forward.
+
+ In line with the IESG's request to evaluate after a period of time,
+ this document concludes the experiments by presenting evidence
+ regarding both deployment and comparative effect of the two
+ protocols. At the end, it presents conclusions based on the data
+ collected.
+
+ It is important to note that this document makes no direct technical
+ comparison of the two protocols in terms of correctness, weaknesses,
+ or use case coverage. The email community at large has already done
+ that through its deployment choices. Rather, the analysis presented
+ here is merely an observation of what has been deployed and supported
+ in the time since the protocols were published and lists conclusions
+ based on those observations.
+
+ The data collected and presented here are presumed to be a reasonable
+ representative view of the global deployment data, which could never
+ itself be fully surveyed within a reasonable period of time.
+
+2. Definitions
+
+ The term "RRTYPE" is used to refer to a Domain Name System ([DNS])
+ Resource Record (RR) type. These are always expressed internally in
+ software as numbers, assigned according to the procedures in
+ [DNS-IANA] Assigned RRTYPEs also have names. The two of interest in
+ this work are the TXT RRTYPE (16) and the SPF RRTYPE (99).
+
+3. Evidence of Deployment
+
+ This section presents the collected research done to determine what
+ parts of the two protocol suites are in general use as well as
+ related issues like [DNS] support.
+
+3.1. DNS Resource Record Types
+
+ Three large-scale DNS surveys were run that looked for the two
+ supported kinds of RRTYPEs that can contain SPF policy statements.
+ These surveys selected substantial sets of distinct domain names from
+ email headers and logs over long periods, regardless of whether the
+ DNS data for those domains included A, MX, or any other RRTYPEs. The
+ nameservers for these domains were queried, asking for both of the
+ RRTYPEs that could be used for SPF and/or Sender ID.
+
+
+
+
+
+
+Kucherawy Informational [Page 3]
+
+RFC 6686 SPF/Sender ID Experiments July 2012
+
+
+ In the tables below, replies were counted only if they included
+ prefixes that indicated the record was intended to be of a form
+ defined in either [SPF] or [SENDER-ID], though complete syntax
+ validation of the replies was not done. That is, the records started
+ either "v=spf1" or "spf2.0/", or they were not counted as replies.
+
+ The tables are broken down into three parts: (a) the size of the
+ sample set, (b) a report about RRTYPE use independent of content, and
+ (c) a report about content independent of RRTYPE.
+
+ "SPF+TXT" indicates the count of domains where both types were in
+ use.
+
+ DNS Survey #1 (Cisco)
+
+ +------------------+-----------+-------+
+ | Domains queried | 1,000,000 | - |
+ +------------------+-----------+-------+
+ | TXT replies | 397,511 | 39.8% |
+ | SPF replies | 6,627 | <1.0% |
+ | SPF+TXT replies | 6,603 | <1.0% |
+ +------------------+-----------+-------+
+ | v=spf1 replies | 395,659 | 39.6% |
+ | spf2.0/* replies | 5,291 | <1.0% |
+ +------------------+-----------+-------+
+
+ Domains were selected as the top million domains as reported by
+ Alexa, which monitors browser activity.
+
+ DNS Survey #2 (The Trusted Domain Project)
+
+ +------------------+-----------+-------+
+ | Domains queried | 278,353 | - |
+ +------------------+-----------+-------+
+ | TXT replies | 156,894 | 56.4% |
+ | SPF replies | 2,876 | 1.0% |
+ | SPF+TXT replies | 2,689 | <1.0% |
+ +------------------+-----------+-------+
+ | v=spf1 replies | 149,985 | 53.9% |
+ | spf2.0/* replies | 7,285 | 2.7% |
+ +------------------+-----------+-------+
+
+ This survey selected its domains from data observed in email headers
+ and previous SPF and Sender ID evaluations, collected from 23
+ reporting hosts across a handful of unrelated operators over a period
+ of 22 months.
+
+
+
+
+
+Kucherawy Informational [Page 4]
+
+RFC 6686 SPF/Sender ID Experiments July 2012
+
+
+ During this second survey, some domains were observed to provide
+ immediate answers for RRTYPE 16 queries, but would time out waiting
+ for replies to RRTYPE 99 queries. For example, it was observed that
+ 4,360 (over 1.6%) distinct domains in the survey returned a result of
+ some kind (a record or an error) for the TXT query in time N, while
+ the SPF query ultimately failed after at least time 4N.
+
+ DNS Survey #3 (Hotmail)
+
+ +------------------+-----------+-------+
+ | Domains queried | 100,000 | - |
+ +------------------+-----------+-------+
+ | TXT replies | 46,221 | 46.2% |
+ | SPF replies | 954 | <1.0% |
+ | SPF+TXT replies | 1,383 | 1.4% |
+ +------------------+-----------+-------+
+
+ Hotmail's domain set was selected from live email traffic at the time
+ the sample was extracted. Only the RRTYPE portion of the report is
+ available.
+
+ A separate survey was done of queries for RRTYPE 16 and RRTYPE 99
+ records by observing nameserver traffic records. Only a few queries
+ were ever received for RRTYPE 99 records, and those almost
+ exclusively came from one large email service provider that queried
+ for both RRTYPEs. The vast majority of other querying agents only
+ ever requested RRTYPE 16.
+
+3.2. Implementations
+
+ It is likely impossible to determine from a survey which Mail
+ Transfer Agents (MTAs) have SPF and/or Sender ID checking enabled at
+ message ingress since it does not appear, for example, in the reply
+ to the EHLO command from extended [SMTP]. Therefore, we relied on
+ evidence found via web searches and observed the following:
+
+ o A web site [SID-IMPL] dedicated to highlighting Sender ID
+ implementations, last updated in late 2007, listed 13 commercial
+ implementations, which we assume means they implement the
+ Purported Responsible Address (PRA) checks. At least one of them
+ is known no longer to be supported by its vendor. There were no
+ free open-source implementations listed.
+
+ o The [OPENSPF] web site maintains a list of implementations of SPF.
+ At the time of this document's writing, it listed six libraries,
+ 22 MTAs with built-in SPF implementations, and numerous patches
+ for MTAs and mail clients. The set included a mix of commercial
+ and free open-source implementations.
+
+
+
+Kucherawy Informational [Page 5]
+
+RFC 6686 SPF/Sender ID Experiments July 2012
+
+
+3.3. The SUBMITTER SMTP Extension
+
+ The PRA is the output of a heuristic that seeks to scan a message
+ header and extract from it the email address most likely to be the
+ one responsible for injection of that message into the mail stream.
+ The SUBMITTER extension to SMTP is a mechanism to provide an early
+ hint (i.e., as part of the MAIL command in an SMTP session) to the
+ receiving MTA of what the PRA would be on full receipt of the
+ message.
+
+ In a review of numerous MTAs in current or recent use, two
+ (Santronics WinServer and McAfee MxLogic) were found to contain
+ implementations of the SMTP SUBMITTER extension as part of the MTA
+ service, which could act as an enabler to Sender ID.
+
+ An unknown number of SMTP clients implement the SUBMITTER SMTP
+ extension. Although information from MTA logs indicates substantial
+ use of the SMTP extension, it is not possible to determine whether
+ the usage is from multiple instances of the same SMTP client or
+ different SMTP client implementations.
+
+ An active survey of MTAs accessible over the Internet was performed.
+ The MTAs selected were found by querying for MX and A resource
+ records of a subset of all domains observed by The Trusted Domain
+ Project's data collection system in the preceding 20 months. The
+ results were as follows:
+
+ SUBMITTER Survey (The Trusted Domain Project)
+
+ +-------------------+-----------+-------+
+ | MTAs selected | 484,980 | - |
+ | MTAs responding | 371,779 | 76.7% |
+ | SUBMITTER enabled | 17,425 | 4.7% |
+ | MXLogic banner | 16,914 | 4.6% |
+ +-------------------+-----------+-------+
+
+ Note: The bottom two rows indicate the percentage of responding MTAs
+ with the stated property, not the percentage of selected MTAs.
+
+ Based on the SMTP banner presented upon connection, the entire set of
+ SUBMITTER-enabled MTAs consisted of the two found during the review
+ (above) and a third whose identity could not be positively
+ determined.
+
+ Of those few responding MTAs advertising the SUBMITTER SMTP
+ extension, 97% were different instances of one MTA. The service
+ operating that MTA (MXLogic, a division of McAfee) reported that
+
+
+
+
+Kucherawy Informational [Page 6]
+
+RFC 6686 SPF/Sender ID Experiments July 2012
+
+
+ about 11% of all observed SMTP sessions involved SMTP clients that
+ make use of the SUBMITTER extension. Note that this represents about
+ 11% of the clients of 4.6% of the responding MTAs in the survey.
+
+4. Evidence of Differences
+
+ Separate surveys from Hotmail and The Trusted Domain Project compared
+ the cases where the PRA (used by Sender ID) and the RFC5321.MailFrom
+ address (used by SPF) differed. The results of these tests showed
+ that, at least 50% of the time, the two addresses were the same, but,
+ beyond that, the percentage varied substantially from one sampling
+ location to the next due to the nature of the mail streams they each
+ receive.
+
+ Further, The Trusted Domain Project analyzed approximately 150,000
+ messages and found that in more than 95% of those cases, Sender ID
+ and SPF reach the same conclusion about a message, meaning either
+ both protocols return a "pass" result or both return a "fail" result.
+ Note that this does not include an evaluation of whether "fail" meant
+ spam or other abusive mail was thus detected or that "pass" mail is
+ good mail; it is merely a measure of how often the two protocols
+ concurred. The data set yielding this response could not further
+ characterize the cases in which the answers differed.
+
+ A second analysis of the same nature by Hotmail found that the two
+ protocols yielded the same result approximately 80% of the time when
+ evaluated across billions of messages.
+
+ Anecdotally, the differences in conclusions have not been noted as
+ causing significant operational problems by the email-receiving
+ community.
+
+5. Analysis
+
+ Given the six years that have passed since the publication of the
+ Experimental RFCs, and the evidence reported in the earlier sections
+ of this document, the following analysis appears to be supported:
+
+ 1. There has not been substantial adoption of the RRTYPE 99 (SPF)
+ DNS resource record. In all large-scale surveys performed for
+ this work, fewer than 2% of responding domains published RRTYPE
+ 99 records, and almost no clients requested them.
+
+ 2. Of the DNS resource records retrieved, fewer than 3% included
+ specific requests for processing of messages using the PRA
+ algorithm, which is an essential part of Sender ID.
+
+
+
+
+
+Kucherawy Informational [Page 7]
+
+RFC 6686 SPF/Sender ID Experiments July 2012
+
+
+ 3. Although the two protocols often used different email address
+ fields as the subject being evaluated, no data collected showed
+ any substantial operational benefit, in terms of improved
+ accuracy, to using one mechanism over the other.
+
+ 4. A review of known implementations shows significant support for
+ both protocols, though there were more implementations in support
+ of SPF than of Sender ID. Further, the SPF implementations
+ showed better upkeep and current interest than the Sender ID
+ implementations.
+
+ 5. A survey of running MTAs shows fewer than 5% of them advertised
+ the SUBMITTER extension, which is a Sender ID enabler. Only
+ three implementations of it were found.
+
+ 6. There remain obstacles to deployment of protocols that use DNS
+ RRTYPEs other than the most common ones, including firewalls and
+ DNS servers that block or discard requests for unknown RRTYPEs.
+ Further, few if any web-based DNS configuration tools offer
+ support for RRTYPE 99 records.
+
+6. Conclusions
+
+ In light of the analysis in the previous section, the following
+ conclusions are supported:
+
+ 1. The experiments comprising the series of RFCs defining the
+ SUBMITTER SMTP extension (RFC4405), the Sender ID mechanism
+ (RFC4406), the Purported Responsible Address algorithm (RFC4407),
+ and SPF (RFC4408), should be considered concluded.
+
+ 2. The absence of significant adoption of the RRTYPE 99 DNS Resource
+ Record suggests that it has not attracted enough support to be
+ useful.
+
+ 3. Unavailability of software implementing the protocols was not a
+ gating factor in terms of the selection of which to use.
+
+ 4. The absence of significant adoption of the [SUBMITTER] extension,
+ [SENDER-ID], and [PRA], indicates that there is not a strong
+ community deploying and using these protocols.
+
+ 5. [SPF] has widespread implementation and deployment, comparable to
+ that of many Standards Track protocols.
+
+ Appendix A is offered as a cautionary review of problems that
+ affected the process of developing SPF and Sender ID in terms of
+ their use of the DNS.
+
+
+
+Kucherawy Informational [Page 8]
+
+RFC 6686 SPF/Sender ID Experiments July 2012
+
+
+7. Security Considerations
+
+ This document contains information for the community, akin to an
+ implementation report, and does not introduce any new security
+ concerns.
+
+8. References
+
+8.1. Normative References
+
+ [DNS] Mockapetris, P., "Domain names - implementation and
+ specification", STD 13, RFC 1035, November 1987.
+
+ [PRA] Lyon, J., "Purported Responsible Address in E-Mail
+ Messages", RFC 4407, April 2006.
+
+ [SENDER-ID] Lyon, J. and M. Wong, "Sender ID: Authenticating
+ E-Mail", RFC 4406, April 2006.
+
+ [SPF] Wong, M. and W. Schlitt, "Sender Policy Framework (SPF)
+ for Authorizing Use of Domains in E-Mail, Version 1",
+ RFC 4408, April 2006.
+
+ [SUBMITTER] Allman, E. and H. Katz, "SMTP Service Extension for
+ Indicating the Responsible Submitter of an E-Mail
+ Message", RFC 4405, April 2006.
+
+8.2. Informative References
+
+ [DNS-EXPAND] IAB, Faltstrom, P., Austein, R., and P. Koch, "Design
+ Choices When Expanding the DNS", RFC 5507, April 2009.
+
+ [DNS-IANA] Eastlake 3rd, D., "Domain Name System (DNS) IANA
+ Considerations", BCP 42, RFC 6195, March 2011.
+
+ [OPENSPF] "Sender Policy Framework: Project Overview",
+ <http://www.openspf.net>.
+
+ [SID-IMPL] "Sender ID Framework Industry Support and Solutions",
+ October 2007, <http://www.microsoft.com/mscorp/safety/
+ technologies/senderid/support.mspx>.
+
+ [SMTP] Klensin, J., "Simple Mail Transfer Protocol", RFC 5321,
+ October 2008.
+
+
+
+
+
+
+
+Kucherawy Informational [Page 9]
+
+RFC 6686 SPF/Sender ID Experiments July 2012
+
+
+Appendix A. Background on the RRTYPE Issue
+
+ SPF was originally created by a community of interested developers
+ outside the IETF, with the intent of bringing it to the IETF for
+ standardization after it had become relatively mature and ready for
+ the IETF Standards process.
+
+ At the time of SPF's initial development, the prospect of getting an
+ RRTYPE allocated for SPF was not seriously considered, partly because
+ doing so had high barriers to entry. As a result, at the time it was
+ brought to the IETF for development and publication, there was
+ already a substantial and growing installed base that had SPF running
+ using TXT RRs. Eventually, the application was made for the new
+ RRTYPE as a result of pressure from the DNS experts in the community,
+ who insisted upon doing so as the preferred path toward using the DNS
+ for storing such things as policy data.
+
+ Later, after RRTYPE 99 was assigned (long after IESG approval of
+ [SPF], in fact), a plan was put into place to effect a gradual
+ transition to using RRTYPE 99 instead of using RRTYPE 16. This plan
+ failed to take effect for four primary reasons:
+
+ 1. there was hesitation to make the transition because existing
+ nameservers (and, in fact, DNS-aware firewalls) would drop or
+ reject requests for unknown RRTYPEs (see Section 3 for evidence
+ of this), which means successful rollout of a new RRTYPE is
+ contingent upon widespread adoption of updated nameservers and
+ resolver functions;
+
+ 2. many DNS provisioning tools (e.g., web interfaces to controlling
+ DNS zone data) were, and still are, typically lethargic about
+ adding support for new RRTYPEs;
+
+ 3. the substantial deployed base was already using RRTYPE 16, and it
+ was working just fine, leading to inertia;
+
+ 4. [SPF] itself included a faulty transition plan, likely because of
+ the late addition of a requirement to develop one -- it said:
+
+ An SPF-compliant domain name SHOULD have SPF records of both RR
+ types. A compliant domain name MUST have a record of at least
+ one type.
+
+ which means both can claim to be fully compliant while failing
+ utterly to interoperate. Publication occurred without proper
+ IETF review, so this was not detected prior to publication.
+
+
+
+
+
+Kucherawy Informational [Page 10]
+
+RFC 6686 SPF/Sender ID Experiments July 2012
+
+
+ It is likely that this will happen again if the bar to creating new
+ RRTYPEs even for experimental development purposes is not lowered,
+ and handling of unknown RRTYPEs in software becomes generally more
+ graceful. Also, important in this regard is encouragement of support
+ for new RRTYPEs in DNS record provisioning tools.
+
+ Fortunately, in the meantime, the requirements for new RRTYPE
+ assignments was changed to be less stringent (see [DNS-IANA]). Also,
+ the publication of [DNS-EXPAND] has provided some useful guidance in
+ this regard. However, there is still a common perception that adding
+ new types of data to the DNS will face resistance due to the lack of
+ appropriate software support.
+
+ There are DNS experts within the community that will undoubtedly
+ point to DNS servers and firewalls that mistreat queries for unknown
+ RRTYPEs, and to overly simplistic provisioning tools, and claim they
+ are broken as a way of answering these concerns. This is undoubtedly
+ correct, but the reality is that they are among us and likely will be
+ for some time, and this needs to be considered as new protocols and
+ IETF procedures are developed.
+
+Appendix B. Acknowledgments
+
+ The following provided operational data that contributed to the
+ evidence presented above:
+
+ Cisco: contributed data about observed Sender ID and SPF records in
+ the DNS for a large number of domains (DNS survey #1)
+
+ Hotmail: contributed data about the difference between
+ RFC5321.MailFrom and RFC5322.From domains across large mail
+ volumes, and a survey of DNS replies observed in response to
+ incoming mail traffic (DNS survey #3)
+
+ John Levine: conducted a survey of DNS server logs to evaluate SPF-
+ related query traffic
+
+ McAfee: provided details about their SUBMITTER implementation and
+ usage statistics
+
+ Santronics: contributed data about the use of the SUBMITTER
+ extension in aggregate SMTP client traffic
+
+ The Trusted Domain Project: contributed data about the difference
+ between Sender ID and SPF results, conducted one of the detailed
+ TXT/SPF RRTYPE surveys including collecting timing data (DNS
+ survey #2), and conducted the MTA SUBMITTER survey
+
+
+
+
+Kucherawy Informational [Page 11]
+
+RFC 6686 SPF/Sender ID Experiments July 2012
+
+
+ The author would also like to thank the following for their
+ contributions to the development of the text in this document: Dave
+ Crocker, Scott Kitterman, Barry Leiba, John Leslie, John Levine,
+ Hector Santos, and Alessandro Vesely.
+
+Author's Address
+
+ Murray S. Kucherawy
+ Cloudmark
+ 128 King St., 2nd Floor
+ San Francisco, CA 94107
+ USA
+
+ Phone: +1 415 946 3800
+ EMail: superuser@gmail.com
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Kucherawy Informational [Page 12]
+