diff options
Diffstat (limited to 'doc/rfc/rfc8963.txt')
| -rw-r--r-- | doc/rfc/rfc8963.txt | 2188 |
1 files changed, 2188 insertions, 0 deletions
diff --git a/doc/rfc/rfc8963.txt b/doc/rfc/rfc8963.txt new file mode 100644 index 0000000..a6250cb --- /dev/null +++ b/doc/rfc/rfc8963.txt @@ -0,0 +1,2188 @@ + + + + +Independent Submission C. Huitema +Request for Comments: 8963 Private Octopus Inc. +Category: Informational January 2021 +ISSN: 2070-1721 + + + Evaluation of a Sample of RFCs Produced in 2018 + +Abstract + + This document presents the author's effort to understand the delays + involved in publishing an idea in the IETF or through the Independent + Stream, from the first individual draft to the publication of the + RFC. We analyze a set of randomly chosen RFCs approved in 2018, + looking for history and delays. We also use two randomly chosen sets + of RFCs published in 2008 and 1998 for comparing delays seen in 2018 + to those observed 10 or 20 years ago. The average RFC in the 2018 + sample was produced in 3 years and 4 months, of which 2 years and 10 + months were spent in the working group, 3 to 4 months for IETF + consensus and IESG review, and 3 to 4 months in RFC production. The + main variation in RFC production delays comes from the AUTH48 phase. + + We also measure the number of citations of the chosen RFC using + Semantic Scholar, and compare citation counts with what we know about + deployment. We show that citation counts indicate academic interest, + but correlate only loosely with deployment or usage of the + specifications. Counting web references could complement that. + +Status of This Memo + + This document is not an Internet Standards Track specification; it is + published for informational purposes. + + This is a contribution to the RFC Series, independently of any other + RFC stream. The RFC Editor has chosen to publish this document at + its discretion and makes no statement about its value for + implementation or deployment. Documents approved for publication by + the RFC Editor are not candidates for any level of Internet Standard; + see Section 2 of RFC 7841. + + Information about the current status of this document, any errata, + and how to provide feedback on it may be obtained at + https://www.rfc-editor.org/info/rfc8963. + +Copyright Notice + + Copyright (c) 2021 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (https://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. + +Table of Contents + + 1. Introduction + 2. Methodology + 2.1. Defining the Important Milestones + 2.2. Selecting a Random Sample of RFCs + 2.3. Conventions Used in This Document + 3. Analysis of 20 Selected RFCs + 3.1. RFC 8411 + 3.2. RFC 8456 + 3.3. RFC 8446 + 3.4. RFC 8355 + 3.5. RFC 8441 + 3.6. RFC 8324 + 3.7. RFC 8377 + 3.8. RFC 8498 + 3.9. RFC 8479 + 3.10. RFC 8453 + 3.11. RFC 8429 + 3.12. RFC 8312 + 3.13. RFC 8492 + 3.14. RFC 8378 + 3.15. RFC 8361 + 3.16. RFC 8472 + 3.17. RFC 8471 + 3.18. RFC 8466 + 3.19. RFC 8362 + 3.20. RFC 8468 + 4. Analysis of Process and Delays + 4.1. Delays from First Draft to RFC + 4.2. Working Group Processing Time + 4.3. Preparation and Publication Delays + 4.4. Copy Editing + 4.5. Independent Stream + 5. Citation Counts + 5.1. Citation Numbers + 5.2. Comparison to 1998 and 2008 + 5.3. Citations versus Deployments + 5.4. Citations versus Web References + 6. Observations and Next Steps + 7. Security Considerations + 8. IANA Considerations + 9. Informative References + Acknowledgements + Author's Address + +1. Introduction + + As stated on the organization's web site, "The IETF is a large open + international community of network designers, operators, vendors, and + researchers concerned with the evolution of the Internet architecture + and the smooth operation of the Internet." The specifications + produced by the IETF are published in the RFC series, along with + documents from the IAB, IRTF, and Independent streams (as per RFC + 8729). In this memo, the author attempts to understand the delays + involved in publishing an idea in the IETF or through the Independent + Stream, from the first individual draft to the publication of the + RFC. This is an individual effort, and the author's conclusions + presented here are personal. There was no attempt to seek IETF + consensus. + + The IETF keeps records of documents and process actions in the IETF + Datatracker [TRKR]. The IETF Datatracker provides information about + RFCs and drafts, from which we can infer statistics about the + production system. We can measure how long it takes to drive a + proposition from initial draft to final publication, and how these + delays can be split between working group discussions, IETF reviews, + IESG assessment, RFC Editor delays and final reviews by the authors + -- or, for Independent Stream RFCs, draft production, reviews by the + Independent Submissions Editor, conflict reviews, RFC Editor delays + and final reviews. Tracker data is available for all RFCs, not just + IETF Stream RFCs. + + Just measuring production delays may be misleading. If the IETF or + the other streams simply rubber-stamped draft proposals and published + them, the delays would be short but the quality and impact might + suffer. We hope that most of the RFCs that are published are useful, + but we need a way to measure that usefulness. We try to do that by + measuring the number of references of the published RFCs in Semantic + Scholar [SSCH], and also by asking the authors of each RFC in the + sample whether the protocols and technologies defined in the RFCs + were implemented and used on the Internet. The citations measured by + the Semantic Scholar include citations in other RFCs and in Internet- + Drafts. We also measure the number of references on the web, which + provides some results but would be hard to automate. + + In order to limit the resources required for this study, we selected + at random 20 RFCs published in 2018, as explained in Section 2.2. + The statistical sampling picked both IETF Stream and Independent + Stream documents. For comparison purposes, we also selected at + random 20 RFCs published in 1998 and 20 published in 2008. Limiting + the sample to 20 out of 209 RFCs published in 2018 allows for in- + depth analysis of each RFC, but readers should be reminded that the + this is a small sample. The sample is too small to apply general + statistical techniques and quantify specific ratios, and discussions + of correlation techniques would be inappropriate. Instead, the + purpose is to identify trends, spot issues, and document future work. + + The information gathered for every RFC in the sample is presented in + Section 3. In Section 4, we analyze the production process and the + sources of delays, comparing the 2018 sample to the selected samples + for 1998 and 2018. In Section 5.1, we present citation counts for + the RFCs in the samples, and analyze whether citation counts could be + used to evaluate the quality of RFCs. + + The measurement of delays could be automated by processing dates and + events recorded in the Datatracker. The measurement of published + RFCs could be complemented by statistics on abandoned drafts, which + would measure the efficiency of the IETF triaging process. More + instrumentation would help understanding how large delays happen + during working group processes. These potential next steps are + developed in Section 6. + +2. Methodology + + The study reported here started with a simple idea: take a sample of + RFCs, and perform an in-depth analysis of the path from the first + presentation of the idea to its publication, while also trying to + access the success of the resulting specification. This requires + defining the key milestones that we want to track, and drawing a + random sample using an unbiased process. + +2.1. Defining the Important Milestones + + The IETF Datatracker records a list of events for each document + processed by IETF working groups. This has a high granularity, and + also a high variability. Most documents start life as an individual + draft, are adopted by a working group, undergo a Working Group Last + Call, are submitted to the IESG, undergo an IETF Last Call and an + IESG review, get eventually approved by the IESG, and are processed + for publication by the RFC Editor, but there are exceptions. Some + documents are first submitted to one working group and then moved to + another. Some documents are published through the Independent + Stream, and are submitted to the Independent Submissions Editor + instead of the IESG. + + In order to simplify tabulation, we break the period from the + submission of the first draft to the publication of the RFC into + three big components: + + * The working group processing time, from the first draft to the + start of the IETF last call; + + * The IETF processing time, which lasts from the beginning of the + IETF last call to the approval by the IESG, including the reviews + by various directorates; + + * The RFC production, from approval by the IESG to publication, + including the AUTH48 reviews. + + For submissions to the Independent Stream, we don't have a working + group. We consider instead the progression of the individual draft + until the adoption by the Independent Submissions Editor (ISE) as the + equivalent of the "Working Group" period, and the delay from adoption + by the ISE until submission to the RFC Editor as the equivalent of + the IETF processing time. + + We measure the starting point of the process using the date of + submission of the first draft listed on that RFC page in the IETF + Datatracker. In most cases, this first draft is an individual draft + that then resubmitted as a working group draft, or maybe resubmitted + with a new name as the draft was searching for a home in an IETF + working group, or before deciding for submission on the Independent + Stream. + + The IETF Datatracker entries for RFCs and drafts do not _always_ list + working group events like Working Group Last Call. The only + intermediate event that we list between the first draft and the + submission to the IESG is the working group adoption, for which we + use the date of submission of version 00 of the draft eventually + published as RFC. We also use that date (of submission of version + 00) for drafts submitted to the Independent Stream. + +2.2. Selecting a Random Sample of RFCs + + Basic production mechanisms could be evaluated by processing data + from the IETF Datatracker, but subjective data requires manual + assessment of results, which can be time-consuming. Since our + resources are limited, we will only perform this analysis for a small + sample of RFCs, selected at random from the list of RFCs approved in + 2018. Specifically, we will pick 20 RFC numbers at random between: + + * RFC 8307, published in January 2018, and + + * RFC 8511, published December 2018. + + The list of 20 selected RFCs is: RFC 8411, RFC 8456, RFC 8446, RFC + 8355, RFC 8441, RFC 8324, RFC 8377, RFC 8498, RFC 8479, RFC 8453, RFC + 8429, RFC 8312, RFC 8492 , RFC 8378, RFC 8361, RFC 8472, RFC 8471, + RFC 8466, RFC 8362, and RFC 8468. + + When evaluating delays and impact, we will compare the year 2018 to + 2008 and 1998, 10 and 20 years ago. To drive this comparison, we + pick 20 RFCs at random among those published in 2008, and another 20 + among those published in 1998. + + The list of the 20 randomly selected RFCs from 2008 is: RFC 5227, RFC + 5174, RFC 5172, RFC 5354, RFC 5195, RFC 5236, RFC 5348, RFC 5281, RFC + 5186, RFC 5326, RFC 5277, RFC 5373, RFC 5404, RFC 5329, RFC 5283, RFC + 5358, RFC 5142, RFC 5271, RFC 5349, and RFC 5301. + + The list of the 20 randomly selected RFCs from 1998 is: RFC 2431, RFC + 2381, RFC 2387, RFC 2348, RFC 2391, RFC 2267, RFC 2312, RFC 2448, RFC + 2374, RFC 2398, RFC 2283, RFC 2382, RFC 2289, RFC 2282, RFC 2404, RFC + 2449, RFC 2317, RFC 2394, RFC 2297, and RFC 2323. + +2.3. Conventions Used in This Document + + The following abbreviations are used in the tables: + + BCP Best Current Practice + Exp Experimental + Info Informational + PS Proposed Standard + DS Draft Standard [This maturity level was retired by RFC 6410.] + + In addition, Status is as defined in RFC 2026, and Stream is as + defined in RFC 8729. + +3. Analysis of 20 Selected RFCs + + We review each of the RFCs listed in Section 2.2 for the year 2018, + trying both to answer the known questions and to gather insight for + further analyses. In many cases, the analysis of the data is + complemented by direct feedback from the RFC authors. + +3.1. RFC 8411 + + "IANA Registration for the Cryptographic Algorithm Object Identifier + Range" [RFC8411]: + + Status (Length): Informational (5 pages) + Overview: 4 individual drafts + First draft: 2017-05-08 + Last Call start: 2017-10-09 + IESG eval. start: 2017-12-28 + IESG approved: 2018-02-26 (draft 03) + AUTH48 start: 2018-04-20 + AUTH48 complete: 2018-07-17 + Published: 2018-08-06 + IANA action: create table + + This RFC was published from the individual draft, which was not + resubmitted as a working group draft. + + The draft underwent minor copy editing before publication. + + Some but not all of the long delay in AUTH48 is due to clustering + with [RFC8410]. MISSREF state concluded on 2018-05-09 and the + document re-entered AUTH48 at once. AUTH48 lasted over two months + after that. (For state definitions, see <https://www.rfc- + editor.org/about/queue/#state_def>.) + + The time after AUTH48 and before publication (3 weeks) partly + overlaps with travel for IETF 102 and is partly due to coordinating + the cluster. + +3.2. RFC 8456 + + "Benchmarking Methodology for Software-Defined Networking (SDN) + Controller Performance" [RFC8456]: + + Status (Length): Informational (64 pages) + Overview: 2 individual drafts; 9 WG drafts + First draft: 2015-03-23 + WG adoption: 2015-10-18 + Last Call start: 2018-01-19 + IESG eval. start: 2018-02-27 + IESG approved: 2018-05-25 + AUTH48 start: 2018-08-31 + AUTH48 complete: 2018-10-16 + Published: 2018-10-30 + + The draft underwent extensive copy editing, covering use of articles, + syntax, and word choice. The changes are enough to cause pagination + differences. The "diff" tool marks pretty much every page as + changed. Some diagrams see change in protocol elements like message + names. + + According to the author, the experience of producing this document + mirrors a typical one in the Benchmarking Methodologies Working Group + (BMWG). There were multiple authors in multiple time zones, which + slowed down the AUTH48 process somewhat, although the AUTH48 delay of + 46 days is only a bit longer than the average draft. + + The RFC was part of cluster with [RFC8455]. + + BMWG publishes Informational RFCs centered around benchmarking, and + the methodologies in RFC 8456 have been implemented in benchmarking + products. + +3.3. RFC 8446 + + "The Transport Layer Security (TLS) Protocol Version 1.3" [RFC8446], + as the title indicates, defines the new version of the TLS protocol. + From the IETF Datatracker, we extract the following: + + Status (Length): Proposed Standard (160 pages) + Overview: 29 WG drafts + First draft: 2014-04-17 + Last Call start: 2018-02-15 + IESG eval. start: 2018-03-02 + IESG approved: 2018-03-21 (draft 28) + AUTH48 start: 2018-06-14 + AUTH48 complete: 2018-08-10 + Published: 2018-08-10 + + This draft started as a WG effort. + + The RFC was a major effort in the IETF. Working group participants + developed and tested several implementations. Researchers analyzed + the specifications and performed formal verifications. Deployment + tests outlined issues that caused extra work when the specification + was almost ready. This complexity largely explains the time spent in + the working group. + + Comparing the final draft to the published version, we find + relatively light copy editing. It includes explaining acronyms on + first use, clarifying some definitions standardizing punctuation and + capitalization, and spelling out some numbers in text. This + generally fall in the category of "style", although some of the + clarifications go into message definitions. However, that simple + analysis does not explain why the AUTH48 phase took almost two + months. + + This document's AUTH48 process was part of the "GitHub experiment", + which tried to use GitHub pull requests to track the AUTH48 changes + and review comments. The RFC Production Center (RPC) staff had to + learn using GitHub for that process, and this required more work than + the usual RFC. The author and AD thoroughly reviewed each proposed + edit, accepting some and rejecting some. The concern there was that + any change in a complex specification might affect a protocol that + was extensively reviewed in the working group, but of course these + reviews added time to the AUTH48 delays. + + There are 21 implementations listed in the Wiki of the TLS 1.3 + project [TLS13IMP]. It has been deployed on major browsers, and is + already used in a large fraction of TLS connections. + +3.4. RFC 8355 + + "Resiliency Use Cases in Source Packet Routing in Networking (SPRING) + Networks" [RFC8355] is an Informational RFC. It originated from an + informational use-case draft; it was mostly used for the BOF creating + the WG, and then to drive initial work and evolutions from the WG. + + Status (Length): Informational (13 pages) + Overview: 2 individual drafts; 13 WG drafts + First draft: 2014-01-31 + WG adoption: 2014-05-13 + Last Call start: 2017-04-20 + IESG eval. start: 2017-05-04 (draft 09) + IESG approved: 2017-12-19 (draft 12) + AUTH48 start: 2018-03-12 + AUTH48 complete: 2018-03-27 + Published: 2018-03-28 + + Minor set of copy edits, mostly for style. + + No implementation of the RFC itself, but the technology behind it + (such as Segment Routing Architecture [RFC8402] and TI-LFA [TI-LFA]) + is widely implemented and deployment is ongoing. + + According to participants in the discussion, the process of adoption + of the source packet routing standards was very contentious. The + establishment of consensus at both the working group level and the + IETF level was difficult and time-consuming. + +3.5. RFC 8441 + + "Bootstrapping WebSockets with HTTP/2" [RFC8441] + + Status (Length): Proposed Standard (8 pages) + Overview: 3 individual drafts; 8 WG drafts; Updates RFC + 6455 + First draft: 2017-10-15 + WG adoption: 2017-12-19 + Last Call start: 2018-05-07 (draft 05) + IESG eval. start: 2018-05-29 (draft 06) + IESG approved: 2018-06-18 (draft 07) + AUTH48 start: 2018-08-13 + AUTH48 complete: 2018-09-15 + Published: 2018-09-18 + IANA action: table entries + + This RFC defines the support of WebSockets in HTTP/2, which is + different from the mechanism defined for HTTP/1.1 in [RFC6455]. The + process was relatively straightforward, involving the usual type of + discussions, some on details and some on important points. + + Comparing the final draft and published RFC shows a minor set of copy + edits, mostly for style. However, the author recalls a painful + process. The RFC includes many charts and graphs that were very + difficult to format correctly in the author's production process that + involved conversions from markdown to XML, and then from XML to text. + The author had to get substantial help from the RFC Editor. + + There are several implementations, including Firefox and Chrome, + making RFC 8441 a very successful specification. + +3.6. RFC 8324 + + "DNS Privacy, Authorization, Special Uses, Encoding, Characters, + Matching, and Root Structure: Time for Another Look?" [RFC8324]. + This is an opinion piece on DNS development, published on the + Independent Stream. + + Status (Length): Informational (29 pages) + Overview: 5 individual drafts; Independent Stream + First draft: 2017-06-02 + ISE review start: 2017-07-10 (draft 03) + IETF conflict review start: 2017-10-29 + Approved: 2017-12-18 (draft 04) + AUTH48 start: 2018-01-29 (draft 05) + AUTH48 complete: 2018-02-26 + Published: 2018-02-27 + + This RFC took only 9 months from first draft to publication, which is + the shortest in the 2018 sample set. In part, this is because the + text was privately circulated and reviewed by the ISE's selected + experts before the first draft was published. The nature of the + document is another reason for the short delay. It is an opinion + piece and does not require the same type of consensus building and + reviews as a protocol specification. + + Comparing the final draft and the published version shows only minor + copy edits, mostly for style. According to the author, this is + because he knows how to write in RFC style with the result that his + documents often need a minimum of editing. He also makes sure that + the document on which the RFC Production Center starts working + already has changes discussed and approved during Last Call and IESG + review incorporated, rather than expecting the Production Center to + operate off of notes about changes to be made. + +3.7. RFC 8377 + + "Transparent Interconnection of Lots of Links (TRILL): Multi- + Topology" [RFC8377] + + Status (Length): Proposed Standard (20 pages) + Overview: 3 individual drafts; 7 WG drafts; Updates RFCs + 6325 and 7177 + First draft: 2013-09-03 + WG adoption: 2015-09-01 + Last Call start: 2018-02-19 (draft 05) + IESG eval. start: 2018-03-06 (draft 05) + IESG approved: 2018-03-12 (draft 06) + AUTH48 start: 2018-04-20 (draft 06) + AUTH48 complete: 2018-07-31 + Published: 2018-07-31 + IANA action: table entries + + Minor set of copy edits, mostly for style, also clarity. + +3.8. RFC 8498 + + "A P-Served-User Header Field Parameter for an Originating Call + Diversion (CDIV) Session Case in the Session Initiation Protocol + (SIP)" [RFC8498]. + + Status (Length): Informational (15 pages) + Overview: 5 individual drafts; 9 WG drafts + First draft: 2016-03-21 + WG adoption: 2017-05-15 + Last Call start: 2018-10-12 (draft 05) + IESG eval. start: 2018-11-28 (draft 07) + IESG approved: 2018-12-11 (draft 08) + AUTH48 start: 2019-01-28 + AUTH48 complete: 2019-02-13 + Published: 2019-02-14 + IANA action: table rows added. + + Copy edits for style, but also clarification of ambiguous sentences. + +3.9. RFC 8479 + + "Storing Validation Parameters in PKCS#8" [RFC8479] + + Status (Length): Informational (8 pages) + Overview: 5 individual drafts; Independent Stream + First draft: 2017-08-08 + ISE review start: 2018-12-10 (draft 00) + IETF conflict review start: 2018-03-29 + Approved: 2018-08-20 (draft 03) + AUTH48 start: 2018-09-20 (draft 04) + AUTH48 complete: 2018-09-25 + Published: 2018-09-26 + + The goal of the draft was to document what the gnutls implementation + was using for storing provably generated RSA keys. This is a short + RFC that was published relatively quickly, although discussion + between the author, the Independent Submissions Editor, and the IESG + lasted several months. In the initial conflict review, the IESG + asked the ISE to not publish this document before IETF working groups + had an opportunity to pick up the work. The author met that + requirement by a presentation to the SECDISPATCH WG during IETF 102. + Since no WG was interested in picking up the work, the document + progressed on the Independent Stream. + + Very minor set of copy edits, moving some references from normative + to informative. + + The author is not aware of other implementations than gnutls relying + on this RFC. + +3.10. RFC 8453 + + "Framework for Abstraction and Control of TE Networks (ACTN)" + [RFC8453] + + Status (Length): Informational (42 pages) + Overview: 3 individual drafts; 16 WG drafts + First draft: 2015-06-15 + WG adoption: 2016-07-15 + Out of WG: 2018-01-26 (draft 11) + Expert review requested: 2018-02-13 + Last Call start: 2018-04-16 (draft 13) + IESG eval. start: 2018-05-16 (draft 14) + IESG approved: 2018-06-01 (draft 15) + AUTH48 start: 2018-08-13 + AUTH48 complete: 2018-08-20 + Published: 2018-08-23 + IANA action: table rows added. + + Minor copy editing. + +3.11. RFC 8429 + + "Deprecate Triple-DES (3DES) and RC4 in Kerberos" [RFC8429] + + Status (Length): BCP (10 pages) + Overview: 6 WG drafts + First draft: 2017-05-01 + Last Call start: 2017-07-16 (draft 03) + IESG eval. start: 2017-08-18 (draft 04) + IESG approved: 2018-05-25 (draft 05) + AUTH48 start: 2018-07-24 + AUTH48 complete: 2018-10-31 + Published: 2018-10-31 + IANA action: table rows added. + + This draft started as a working group effort. + + This RFC recommends deprecating two encryption algorithms that are + now considered obsolete and possibly broken. The document was sent + back to the WG after the first Last Call, edited, and then there was + a second Last Call. The delay from first draft to Working Group Last + Call was relatively short, but the number may be misleading. The + initial draft was a replacement of a similar draft in the KITTEN + Working Group, which stagnated for some time before the CURDLE + Working Group took up the work. The deprecation of RC4 was somewhat + contentious, but the WG had already debated this prior to the + production of this draft, and the draft was not delayed by this + debate. + + Most of the 280 days between IETF LC and IESG approval were because + the IESG had to talk about whether this document should obsolete RFC + 4757 or move it to Historic status, and no one was really actively + pushing that discussion for a while. + + The 99 days in AUTH48 are mostly because one of the authors was a + sitting AD, and those duties ended up taking precedence over + reviewing this document. + + Minor copy editing, for style. + + The implementation of the draft would be the actual removal of + support for 3DES and RC4 in major implementations. This is + happening, but very slowly. + +3.12. RFC 8312 + + "CUBIC for Fast Long-Distance Networks" [RFC8312] + + Status (Length): Informational (18 pages) + Overview: 2 individual drafts; 8 WG drafts + First draft: 2014-09-01 + WG adoption: 2015-06-08 + Last Call start: 2017-09-18 (draft 06) + IESG eval. start: 2017-10-04 + IESG approved: 2017-11-14 (draft 07) + AUTH48 start: 2018-01-08 + AUTH48 complete: 2018-02-07 + Published: 2018-02-07 + IANA action: table rows added. + + Minor copy editing, for style. + + The TCP congestion control algorithm Cubic was first defined in 2005, + was implemented in Linux soon after, and was implemented in major + OSes after that. After some debates from 2015 to 2015, the TCPM + Working Group adopted the draft, with a goal of documenting Cubic in + the RFC Series. According to the authors, this was not a high- + priority effort, as Cubic was already implemented in multiple OSes + and documented in research papers. At some point, only one of the + authors was actively working on the draft. This may explain why + another two years was spent progressing the draft after adoption by + the WG. + + The RFC publication may or may not have triggered further + implementations. On the other hand, several OSes picked up bug fixes + from the draft and the RFC. + +3.13. RFC 8492 + + "Secure Password Ciphersuites for Transport Layer Security (TLS)" + [RFC8492] + + Status (Length): Informational (40 pages) + Overview: 10 individual drafts; 8 WG drafts; Independent + Stream + First draft: 2012-09-07 + Targeted to ISE: 2016-08-05 + ISE review start: 2017-05-10 (draft 01) + IETF conflict review start: 2017-09-04 + Approved: 2017-10-29 (draft 02) + AUTH48 start: 2018-10-19 (draft 05) + AUTH48 complete: 2019-02-19 + Published: 2019-02-21 + IANA action: table rows added. + + This RFC has a complex history. The first individual draft was + submitted to the TLS Working Group on September 7, 2012. It + progressed there, and was adopted by the WG after 3 revisions. There + were then 8 revisions in the TLS WG, until the WG decided to not + progress it. The draft was parked in 2013 by the WG chairs after + failing to get consensus in WG Last Call. The AD finally pulled the + plug in 2016, and the draft was then resubmitted to the ISE. + + At that point, the author was busy and was treating this RFC with a + low priority because, in his words, it would not be a "real RFC". + There were problems with the draft that only came up late. In + particular, it had to wait for a change in registry policy that only + came about with the publication of TLS 1.3, which caused the draft to + be published after RFC 8446, and also required adding references to + TLS 1.3. The author also got a very late comment while in AUTH48 + that caused some rewriting. Finally, there was some IANA issue with + the extension registry where a similar extension was added by someone + else. The draft was changed to just use it. + + Changes in AUTH48 include adding a reference to TLS 1.3, copy editing + for style, some added requirements, added paragraphs, and changes in + algorithms specification. + +3.14. RFC 8378 + + "Signal-Free Locator/ID Separation Protocol (LISP) Multicast" + [RFC8378] is an Experimental RFC, defining how to implement Multicast + in the LISP architecture. + + Status (Length): Experimental (21 pages) + Overview: 5 individual drafts; 10 WG drafts + First draft: 2014-02-28 + WG adoption: 2015-12-21 + Last Call start: 2018-02-13 (draft 07) + IESG eval. start: 2018-02-28 (draft 08) + IESG approved: 2018-03-12 (draft 09) + AUTH48 start: 2018-04-23 + AUTH48 complete: 2018-05-02 + Published: 2018-05-02 + + Preparing the RFC took more than 4 years. According to the authors, + they were not aggressively pushing it and just let the working group + process decide to pace it. They also did implementations during that + time. + + Minor copy editing, for style. + + The RFC was implemented by lispers.net and Cisco, and it was used in + doing IPv6 multicast over IPv4 unicast/multicast at the Olympics in + PyeungChang. The plan is to work on a Proposed Standard once the + experiment concludes. + +3.15. RFC 8361 + + "Transparent Interconnection of Lots of Links (TRILL): Centralized + Replication for Active-Active Broadcast, Unknown Unicast, and + Multicast (BUM) Traffic" [RFC8361] + + Status (Length): Proposed Standard (17 pages) + Overview: 3 individual drafts; 14 WG drafts + First draft: 2013-11-12 + WG adoption: 2014-12-16 + Last Call start: 2017-11-28 (draft 10) + IESG eval. start: 2017-12-18 (draft 11) + IESG approved: 2018-01-29 (draft 13) + AUTH48 start: 2018-03-09 + AUTH48 complete: 2018-04-09 + Published: 2018-04-12 + + According to the authors, the long delays in producing this RFC were + due to a slow uptake of the technology in the industry. + + Minor copy editing, for style. + + There was at least one partial implementation. + +3.16. RFC 8472 + + "Transport Layer Security (TLS) Extension for Token Binding Protocol + Negotiation" [RFC8472] + + Status (Length): Proposed Standard (8 pages) + Overview: 1 individual draft; 15 WG drafts + First draft: 2015-05-29 + WG adoption: 2015-09-11 + Last Call start: 2017-11-13 (draft 10) + IESG eval. start: 2018-03-19 + IESG approved: 2018-07-20 (draft 14) + AUTH48 start: 2018-09-17 + AUTH48 complete: 2018-09-25 + Published: 2018-10-08 + + This is a pretty simple document, but it took over 3 years from + individual draft to RFC. According to the authors,the biggest + setbacks occurred at the start: it took a while to find a home for + this draft. It was presented in the TLS WG (because it's a TLS + extension) and UTA WG (because it has to do with applications using + TLS). Then the ADs determined that a new WG was needed, so the + authors had to work through the WG creation process, including + running a BOF. + + Minor copy editing, for style, with the addition of a reference to + TLS 1.3. + + Perhaps partially due to the delays, some of the implementers lost + interest in supporting this RFC. + +3.17. RFC 8471 + + "The Token Binding Protocol Version 1.0" [RFC8471] + + Status (Length): Proposed Standard (18 pages) + Overview: 1 individual draft; 19 WG drafts + First draft: 2014-10-13 + WG adoption: 2015-03-15 + Last Call start: 2017-11-13 (draft 16) + IESG eval. start: 2018-03-19 + IESG approved: 2018-07-20 (draft 19) + AUTH48 start: 2018-09-17 + AUTH48 complete: 2018-09-25 + Published: 2018-10-08 + + This document presents a Token Binding Protocol for TLS. We can + notice a period of 5 months before adoption of the draft by the WG. + That explains in part the overall time of almost 4 years from first + draft to publication. + + Minor copy editing, for style. + + The web references indicate adoption in multiple development + projects. + +3.18. RFC 8466 + + "A YANG Data Model for Layer 2 Virtual Private Network (L2VPN) + Service Delivery" [RFC8466] + + Status (Length): Proposed Standard (158 pages) + Overview: 5 individual drafts; 11 WG drafts + First draft: 2016-09-01 + WG adoption: 2017-02-26 + Last Call start: 2018-02-21 (draft 07) + IESG eval. start: 2018-03-14 (draft 08) + IESG approved: 2018-06-25 (draft 10) + AUTH48 start: 2018-09-17 + AUTH48 complete: 2018-10-09 + Published: 2018-10-12 + + Copy editing for style and clarity, with also corrections to the YANG + model. + +3.19. RFC 8362 + + "OSPFv3 Link State Advertisement (LSA) Extensibility" [RFC8362] is a + major extension to the OSPF protocol. It makes OSPFv3 fully + extensible. + + Status (Length): Proposed Standard (33 pages) + Overview: 4 individual drafts; 24 WG drafts + First draft: 2013-02-17 + WG adoption: 2013-10-15 + Last Call start: 2017-12-19 (draft 19) + IESG eval. start: 2018-01-18 (draft 20) + IESG approved: 2018-01-29 (draft 23) + AUTH48 start: 2018-03-19 + AUTH48 complete: 2018-03-30 + Published: 2018-04-03 + + The specification was first submitted as an individual draft in the + IPv6 WG, then moved to the OSPF WG. The long delay of producing this + RFC is due to the complexity of the problem, and the need to wait for + implementations. It is a very important change to OSPF that makes + OSPFv3 fully extensible. Since it was a non-backward compatible + change, the developers started out with some very complex migration + scenarios but ended up with either legacy or extended OSPFv3 LSAs + within an OSPFv3 routing domain. The initial attempts to have a + hybrid mode of operation with both legacy and extended LSAs also + delayed implementation due to the complexity. + + Copy editing for style and clarity. + + This specification either was or will be implemented by all the + router vendors. + +3.20. RFC 8468 + + "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for the IP + Performance Metrics (IPPM) Framework" [RFC8468]. + + Status (Length): Informational (15 pages) + Overview: 3 individual drafts; 7 WG drafts + First draft: 2015-08-06 + WG adoption: 2016-07-04 + Last Call start: 2018-04-11 (draft 04) + IESG eval. start: 2018-05-24 (draft 05) + IESG approved: 2018-07-10 (draft 06) + AUTH48 start: 2018-09-13 + AUTH48 complete: 2018-11-05 + Published: 2018-11-14 + + RFC 8468 was somehow special in that there was not a technical reason + or interest that triggered it, but rather a formal requirement. + While writing RFC 7312, the IP Performance Metrics (IPPM) Working + Group realized that RFC 2330, the IP Performance Metrics Framework + supported IPv4 only and explicitly excluded support for IPv6. + Nevertheless, people used the metrics that were defined on top of RFC + 2330 (and, therefore, IPv4 only) for IPv6, too. Although the IPPM WG + agreed that the work was needed, the interest of IPPM attendees in + progressing (and reading/reviewing) the IPv6 draft was limited. + Resolving the IPv6 technical part was straightforward, but + subsequently some people asked for a broader scope (topics like + header compression, 6LoWPAN, etc.), and it took some time to figure + out and later on convince people that these topics are out of scope. + The group also had to resolve contentious topics, for example, how to + measure the processing of IPv6 extension headers, which is sometimes + nonstandard. + + The time in AUTH48 state for this document was longer than average. + According to the authors, the main reasons include: + + * Workload and travel caused by busy work periods of all coauthors + + * Time zone difference between coauthors and editor (at least US, + Europe, and India, not considering travel) + + * RFC Production Center proposed and committed some unacceptable + modifications that needed to be reverted + + * Lengthy discussions on a new document title (required high effort + and took a long time, in particular reaching consensus between + coauthors and editor was time-consuming and involved the AD) + + * RFC Production Center correctly identified some nits (obsoleted + personal websites of coauthors) and coauthors attempting to fix + them. + + The differences between the final draft and the published RFC show + copy editing for style and clarity, but do not account for the back + and forth between authors and editors mentioned by the authors. + +4. Analysis of Process and Delays + + We examine the 20 RFCs in the sample, measuring various + characteristics such as delay and citation counts, in an attempt to + identify patterns in the IETF processes. + +4.1. Delays from First Draft to RFC + + We look at the distribution of delays between the submission of the + first draft and the publication of the RFC, using the three + milestones defined in Section 2.1: processing time in the working + group, IETF processing time, and RFC production time. The following + table shows the number of days in each phase for the 20 RFCs in the + sample: + + +======+============+=======+=========+======+======+======+ + | RFC | Status | Pages | Overall | WG | IETF | Edit | + +======+============+=======+=========+======+======+======+ + | 8411 | Info | 5 | 455 | 154 | 140 | 161 | + +------+------------+-------+---------+------+------+------+ + | 8456 | Info | 64 | 1317 | 1033 | 126 | 158 | + +------+------------+-------+---------+------+------+------+ + | 8446 | PS | 160 | 1576 | 1400 | 34 | 142 | + +------+------------+-------+---------+------+------+------+ + | 8355 | Info | 13 | 1517 | 1175 | 243 | 99 | + +------+------------+-------+---------+------+------+------+ + | 8441 | PS | 8 | 327 | 204 | 31 | 92 | + +------+------------+-------+---------+------+------+------+ + | 8324 | Info (ISE) | 29 | 270 | 38 | 161 | 71 | + +------+------------+-------+---------+------+------+------+ + | 8377 | PS | 8 | 1792 | 1630 | 21 | 141 | + +------+------------+-------+---------+------+------+------+ + | 8498 | Info | 15 | 1059 | 935 | 59 | 65 | + +------+------------+-------+---------+------+------+------+ + | 8479 | Info (ISE) | 8 | 414 | 233 | 144 | 37 | + +------+------------+-------+---------+------+------+------+ + | 8453 | Info | 42 | 1165 | 1036 | 46 | 83 | + +------+------------+-------+---------+------+------+------+ + | 8429 | BCP | 10 | 548 | 76 | 313 | 159 | + +------+------------+-------+---------+------+------+------+ + | 8312 | Info | 18 | 1214 | 1113 | 16 | 85 | + +------+------------+-------+---------+------+------+------+ + | 8492 | Info (ISE) | 40 | 2358 | 1706 | 172 | 480 | + +------+------------+-------+---------+------+------+------+ + | 8378 | Exp | 21 | 1524 | 1446 | 27 | 51 | + +------+------------+-------+---------+------+------+------+ + | 8361 | PS | 17 | 1612 | 1477 | 62 | 73 | + +------+------------+-------+---------+------+------+------+ + | 8472 | PS | 8 | 1228 | 899 | 249 | 80 | + +------+------------+-------+---------+------+------+------+ + | 8471 | PS | 18 | 1228 | 899 | 249 | 80 | + +------+------------+-------+---------+------+------+------+ + | 8466 | PS | 158 | 771 | 538 | 124 | 109 | + +------+------------+-------+---------+------+------+------+ + | 8362 | PS | 33 | 1871 | 1766 | 41 | 64 | + +------+------------+-------+---------+------+------+------+ + | 8468 | Info | 15 | 1196 | 979 | 90 | 127 | + +------+------------+-------+---------+------+------+------+ + | average | 35 | 1172 | 948 | 117 | 118 | + +-------------------+-------+---------+------+------+------+ + | average (not ISE) | 36 | 1200 | 999 | 110 | 104 | + +-------------------+-------+---------+------+------+------+ + + Table 1 + + The average delay from first draft to publication is about 3 years + and 3 months, but this varies widely. Excluding the RFCs from the + Independent Stream, the average delay from start to finish is 3 years + and 4 months, of which on average 2 years and 9 months are spent + getting consensus in the working group, and 3 to 4 months each for + IETF consensus and for RFC production. + + The longest delay is found for [RFC8492], 6.5 years from start to + finish. This is however a very special case -- a draft that was + prepared for the TLS Working Group and failed to reach consensus. + After that, it was resubmitted to the ISE, and incurred atypical + production delays. + + On average, we see that 80% of the delay is incurred in WG + processing, 10% in IETF review, and 10% for edition and publication. + + For IETF Stream RFCs, it appears that the delays for Informational + documents are slightly shorter than those for protocol + specifications, maybe six months shorter on average. However, there + are lots of differences between individual documents. The delays + range from less than a year to more than 5 years for protocol + specifications, and from a year and 3 months to a bit more than 4 + years for Informational documents. + + We can compare the delays in the 2018 samples to those observed 10 + years ago and 20 years before: + + +============+============+=======+=======+ + | RFC (2008) | Status | Pages | Delay | + +============+============+=======+=======+ + | 5326 | Exp | 54 | 1584 | + +------------+------------+-------+-------+ + | 5348 | PS | 58 | 823 | + +------------+------------+-------+-------+ + | 5281 | Info | 51 | 1308 | + +------------+------------+-------+-------+ + | 5354 | Exp | 23 | 2315 | + +------------+------------+-------+-------+ + | 5227 | PS | 21 | 2434 | + +------------+------------+-------+-------+ + | 5329 | PS | 12 | 1980 | + +------------+------------+-------+-------+ + | 5277 | PS | 35 | 912 | + +------------+------------+-------+-------+ + | 5236 | Info (ISE) | 26 | 1947 | + +------------+------------+-------+-------+ + | 5358 | BCP | 7 | 884 | + +------------+------------+-------+-------+ + | 5271 | Info | 22 | 1066 | + +------------+------------+-------+-------+ + | 5195 | PS | 10 | 974 | + +------------+------------+-------+-------+ + | 5283 | PS | 12 | 1096 | + +------------+------------+-------+-------+ + | 5186 | Info | 6 | 2253 | + +------------+------------+-------+-------+ + | 5142 | PS | 13 | 1005 | + +------------+------------+-------+-------+ + | 5373 | PS | 24 | 1249 | + +------------+------------+-------+-------+ + | 5404 | PS | 27 | 214 | + +------------+------------+-------+-------+ + | 5172 | PS | 7 | 305 | + +------------+------------+-------+-------+ + | 5349 | Info | 10 | 1096 | + +------------+------------+-------+-------+ + | 5301 | PS | 6 | 396 | + +------------+------------+-------+-------+ + | 5174 | Info | 8 | 427 | + +------------+------------+-------+-------+ + + Table 2 + + +============+============+=======+=========+ + | RFC (1998) | Status | Pages | Delay | + +============+============+=======+=========+ + | 2289 | PS | 25 | 396 | + +------------+------------+-------+---------+ + | 2267 | Info | 10 | unknown | + +------------+------------+-------+---------+ + | 2317 | BCP | 10 | 485 | + +------------+------------+-------+---------+ + | 2404 | PS | 7 | 488 | + +------------+------------+-------+---------+ + | 2374 | PS | 12 | 289 | + +------------+------------+-------+---------+ + | 2449 | PS | 19 | 273 | + +------------+------------+-------+---------+ + | 2283 | PS | 9 | 153 | + +------------+------------+-------+---------+ + | 2394 | Info | 6 | 365 | + +------------+------------+-------+---------+ + | 2348 | DS | 5 | 699 | + +------------+------------+-------+---------+ + | 2382 | Info | 30 | 396 | + +------------+------------+-------+---------+ + | 2297 | Info (ISE) | 109 | 28 | + +------------+------------+-------+---------+ + | 2381 | PS | 43 | 699 | + +------------+------------+-------+---------+ + | 2312 | Info | 20 | 365 | + +------------+------------+-------+---------+ + | 2387 | PS | 10 | 122 | + +------------+------------+-------+---------+ + | 2398 | Info | 15 | 396 | + +------------+------------+-------+---------+ + | 2391 | PS | 10 | 122 | + +------------+------------+-------+---------+ + | 2431 | PS | 10 | 457 | + +------------+------------+-------+---------+ + | 2282 | Info | 14 | 215 | + +------------+------------+-------+---------+ + | 2323 | Info (ISE) | 5 | unknown | + +------------+------------+-------+---------+ + | 2448 | Info (ISE) | 7 | 92 | + +------------+------------+-------+---------+ + + Table 3 + + We can compare the median delay, and the delays observed by the + fastest and slowest quartiles in the three years: + + +======+=============+========+=============+ + | Year | Fastest 25% | Median | Slowest 25% | + +======+=============+========+=============+ + | 2018 | 715 | 1221 | 1537 | + +------+-------------+--------+-------------+ + | 2008 | 869 | 1081 | 1675 | + +------+-------------+--------+-------------+ + | 1998 | 169 | 365 | 442 | + +------+-------------+--------+-------------+ + + Table 4 + + The IETF takes three to four times more to produce an RFC in 2018 + than it did in 1998, but about the same time as it did in 2008. We + can get a rough estimate of how this translates in terms of "level of + attention" per RFC by comparing the number of participants in the + IETF meetings of 2018, 2008, and 1998 [IETFCOUNT] to the number of + RFCs published these years [RFCYEAR]. + + +======+=========+========+========+======+=========+============+ + | Year | Number | Spring | Summer | Fall | Average | Attendees/ | + | | of RFCs | P. | P. | P. | P. | RFC | + +======+=========+========+========+======+=========+============+ + | 2018 | 208 | 1235 | 1078 | 879 | 1064 | 5.1 | + +------+---------+--------+--------+------+---------+------------+ + | 2008 | 290 | 1128 | 1181 | 962 | 1090 | 3.8 | + +------+---------+--------+--------+------+---------+------------+ + | 1998 | 234 | 1775 | 2106 | 1705 | 1862 | 8.0 | + +------+---------+--------+--------+------+---------+------------+ + + Table 5 + + The last column in the table provides the ratio of average number of + participants to the number of RFCs published. If the IETF were a + centralized organization, and if all participants and documents were + equivalent, this ratio would be the number of participants dedicated + to produce an RFC on a given year. This is of course a completely + abstract figure because none of the hypotheses above are true, but it + still gives a vague indication of the "level of attention" applied to + documents. We see that this ratio has increased from 2008 to 2018, + as the number of participants was about the same for these two years + but the number of published RFCs decreased. However, this ratio was + much higher in 1998. The IETF had many more participants, and there + were probably many more eyes available to review any given draft. If + we applied the ratios of 1998, the IETF would be producing 119 + documents in 2018 instead of 208. + +4.2. Working Group Processing Time + + The largest part of the delays is spent in the working groups, before + the draft is submitted to the IESG for IETF review. As mentioned in + Section 2.1, the only intermediate milestone that we can extract from + the IETF Datatracker is the date at which the document was adopted by + the working group, or targeted for independent submission. The + breakdown of the delays for the documents in our sample is: + + +======+============+======+================+================+ + | RFC | Status | WG | Until adoption | After adoption | + +======+============+======+================+================+ + | 8411 | Info | 154 | 0 | 154 | + +------+------------+------+----------------+----------------+ + | 8456 | Info | 1033 | 209 | 824 | + +------+------------+------+----------------+----------------+ + | 8446 | PS | 1400 | 0 | 1400 | + +------+------------+------+----------------+----------------+ + | 8355 | Info | 1175 | 102 | 1073 | + +------+------------+------+----------------+----------------+ + | 8441 | PS | 204 | 65 | 139 | + +------+------------+------+----------------+----------------+ + | 8324 | Info (ISE) | 38 | 0 | 38 | + +------+------------+------+----------------+----------------+ + | 8377 | PS | 1630 | 728 | 902 | + +------+------------+------+----------------+----------------+ + | 8498 | Info | 935 | 420 | 515 | + +------+------------+------+----------------+----------------+ + | 8479 | Info (ISE) | 233 | 0 | 233 | + +------+------------+------+----------------+----------------+ + | 8453 | Info | 1036 | 396 | 640 | + +------+------------+------+----------------+----------------+ + | 8429 | BCP | 76 | 0 | 76 | + +------+------------+------+----------------+----------------+ + | 8312 | Info | 1113 | 280 | 833 | + +------+------------+------+----------------+----------------+ + | 8492 | Info (ISE) | 1706 | 1428 | 278 | + +------+------------+------+----------------+----------------+ + | 8378 | Exp | 1446 | 661 | 785 | + +------+------------+------+----------------+----------------+ + | 8361 | PS | 1477 | 399 | 1078 | + +------+------------+------+----------------+----------------+ + | 8472 | PS | 899 | 105 | 794 | + +------+------------+------+----------------+----------------+ + | 8471 | PS | 1127 | 153 | 794 | + +------+------------+------+----------------+----------------+ + | 8466 | PS | 538 | 178 | 360 | + +------+------------+------+----------------+----------------+ + | 8362 | PS | 1766 | 240 | 1526 | + +------+------------+------+----------------+----------------+ + | 8468 | Info | 979 | 333 | 646 | + +------+------------+------+----------------+----------------+ + | Average | 948 | 285 | 663 | + +-------------------+------+----------------+----------------+ + + Table 6 + + The time before working group adoption averages to a bit more than 9 + months, compared to 1 year and almost 10 months for processing time + after adoption. We see that RFC 8492 stands out, with long delays + spent attempting publication through a working group before + submission to the Independent Submissions Editor. If we remove RFC + 8492 from the list, the average time until adoption drops to just + over 7 months, and becomes just 25% of the total processing time in + the WG. + + There are a few documents that started immediately as working group + efforts, or were immediately targeted for publication in the + Independent Stream. Those documents tend to see short processing + times, with the exception of RFC 8446 on which the TLS Working Group + spent a long time working. + +4.3. Preparation and Publication Delays + + The preparation and publication delays include three components: + + * the delay from submission to the RFC Editor to beginning of + AUTH48, during which the document is prepared (referred to as "RFC + edit" below); + + * the AUTH48 delay, during which authors review and eventually + approve the changes proposed by the editors (referred to as + "AUTH48" below); + + * the publication delay, from final agreement by authors and editors + to actual publication (referred to as "RFC Pub" below). + + The breakdown of the publication delays for each RFC is shown in the + following table. + + +=======+========+=======+==========+========+=========+=========+ + | RFC | Status | Pages | RFC edit | AUTH48 | RFC Pub | Edit | + | | | | | | | (total) | + +=======+========+=======+==========+========+=========+=========+ + | 8411 | Info | 5 | 53 | 88 | 20 | 161 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8456 | Info | 64 | 98 | 46 | 14 | 158 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8446 | PS | 160 | 85 | 57 | 0 | 142 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8355 | Info | 13 | 83 | 15 | 1 | 99 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8441 | PS | 8 | 56 | 33 | 3 | 92 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8324 | Info | 29 | 42 | 28 | 1 | 71 | + | | (ISE) | | | | | | + +-------+--------+-------+----------+--------+---------+---------+ + | 8377 | PS | 8 | 39 | 102 | 0 | 141 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8498 | Info | 15 | 48 | 16 | 1 | 65 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8479 | Info | 8 | 31 | 5 | 1 | 37 | + | | (ISE) | | | | | | + +-------+--------+-------+----------+--------+---------+---------+ + | 8453 | Info | 42 | 73 | 7 | 3 | 83 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8429 | BCP | 10 | 60 | 99 | 0 | 159 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8312 | Info | 18 | 55 | 28 | 2 | 85 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8492 | Info | 40 | 355 | 123 | 2 | 480 | + | | (ISE) | | | | | | + +-------+--------+-------+----------+--------+---------+---------+ + | 8378 | Exp | 21 | 42 | 9 | 0 | 51 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8361 | PS | 17 | 39 | 31 | 3 | 73 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8472 | PS | 8 | 59 | 8 | 13 | 80 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8471 | PS | 18 | 59 | 8 | 13 | 80 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8466 | PS | 158 | 84 | 22 | 3 | 109 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8362 | PS | 33 | 49 | 11 | 4 | 64 | + +-------+--------+-------+----------+--------+---------+---------+ + | 8468 | Info | 15 | 65 | 53 | 9 | 127 | + +-------+--------+-------+----------+--------+---------+---------+ + | Average | | 74 | 39 | 5 | 118 | + +----------------+-------+----------+--------+---------+---------+ + | Average | | 59 | 35 | 5 | 99 | + | (without 8492) | | | | | | + +----------------+-------+----------+--------+---------+---------+ + + Table 7 + + On average, the total delay appears to be about four months, but the + average is skewed by the extreme values encountered for [RFC8492]. + If we exclude that RFC from the computations, the average delay drops + to a just a bit more than 3 months: about 2 months for the + preparation, a bit more than one month for the AUTH48 phase, and 5 + days for the publishing. + + Of course, these delays vary from RFC to RFC. To try explain the + causes of the delay, we compute the correlation factor between the + observed delays and several plausible explanation factors: + + * the number of pages in the document, + + * the amount of copy editing, as discussed in Section 4.4, + + * whether or not IANA actions were required, + + * the number of authors, + + * the number of draft revisions, + + * the working group delay. + + We find the following values: + + +===================+==========+========+=============+ + | Correlation | RFC edit | AUTH48 | Edit(total) | + +===================+==========+========+=============+ + | Number of pages | 0.50 | -0.04 | 0.21 | + +-------------------+----------+--------+-------------+ + | Copy-Edit | 0.42 | 0.24 | 0.45 | + +-------------------+----------+--------+-------------+ + | IANA | -0.14 | -0.21 | 0.12 | + +-------------------+----------+--------+-------------+ + | Number of authors | 0.39 | -0.07 | 0.18 | + +-------------------+----------+--------+-------------+ + | Number of drafts | 0.18 | -0.33 | -0.19 | + +-------------------+----------+--------+-------------+ + | WG delay | 0.03 | -0.16 | -0.15 | + +-------------------+----------+--------+-------------+ + + Table 8 + + We see some plausible explanations for the production delay. It will + be somewhat longer for longer documents or for documents that require + a lot of copy editing (see Section 4.4). Somewhat surprisingly, it + also tends to increase with the number of authors. It does not + appear significantly correlated with the presence or absence of IANA + action. + + The analysis of RFC 8324 in Section 3.6 explains its short editing + delays by the experience of the author. This makes sense: if a + document needs less editing, the editing delays would be shorter. + This is partially confirmed by the relation between the amount of + copy editing and the publication delay. + + We see fewer plausible explanations for the AUTH48 delays. These + delays vary much more than the preparation delay, with a standard + deviation of 20 days for AUTH48 versus 10 days for the preparation + delay. In theory, AUTH48 is just a final verification: the authors + receive the document prepared by the RFC production center, and just + have to give their approval, or maybe request a last minute + correction. The name indicates that this is expected to last just + two days, but in average it lasts more than a month. + + We often hypothesize that the number of authors influences the AUTH48 + delay, or that authors who have spent a long time working on the + document in the working group somehow get demotivated and spend even + longer to answer questions during AUTH48. This may happen sometimes, + but our statistics don't show that - if anything, the numerical + results point in the opposite direction. + + After asking the authors of the RFCs in the sample why the AUTH48 + phase took a long time, we got three explanations: + + 1. Some RFCs have multiple authors in multiple time zones. This + slows down the coordination required for approving changes. + + 2. Some authors found some of the proposed changes unnecessary or + undesirable, and asked that they be reversed. This required long + exchanges between authors and editors. + + 3. Some authors were not giving high priority to AUTH48 responses. + + As mentioned above, we were not able to verify these hypotheses by + looking at the data. The author's experience with this document + suggests another potential delay for the Independent Stream RFC: + processing delay by the Independent Submissions Editor, discussed in + Section 4.5. + +4.4. Copy Editing + + We can assess the amount of copy editing applied to each published + RFC by comparing the text of the draft approved for publication and + the text of the RFC. We do expect differences in the "boilerplate" + and in the IANA section, but we will also see differences due to copy + editing. Assessing the amount of copy editing is subjective, and we + do it using a scale of 1 to 4: + + 1: Minor editing + + 2: Editing for style, such as capitalization, hyphens, "that" versus + "which", and expanding all acronyms at least once. + + 3: Editing for clarity in addition to style, such as rewriting + ambiguous sentences and clarifying use of internal references. + For YANG models, that may include model corrections suggested by + the verifier. + + 4: Extensive editing. + + The following table shows that about half of the RFCs required + editing for style, and the other half at least some editing for + clarity. + + +======+============+===========+ + | RFC | Status | Copy Edit | + +======+============+===========+ + | 8411 | Info | 2 | + +------+------------+-----------+ + | 8456 | Info | 4 | + +------+------------+-----------+ + | 8446 | PS | 3 | + +------+------------+-----------+ + | 8355 | Info | 2 | + +------+------------+-----------+ + | 8441 | PS | 2 | + +------+------------+-----------+ + | 8324 | Info (ISE) | 2 | + +------+------------+-----------+ + | 8377 | PS | 3 | + +------+------------+-----------+ + | 8498 | Info | 3 | + +------+------------+-----------+ + | 8479 | Info (ISE) | 1 | + +------+------------+-----------+ + | 8453 | Info | 2 | + +------+------------+-----------+ + | 8429 | BCP | 2 | + +------+------------+-----------+ + | 8312 | Info | 2 | + +------+------------+-----------+ + | 8492 | Info (ISE) | 3 | + +------+------------+-----------+ + | 8378 | Exp | 2 | + +------+------------+-----------+ + | 8361 | PS | 2 | + +------+------------+-----------+ + | 8472 | PS | 2 | + +------+------------+-----------+ + | 8471 | PS | 2 | + +------+------------+-----------+ + | 8466 | PS | 3 | + +------+------------+-----------+ + | 8362 | PS | 3 | + +------+------------+-----------+ + | 8468 | Info | 3 | + +------+------------+-----------+ + + Table 9 + + This method of assessment does not take into account the number of + changes proposed by the editors and eventually rejected by the + authors, since these changes are not present in either the final + draft or the published RFC. It might be possible to get an + evaluation of these "phantom changes" from the RFC Production Center. + +4.5. Independent Stream + + Out of 20 randomly selected RFCs, 3 were published through the + Independent Stream. One is an independent opinion, another a + description of a non-IETF protocol format, and the third was + [RFC8492], which is a special case. Apart from this special case, + the publication delays were significantly shorter for the Independent + Stream than for the IETF Stream. + + The authors of these 3 RFCs are regular IETF contributors. This + observation motivated a secondary analysis of all the RFCs published + in the Independent Stream in 2018. There are 14 such RFCs: 8507, + 8494, 8493, 8492, 8483, 8479, 8433, 8409, 8374, 8369, 8367, 8351, + 8328, and 8324. (RFCs 8367 and 8369 were published on 1 April 2018.) + The majority of the documents were published by regular IETF + participants, but two of them were not. One describes "The BagIt + File Packaging Format (V1.0)" [RFC8493], and the other the "Yeti DNS + Testbed" [RFC8483]. They document a data format and a system + developed outside the IETF and illustrate the outreach function of + the Independent Stream. In both cases, the authors include one + experienced IETF participant, who presumably helped outsiders + navigate the publication process. + + The present document experienced some publication delays due to the + Independent Submissions Editor. The ISE is a bottleneck and is a + volunteer resource. Although the ISE as a lone person operating as a + volunteer is still roughly adequate resource for the job, the + delivery will necessarily be best effort with delays caused by spikes + in ISE load, work commitments, and other life events. These delays + may not be fundamentally critical to RFC delivery, but they are + capable of introducing a significant percentage delay into what might + otherwise be a smooth process. + +5. Citation Counts + + In this exploration, we want to examine whether citation counts + provide a meaningful assessment of the popularity of RFCs. We obtain + the citation counts through the Semantic Scholar API, using queries + of the form: <https://api.semanticscholar.org/v1/paper/10.17487/ + rfc8446?include_unknown_references=true> + + In these queries, the RFC is uniquely identified by its DOI + reference, which is composed of the RFC Series prefix 10.17487 and + the RFC identifier. The queries return a series of properties, + including a list of citations for the RFC. Based on that list of + citations, we compute three numbers: + + * The total number of citations + + * The number of citations in the year of publication and the year + after that + + * For the RFC published in 1998 or 2008 that we use for comparison, + the number of citations in the years 2018 and 2019. + + All the numbers were retrieved on October 6, 2019. + +5.1. Citation Numbers + + As measured on October 6, 2019, the citation counts for the RFC in + our sample set were: + + +============+============+=======+===========+ + | RFC (2018) | Status | Total | 2018-2019 | + +============+============+=======+===========+ + | 8411 | Info | 1 | 0 | + +------------+------------+-------+-----------+ + | 8456 | Info | 1 | 1 | + +------------+------------+-------+-----------+ + | 8446 | PS | 418 | 204 | + +------------+------------+-------+-----------+ + | 8355 | Info | 3 | 3 | + +------------+------------+-------+-----------+ + | 8441 | PS | 1 | 1 | + +------------+------------+-------+-----------+ + | 8324 | Info (ISE) | 0 | 0 | + +------------+------------+-------+-----------+ + | 8377 | PS | 0 | 0 | + +------------+------------+-------+-----------+ + | 8498 | Info | 0 | 0 | + +------------+------------+-------+-----------+ + | 8479 | Info (ISE) | 0 | 0 | + +------------+------------+-------+-----------+ + | 8453 | Info | 3 | 3 | + +------------+------------+-------+-----------+ + | 8429 | BCP | 0 | 0 | + +------------+------------+-------+-----------+ + | 8312 | Info | 25 | 16 | + +------------+------------+-------+-----------+ + | 8492 | Info (ISE) | 4 | 4 | + +------------+------------+-------+-----------+ + | 8378 | Exp | 1 | 1 | + +------------+------------+-------+-----------+ + | 8361 | PS | 0 | 0 | + +------------+------------+-------+-----------+ + | 8472 | PS | 1 | 1 | + +------------+------------+-------+-----------+ + | 8471 | PS | 1 | 1 | + +------------+------------+-------+-----------+ + | 8466 | PS | 0 | 0 | + +------------+------------+-------+-----------+ + | 8362 | PS | 1 | 1 | + +------------+------------+-------+-----------+ + | 8468 | Info | 1 | 1 | + +------------+------------+-------+-----------+ + + Table 10 + + The results indicate that [RFC8446] is by far the most cited of the + 20 RFC in our sample. This is not surprising, since TLS is a key + Internet Protocol. The TLS 1.3 protocol was also the subject of + extensive studies by researchers, and thus was mentioned in a number + of published papers. Surprisingly, the Semantic Scholar mentions a + number of citations that predate the publication date. These are + probably citations of the various draft versions of the protocol. + + The next most cited RFC in the sample is [RFC8312] which describes + the Cubic congestion control algorithm for TCP. That protocol was + also the target of a large number of academic publications. The + other RFCs in the sample only have a small number of citations. + + There is probably a small bias when measuring citations at a fixed + date. An RFC published in January 2018 would have more time to + accrue citations than one published in December. That may be true to + some extent, as the second most cited RFC in the set was published in + January. However, the effect has to be limited. The most cited RFC + was published in August, and the second most cited was published in + 2019. (That RFC got an RFC number in 2018, but publication was + slowed by long AUTH48 delays.) + +5.2. Comparison to 1998 and 2008 + + In order to get a baseline, we can look at the number of references + for the RFCs published in 2008 and 1998. However, we need to take + time into account. Documents published a long time ago are expected + to have accrued more references. We try to address this by looking + at three counts for each document: the overall number of references + over the document's lifetime, the number of references obtained in + the year following publication, and the number of references observed + since 2018: + + +============+============+=======+===========+===========+ + | RFC (2008) | Status | Total | 2008-2009 | 2018-2019 | + +============+============+=======+===========+===========+ + | 5326 | Exp | 138 | 14 | 15 | + +------------+------------+-------+-----------+-----------+ + | 5348 | PS | 14 | 3 | 0 | + +------------+------------+-------+-----------+-----------+ + | 5281 | Info | 69 | 15 | 7 | + +------------+------------+-------+-----------+-----------+ + | 5354 | Exp | 17 | 13 | 0 | + +------------+------------+-------+-----------+-----------+ + | 5227 | PS | 19 | 1 | 2 | + +------------+------------+-------+-----------+-----------+ + | 5329 | PS | 24 | 6 | 1 | + +------------+------------+-------+-----------+-----------+ + | 5277 | PS | 32 | 3 | 2 | + +------------+------------+-------+-----------+-----------+ + | 5236 | Info (ISE) | 25 | 5 | 4 | + +------------+------------+-------+-----------+-----------+ + | 5358 | BCP | 21 | 2 | 0 | + +------------+------------+-------+-----------+-----------+ + | 5271 | Info | 7 | 2 | 0 | + +------------+------------+-------+-----------+-----------+ + | 5195 | PS | 7 | 4 | 2 | + +------------+------------+-------+-----------+-----------+ + | 5283 | PS | 8 | 1 | 0 | + +------------+------------+-------+-----------+-----------+ + | 5186 | Info | 14 | 4 | 2 | + +------------+------------+-------+-----------+-----------+ + | 5142 | PS | 8 | 4 | 0 | + +------------+------------+-------+-----------+-----------+ + | 5373 | PS | 5 | 2 | 0 | + +------------+------------+-------+-----------+-----------+ + | 5404 | PS | 1 | 1 | 0 | + +------------+------------+-------+-----------+-----------+ + | 5172 | PS | 2 | 0 | 0 | + +------------+------------+-------+-----------+-----------+ + | 5349 | Info | 8 | 0 | 2 | + +------------+------------+-------+-----------+-----------+ + | 5301 | PS | 5 | 1 | 0 | + +------------+------------+-------+-----------+-----------+ + | 5174 | Info | 0 | 0 | 0 | + +------------+------------+-------+-----------+-----------+ + + Table 11 + + +============+============+=======+===========+===========+ + | RFC (1998) | Status | Total | 1998-1999 | 2018-2019 | + +============+============+=======+===========+===========+ + | 2289 | PS | 2 | 0 | 1 | + +------------+------------+-------+-----------+-----------+ + | 2267 | Info | 982 | 5 | 61 | + +------------+------------+-------+-----------+-----------+ + | 2317 | BCP | 9 | 1 | 2 | + +------------+------------+-------+-----------+-----------+ + | 2404 | PS | 137 | 6 | 1 | + +------------+------------+-------+-----------+-----------+ + | 2374 | PS | 42 | 4 | 0 | + +------------+------------+-------+-----------+-----------+ + | 2449 | PS | 7 | 2 | 0 | + +------------+------------+-------+-----------+-----------+ + | 2283 | PS | 17 | 3 | 2 | + +------------+------------+-------+-----------+-----------+ + | 2394 | Info | 13 | 2 | 1 | + +------------+------------+-------+-----------+-----------+ + | 2348 | DS | 5 | 0 | 0 | + +------------+------------+-------+-----------+-----------+ + | 2382 | Info | 17 | 12 | 0 | + +------------+------------+-------+-----------+-----------+ + | 2297 | Info (ISE) | 36 | 11 | 0 | + +------------+------------+-------+-----------+-----------+ + | 2381 | PS | 39 | 12 | 0 | + +------------+------------+-------+-----------+-----------+ + | 2312 | Info | 14 | 3 | 0 | + +------------+------------+-------+-----------+-----------+ + | 2387 | PS | 4 | 1 | 0 | + +------------+------------+-------+-----------+-----------+ + | 2398 | Info | 17 | 0 | 1 | + +------------+------------+-------+-----------+-----------+ + | 2391 | PS | 31 | 3 | 0 | + +------------+------------+-------+-----------+-----------+ + | 2431 | PS | 3 | 0 | 0 | + +------------+------------+-------+-----------+-----------+ + | 2282 | Info | 8 | 0 | 0 | + +------------+------------+-------+-----------+-----------+ + | 2323 | Info (ISE) | 1 | 0 | 0 | + +------------+------------+-------+-----------+-----------+ + | 2448 | Info (ISE) | 0 | 0 | 0 | + +------------+------------+-------+-----------+-----------+ + + Table 12 + + We can compare the median number of citations and the numbers of + citations for the least and most popular quartiles in the three + years: + + +============================+===========+========+============+ + | References | Lower 25% | Median | Higher 25% | + +============================+===========+========+============+ + | RFC (2018) | 0 | 1 | 3 | + +----------------------------+-----------+--------+------------+ + | RFC (2008) | 6.5 | 11 | 21.75 | + +----------------------------+-----------+--------+------------+ + | RFC (2008), until 2009 | 1 | 2.5 | 4.5 | + +----------------------------+-----------+--------+------------+ + | RFC (2008), 2018 and after | 0 | 0 | 2 | + +----------------------------+-----------+--------+------------+ + | RFC (1998) | 4.75 | 13.5 | 32.25 | + +----------------------------+-----------+--------+------------+ + | RFC (1998), until 1999 | 0 | 2 | 4.25 | + +----------------------------+-----------+--------+------------+ + | RFC (1998), 2018 and after | 0 | 0 | 1 | + +----------------------------+-----------+--------+------------+ + + Table 13 + + The total numbers show new documents with fewer citations than the + older ones. This can be explained to some degree by the passage of + time. If we restrict the analysis to the number of citations accrued + in the year of publishing and the year after that, we still see about + the same distribution for the three samples. + + We also see that the number of references to RFCs fades over time. + Only the most popular of the RFC produced in 1998 are still cited in + 2019. + +5.3. Citations versus Deployments + + The following table shows side by side the number of citations as + measured in Section 5.1 and the estimation of deployment as indicated + in Section 3. + + +============+============+===========+============+ + | RFC (2018) | Status | Citations | Deployment | + +============+============+===========+============+ + | 8411 | Info | 1 | medium | + +------------+------------+-----------+------------+ + | 8456 | Info | 1 | medium | + +------------+------------+-----------+------------+ + | 8446 | PS | 418 | high | + +------------+------------+-----------+------------+ + | 8355 | Info | 3 | medium | + +------------+------------+-----------+------------+ + | 8441 | PS | 1 | high | + +------------+------------+-----------+------------+ + | 8324 | Info (ISE) | 0 | N/A | + +------------+------------+-----------+------------+ + | 8377 | PS | 0 | unknown | + +------------+------------+-----------+------------+ + | 8498 | Info | 0 | unknown | + +------------+------------+-----------+------------+ + | 8479 | Info (ISE) | 0 | one | + +------------+------------+-----------+------------+ + | 8453 | Info | 3 | unknown | + +------------+------------+-----------+------------+ + | 8429 | BCP | 0 | some | + +------------+------------+-----------+------------+ + | 8312 | Info | 25 | high | + +------------+------------+-----------+------------+ + | 8492 | Info (ISE) | 4 | one | + +------------+------------+-----------+------------+ + | 8378 | Exp | 1 | some | + +------------+------------+-----------+------------+ + | 8361 | PS | 0 | one | + +------------+------------+-----------+------------+ + | 8472 | PS | 1 | medium | + +------------+------------+-----------+------------+ + | 8471 | PS | 1 | medium | + +------------+------------+-----------+------------+ + | 8466 | PS | 0 | unknown | + +------------+------------+-----------+------------+ + | 8362 | PS | 1 | medium | + +------------+------------+-----------+------------+ + | 8468 | Info | 1 | some | + +------------+------------+-----------+------------+ + + Table 14 + + From looking at these results, it is fairly obvious that citation + counts cannot be used as proxies for the "value" of an RFC. In our + sample, the two RFCs that have high citation counts were both widely + deployed, and can certainly be described as successful, but we also + see many RFCs that saw significant deployment without garnering a + high level of citations. + + Citation counts are driven by academic interest, but are only loosely + correlated with actual deployment. We saw that [RFC8446] was widely + cited in part because the standardization process involved many + researchers, and that the high citation count of [RFC8312] is largely + due to the academic interest in evaluating congestion control + protocols. If we look at previous years, the most cited RFC in the + 2008 sample is [RFC5326], an experimental RFC defining security + extensions to an experimental delay tolerant transport protocol. + This protocol does not carry a significant proportion of Internet + traffic, but has been the object of a fair number of academic + studies. + + The citation process tends to privilege the first expression of a + concept. We see that with the most cited RFC in the 1998 set is + [RFC2267], an informational RFC defining Network Ingress Filtering + that was obsoleted in May 2000 by [RFC2827]. It is still cited + frequently in 2018 and 2019, regardless of its formal status in the + RFC Series. We see the same effect at work with [RFC8441], which + garners very few citations although it updates [RFC6455] that has a + large number of citations. The same goes for [RFC8468], which is + sparsely cited while the [RFC2330] is widely cited. Just counting + citations will not indicate whether developers still use an old + specification or have adopted the revised RFC. + +5.4. Citations versus Web References + + Web references might be another indicator of the popularity of an + RFC. In order to evaluate these references, we list here the number + of results returned by searches on Google and Bing, looking for the + search term "RFCnnnn" (e.g., "RFC8411"), and copying the number of + results returned by the search engines. The table below presents the + results of these searches, performed on April 4, 2020. + + +============+============+===========+========+=======+ + | RFC (2018) | Status | Citations | Google | Bing | + +============+============+===========+========+=======+ + | 8411 | Info | 1 | 301 | 94 | + +------------+------------+-----------+--------+-------+ + | 8456 | Info | 1 | 266 | 8456 | + +------------+------------+-----------+--------+-------+ + | 8446 | PS | 418 | 25900 | 47800 | + +------------+------------+-----------+--------+-------+ + | 8355 | Info | 3 | 521 | 114 | + +------------+------------+-----------+--------+-------+ + | 8441 | PS | 1 | 2430 | 59500 | + +------------+------------+-----------+--------+-------+ + | 8324 | Info (ISE) | 0 | 393 | 138 | + +------------+------------+-----------+--------+-------+ + | 8377 | PS | 0 | 264 | 10900 | + +------------+------------+-----------+--------+-------+ + | 8498 | Info | 0 | 335 | 10100 | + +------------+------------+-----------+--------+-------+ + | 8479 | Info (ISE) | 0 | 564 | 11000 | + +------------+------------+-----------+--------+-------+ + | 8453 | Info | 3 | 817 | 11400 | + +------------+------------+-----------+--------+-------+ + | 8429 | BCP | 0 | 391 | 41600 | + +------------+------------+-----------+--------+-------+ + | 8312 | Info | 25 | 1620 | 2820 | + +------------+------------+-----------+--------+-------+ + | 8492 | Info (ISE) | 4 | 323 | 9400 | + +------------+------------+-----------+--------+-------+ + | 8378 | Exp | 1 | 418 | 11600 | + +------------+------------+-----------+--------+-------+ + | 8361 | PS | 0 | 499 | 92 | + +------------+------------+-----------+--------+-------+ + | 8472 | PS | 1 | 496 | 169 | + +------------+------------+-----------+--------+-------+ + | 8471 | PS | 1 | 1510 | 11600 | + +------------+------------+-----------+--------+-------+ + | 8466 | PS | 0 | 766 | 173 | + +------------+------------+-----------+--------+-------+ + | 8362 | PS | 1 | 67 | 147 | + +------------+------------+-----------+--------+-------+ + | 8468 | Info | 1 | 453 | 127 | + +------------+------------+-----------+--------+-------+ + + Table 15 + + The result counts from Bing are sometimes surprising. Why would RFC + 8441 gather 59,500 web references? Looking at the results in detail, + we find a mix of data. Some of them are logs of development projects + implementing Web Sockets, which is exactly what we are looking for, + but others appear spurious. For example, a shop selling rugby + jerseys is listed because its phone number ends with "8441". Other + pages were listed because street numbers or product numbers matched + the RFC number. The same type of collision may explain the large + reference counts on Bing for RFCs 8377, 8498, 8479, 8453, 8429, 8378, + and 8471. The result counts on Bing do not appear to provide a good + metric. + + On Google, all RFCs garner at least a 250 references, largely because + the whole RFC catalog is replicated on a large number of web servers. + Deviations from that baseline are largely correlated with the number + of citations in the Semantic Scholar, with a couple of exception: RFC + 8441 and RFC 8471 garner more references than the low citation counts + would predict. Looking at the results, we find many references in + development databases explaining how these protocols are implemented + in various code bases and open source projects. This means that + counting Google results would give some indication about an RFC's + popularity, complementing the citation counts. + + There are some practical problems in using the counts of Google + results. Google searches are personalized, the results depend on the + source of the queries, and the counts may vary as well. The search + results depend on the search algorithm, and there is no guarantee + that counts will not change when the algorithm changes. On the other + hand, the results do indicate that some of the RFCs in our sample are + being used by developers or in deployments. + +6. Observations and Next Steps + + The author's goal was to get a personal understanding of the "chain + of production" of the RFCs, and in particular to look at the various + causes of delays in the process. As shown in Section 4, the average + RFC was produced in 3 years and 4 months, which is similar to what + was found in the 2008 sample, but more than three times larger than + the delays for the 1998 sample. + + The working group process appears to be the main source of delays. + Efforts to diminish delays should probably focus there, instead of on + the IETF and IESG reviews or the RFC production. For the RFC + production phase, most of the variability originates in the AUTH48 + process, which is influenced by a variety of factors such as number + of authors or level of engagement of these authors. + + Most of the delay is spent in the working group, but the IETF + Datatracker does not hold much information about what happens inside + the working groups. For example, events like Working Group Last + Calls were not recorded in the history of the selected drafts + available in the Datatracker. Such information would have been + interesting. Of course, requiring that information would create an + administrative burden, so there is clearly a trade-off between + requiring more work from working group chairs and providing better + data for process analysis. (It appears that this information can be + available in the Datatracker for more recent drafts, if the WG chairs + use the Datatracker properly.) + + The Independent Stream operates as expected. The majority of the + authors of the Independent Stream RFCs appear to be in IETF insiders, + but there is significant amount of engagement by outside parties. + + The analysis of citations in Section 5.1 shows that citation numbers + are a very poor indication of the "value" of an RFC. Citation + numbers measure the engagement of academic researchers with specific + topics, but have little correlation with the level of adoption and + deployment of a specific RFC. The result counts of Google searches + do capture references outside academia, such as logs of development + projects. This might be informative, but it is not clear that the + counts would not change over time due to algorithm changes or + personalization. + + This document analyses a small sample of RFCs "in depth". This + allowed gathering of detailed feedback on the process and the + deployments. On the other hand, much of the data on delays is + available from the IETF Datatracker. It may be worth considering + adding an automated reporting of delay metrics in the IETF + Datatracker. + + This document only considers the RFCs that were published in a given + year. This approach can be criticized as introducing a form of + "survivor bias". There are many drafts proposed to the IETF, and + only a fraction of them end up being published as RFCs. On one hand, + this is expected, because part of the process is to triage between + ideas that can gather consensus and those that don't. On the other + hand, we don't know whether that triage is too drastic and has + discouraged progress on good ideas. + + One way to evaluate the triage process would be to look at + publication attempts that were abandoned -- for example, drafts that + expired without progressing or being replaced. The sampling + methodology could also be used for that purpose. Pick maybe 20 + drafts at random, among those abandoned in a target year, and + investigate why they were abandoned. Was it because better solutions + emerged in the working group? Or maybe because the authors + discovered a flaw in their proposal? Or was it because some + factional struggle blocked a good idea? Was the idea pursued in a + different venue? Hopefully, someone will try this kind of + investigation. + +7. Security Considerations + + This document does not specify any protocol. + + We might want to analyze whether security issues were discovered + after publication of specific standards. + +8. IANA Considerations + + This document has no IANA actions. + + Preliminary analysis does not indicate that IANA is causing any + particular delay in the RFC publication process. + +9. Informative References + + [IETFCOUNT] + IETF, "Past IETF Meetings", + <https://www.ietf.org/how/meetings/past/>. + + [RFC2267] Ferguson, P. and D. Senie, "Network Ingress Filtering: + Defeating Denial of Service Attacks which employ IP Source + Address Spoofing", RFC 2267, DOI 10.17487/RFC2267, January + 1998, <https://www.rfc-editor.org/info/rfc2267>. + + [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, + "Framework for IP Performance Metrics", RFC 2330, + DOI 10.17487/RFC2330, May 1998, + <https://www.rfc-editor.org/info/rfc2330>. + + [RFC2827] Ferguson, P. and D. Senie, "Network Ingress Filtering: + Defeating Denial of Service Attacks which employ IP Source + Address Spoofing", BCP 38, RFC 2827, DOI 10.17487/RFC2827, + May 2000, <https://www.rfc-editor.org/info/rfc2827>. + + [RFC5326] Ramadas, M., Burleigh, S., and S. Farrell, "Licklider + Transmission Protocol - Specification", RFC 5326, + DOI 10.17487/RFC5326, September 2008, + <https://www.rfc-editor.org/info/rfc5326>. + + [RFC6455] Fette, I. and A. Melnikov, "The WebSocket Protocol", + RFC 6455, DOI 10.17487/RFC6455, December 2011, + <https://www.rfc-editor.org/info/rfc6455>. + + [RFC8312] Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and + R. Scheffenegger, "CUBIC for Fast Long-Distance Networks", + RFC 8312, DOI 10.17487/RFC8312, February 2018, + <https://www.rfc-editor.org/info/rfc8312>. + + [RFC8324] Klensin, J., "DNS Privacy, Authorization, Special Uses, + Encoding, Characters, Matching, and Root Structure: Time + for Another Look?", RFC 8324, DOI 10.17487/RFC8324, + February 2018, <https://www.rfc-editor.org/info/rfc8324>. + + [RFC8355] Filsfils, C., Ed., Previdi, S., Ed., Decraene, B., and R. + Shakir, "Resiliency Use Cases in Source Packet Routing in + Networking (SPRING) Networks", RFC 8355, + DOI 10.17487/RFC8355, March 2018, + <https://www.rfc-editor.org/info/rfc8355>. + + [RFC8361] Hao, W., Li, Y., Durrani, M., Gupta, S., and A. Qu, + "Transparent Interconnection of Lots of Links (TRILL): + Centralized Replication for Active-Active Broadcast, + Unknown Unicast, and Multicast (BUM) Traffic", RFC 8361, + DOI 10.17487/RFC8361, April 2018, + <https://www.rfc-editor.org/info/rfc8361>. + + [RFC8362] Lindem, A., Roy, A., Goethals, D., Reddy Vallem, V., and + F. Baker, "OSPFv3 Link State Advertisement (LSA) + Extensibility", RFC 8362, DOI 10.17487/RFC8362, April + 2018, <https://www.rfc-editor.org/info/rfc8362>. + + [RFC8377] Eastlake 3rd, D., Zhang, M., and A. Banerjee, "Transparent + Interconnection of Lots of Links (TRILL): Multi-Topology", + RFC 8377, DOI 10.17487/RFC8377, July 2018, + <https://www.rfc-editor.org/info/rfc8377>. + + [RFC8378] Moreno, V. and D. Farinacci, "Signal-Free Locator/ID + Separation Protocol (LISP) Multicast", RFC 8378, + DOI 10.17487/RFC8378, May 2018, + <https://www.rfc-editor.org/info/rfc8378>. + + [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., + Decraene, B., Litkowski, S., and R. Shakir, "Segment + Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, + July 2018, <https://www.rfc-editor.org/info/rfc8402>. + + [RFC8410] Josefsson, S. and J. Schaad, "Algorithm Identifiers for + Ed25519, Ed448, X25519, and X448 for Use in the Internet + X.509 Public Key Infrastructure", RFC 8410, + DOI 10.17487/RFC8410, August 2018, + <https://www.rfc-editor.org/info/rfc8410>. + + [RFC8411] Schaad, J. and R. Andrews, "IANA Registration for the + Cryptographic Algorithm Object Identifier Range", + RFC 8411, DOI 10.17487/RFC8411, August 2018, + <https://www.rfc-editor.org/info/rfc8411>. + + [RFC8429] Kaduk, B. and M. Short, "Deprecate Triple-DES (3DES) and + RC4 in Kerberos", BCP 218, RFC 8429, DOI 10.17487/RFC8429, + October 2018, <https://www.rfc-editor.org/info/rfc8429>. + + [RFC8441] McManus, P., "Bootstrapping WebSockets with HTTP/2", + RFC 8441, DOI 10.17487/RFC8441, September 2018, + <https://www.rfc-editor.org/info/rfc8441>. + + [RFC8446] Rescorla, E., "The Transport Layer Security (TLS) Protocol + Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018, + <https://www.rfc-editor.org/info/rfc8446>. + + [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for + Abstraction and Control of TE Networks (ACTN)", RFC 8453, + DOI 10.17487/RFC8453, August 2018, + <https://www.rfc-editor.org/info/rfc8453>. + + [RFC8455] Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V., + and S. Banks, "Terminology for Benchmarking Software- + Defined Networking (SDN) Controller Performance", + RFC 8455, DOI 10.17487/RFC8455, October 2018, + <https://www.rfc-editor.org/info/rfc8455>. + + [RFC8456] Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V., + and S. Banks, "Benchmarking Methodology for Software- + Defined Networking (SDN) Controller Performance", + RFC 8456, DOI 10.17487/RFC8456, October 2018, + <https://www.rfc-editor.org/info/rfc8456>. + + [RFC8466] Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG + Data Model for Layer 2 Virtual Private Network (L2VPN) + Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October + 2018, <https://www.rfc-editor.org/info/rfc8466>. + + [RFC8468] Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V. + Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for + the IP Performance Metrics (IPPM) Framework", RFC 8468, + DOI 10.17487/RFC8468, November 2018, + <https://www.rfc-editor.org/info/rfc8468>. + + [RFC8471] Popov, A., Ed., Nystroem, M., Balfanz, D., and J. Hodges, + "The Token Binding Protocol Version 1.0", RFC 8471, + DOI 10.17487/RFC8471, October 2018, + <https://www.rfc-editor.org/info/rfc8471>. + + [RFC8472] Popov, A., Ed., Nystroem, M., and D. Balfanz, "Transport + Layer Security (TLS) Extension for Token Binding Protocol + Negotiation", RFC 8472, DOI 10.17487/RFC8472, October + 2018, <https://www.rfc-editor.org/info/rfc8472>. + + [RFC8479] Mavrogiannopoulos, N., "Storing Validation Parameters in + PKCS#8", RFC 8479, DOI 10.17487/RFC8479, September 2018, + <https://www.rfc-editor.org/info/rfc8479>. + + [RFC8483] Song, L., Ed., Liu, D., Vixie, P., Kato, A., and S. Kerr, + "Yeti DNS Testbed", RFC 8483, DOI 10.17487/RFC8483, + October 2018, <https://www.rfc-editor.org/info/rfc8483>. + + [RFC8492] Harkins, D., Ed., "Secure Password Ciphersuites for + Transport Layer Security (TLS)", RFC 8492, + DOI 10.17487/RFC8492, February 2019, + <https://www.rfc-editor.org/info/rfc8492>. + + [RFC8493] Kunze, J., Littman, J., Madden, E., Scancella, J., and C. + Adams, "The BagIt File Packaging Format (V1.0)", RFC 8493, + DOI 10.17487/RFC8493, October 2018, + <https://www.rfc-editor.org/info/rfc8493>. + + [RFC8498] Mohali, M., "A P-Served-User Header Field Parameter for an + Originating Call Diversion (CDIV) Session Case in the + Session Initiation Protocol (SIP)", RFC 8498, + DOI 10.17487/RFC8498, February 2019, + <https://www.rfc-editor.org/info/rfc8498>. + + [RFCYEAR] RFC Editor, "Number of RFC Published per YEAR", + <https://www.rfc-editor.org/rfcs-per-year/>. + + [SSCH] Allen Institute for AI, "Semantic Scholar | AI-Powered + Research Tool", <https://www.semanticscholar.org/>. + + [TI-LFA] Litkowski, S., Bashandy, A., Filsfils, C., Decraene, B., + and D. Voyer, "Topology Independent Fast Reroute using + Segment Routing", Work in Progress, Internet-Draft, draft- + ietf-rtgwg-segment-routing-ti-lfa-05, 15 November 2020, + <https://tools.ietf.org/html/draft-ietf-rtgwg-segment- + routing-ti-lfa-05>. + + [TLS13IMP] TLS WG, "TLS 1.3 Implementations", commit dcb7890, 14 + October 2019, <https://github.com/tlswg/tlswg- + wiki/blob/master/IMPLEMENTATIONS.md>. + + [TRKR] IETF, "IETF Datatracker", <https://datatracker.ietf.org/>. + +Acknowledgements + + Many thanks to the authors of the selected RFCs who were willing to + provide feedback on the process: Michael Ackermann, Zafar Ali, Sarah + Banks, Bruno Decraene, Lars Eggert, Nalini Elkins, Joachim Fabini, + Dino Farinacci, Clarence Filsfils, Sujay Gupta, Dan Harkins, Vinayak + Hegde, Benjamin Kaduk, John Klensin, Acee Lindem, Nikos + Mavrogiannopoulos, Patrick McManus, Victor Moreno, Al Morton, Andrei + Popov, Eric Rescorla, Michiko Short, Bhuvaneswaran Vengainathan, Lao + Weiguo, and Li Yizhou. Many thanks to Adrian Farrel for his useful + advice, to Stephen Farrell and Colin Perkins for their guidance on + the use of citations, and to Dave Crocker for a comprehensive review. + Thanks also to Alice Russo and the RFC Editor team for their work + improving this document and checking the accuracy of the data. + +Author's Address + + Christian Huitema + Private Octopus Inc. + 427 Golfcourse Rd + Friday Harbor, WA 98250 + United States of America + + Email: huitema@huitema.net |