From 4bfd864f10b68b71482b35c818559068ef8d5797 Mon Sep 17 00:00:00 2001 From: Thomas Voss Date: Wed, 27 Nov 2024 20:54:24 +0100 Subject: doc: Add RFC documents --- doc/rfc/rfc6385.txt | 1291 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1291 insertions(+) create mode 100644 doc/rfc/rfc6385.txt (limited to 'doc/rfc/rfc6385.txt') diff --git a/doc/rfc/rfc6385.txt b/doc/rfc/rfc6385.txt new file mode 100644 index 0000000..722131a --- /dev/null +++ b/doc/rfc/rfc6385.txt @@ -0,0 +1,1291 @@ + + + + + + +Internet Engineering Task Force (IETF) M. Barnes +Request for Comments: 6385 Polycom +Category: Informational A. Doria +ISSN: 2070-1721 Research Consultant + H. Alvestrand + Google + B. Carpenter + University of Auckland + October 2011 + + + General Area Review Team (Gen-ART) Experiences + +Abstract + + The General Area Review Team (Gen-ART) has been doing reviews of + Internet-Drafts (I-Ds) since 2004. This document discusses the + experience and the lessons learned over the past 7 years of this + process. The review team initially reviewed the I-Ds before each of + the IESG telechats. Since late 2005, review team members have been + assigned to review I-Ds during IETF Last Call, unless no IETF Last + Call is necessary for the I-D. The same reviewer then reviews any + updates when the I-D is placed on an IESG telechat agenda. + +Status of This Memo + + This document is not an Internet Standards Track specification; it is + published for informational purposes. + + This document is a product of the Internet Engineering Task Force + (IETF). It represents the consensus of the IETF community. It has + received public review and has been approved for publication by the + Internet Engineering Steering Group (IESG). Not all documents + approved by the IESG are a candidate for any level of Internet + Standard; see Section 2 of RFC 5741. + + Information about the current status of this document, any errata, + and how to provide feedback on it may be obtained at + http://www.rfc-editor.org/info/rfc6385. + + + + + + + + + + + + +Barnes, et al. Informational [Page 1] + +RFC 6385 Gen-ART October 2011 + + +Copyright Notice + + Copyright (c) 2011 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (http://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the Simplified BSD License. + +Table of Contents + + 1. Introduction ....................................................2 + 2. Who Are the Gen-ART Members? ....................................3 + 3. Goals of Gen-ART ................................................3 + 4. Gen-ART Reviews .................................................4 + 4.1. IETF Last Call Review Process ..............................4 + 4.2. IESG Telechat Review Process ...............................5 + 4.3. Form of Review .............................................5 + 4.4. Gen-ART Process Overview ...................................8 + 5. Secretarial Process ............................................10 + 5.1. Maintaining Review Spreadsheet ............................10 + 5.2. Last Call Assignment Procedure ............................12 + 5.3. Telechat Assignment Procedure .............................13 + 5.4. Capturing Reviews .........................................14 + 6. Results ........................................................14 + 7. Impressions ....................................................15 + 7.1. Reviewers' Impressions ....................................15 + 7.2. General Area Directors' Impressions .......................17 + 7.3. Gen-ART Secretaries' Impressions ..........................18 + 8. Needed Improvements ............................................18 + 9. Applicability ..................................................20 + 10. Security Considerations .......................................20 + 11. Acknowledgments ...............................................20 + 12. Informative References ........................................21 + +1. Introduction + + The General Area Review Team (Gen-ART) was created personally by the + General Area Director in 2004. This document discusses the + experiences and the lessons learned as the process has evolved over + the past 7 years. The process described in this document reflects + that which was in place at the time this document was published. + + + +Barnes, et al. Informational [Page 2] + +RFC 6385 Gen-ART October 2011 + + + This process is likely to continue to change over time. The review + team has been retained by subsequent General Area Directors. It has + no official role in the IETF standards process, except as a set of + individuals entitled, like everyone, to comment on Internet-Drafts + (I-Ds). Its volunteers, including a secretary and the team of + reviewers, serve at the invitation of the General AD. Both the + reviews and the reviewer names are public. + +2. Who Are the Gen-ART Members? + + The reviewers are typically individuals that have a fair amount of + experience within various IETF Working Groups (WGs), have authored WG + I-Ds and RFCs, and are often considered to be subject matter experts + (SMEs) in their particular areas of work. The current review team is + comprised of such technical experts, including several WG chairs as + well as past and current Internet Architecture Board (IAB) members. + Several past and current ADs have served as reviewers. Two past + General ADs have also served as reviewers, with one currently + serving. + + Members of the review team sometimes excuse themselves from the team + for various reasons, typically due to "day job" demands. However, + they often rejoin (for periods of time) as their schedules allow. + Also, some reviewers remain on the team, while their review workload + is decreased by assigning them just one I-D (at Last Call time) to + review each month. Section 11 provides a list of currently active + reviewers, along with those who have served on the review team in the + past. + +3. Goals of Gen-ART + + The original and continuing goal of the Gen-ART was, and is, to + offload from the General AD some of the burden of IESG reviews. The + load for the bi-weekly IESG reviews is often quite large; + occasionally, there are more than 20 I-Ds scheduled for discussion in + a single telechat. Thus, ADs also have less than a week's notice for + many of the I-Ds on the telechat agenda. + + Gen-ART was based on a model that had proved productive in the + Operations (OPS) Directorate: quick review close to telechat time, to + advise the AD on issues that remain serious. By having a trusted + group of reviewers read and evaluate the I-Ds, the General AD would + be able to concentrate on those I-Ds where there was a concern + expressed by the reviewer. The reviewers are expected to provide + feedback based on a whole set of criteria, including the criteria + + + + + + +Barnes, et al. Informational [Page 3] + +RFC 6385 Gen-ART October 2011 + + + summarized in Section 4.3. The overall objective is to ensure that + the I-Ds are well structured; can be easily understood, at least at a + high level; and provide a reasonable basis for implementation (for + I-Ds intended for the Standards Track). + + While other area (and WG) directorates/review teams existed prior to + Gen-ART and more have been established since Gen-ART, the roles of + each are fairly distinct. Thus, there is little overlap between the + goals and review criteria for the various review teams. It is also + very valuable for these other review teams to operate independently. + For example, when both Gen-ART reviews and Security Directorate + (SecDir) reviews raise the same sorts of concerns, it's a clear red + flag that the I-D needs more work before progressing. In addition, + due to the typical thoroughness (and objectiveness) of the various + review teams' reviews, the sponsoring AD and document shepherd are + often able to work with the editors/WG (and vice versa, depending + upon area and WG structure) to improve the overall quality of the + final I-D. It should also be noted that some ADs take the Gen-ART + reviews into consideration when preparing their own evaluations. + + Statistics from the Gen-ART reviews over the past 6+ years show a + trend of increased quality and readiness for progression of I-Ds by + the time they are placed on the telechat agenda. Additional + statistics are discussed in Section 6. + +4. Gen-ART Reviews + +4.1. IETF Last Call Review Process + + While the original process was meant only for reviews just before the + IESG telechat, the decision was made to include IETF Last Call (LC) + reviews in early 2005. Over time the latter has proven to be quite + effective. Assigning the I-Ds at IETF LC time typically gives a + reviewer more time to review an I-D. And, in some cases, the IETF LC + version is the one to appear on the telechat. Thus, by the time I-Ds + are added to the telechat agenda, a majority (typically at least 70%) + have already been reviewed. For those I-Ds that have been + up-versioned, the amount of time dedicated to re-review depends upon + the review summary for the IETF LC review. + + The assignments at IETF LC time evolved to minimize the gap between + LC announcements and assignment time, with the secretary doing LC + assignments every Thursday night. This typically allows the reviewer + at least one week and sometimes two to three weeks to complete the + review. The reviews are obviously most helpful when done on or + before the end of the IETF LC. + + + + + +Barnes, et al. Informational [Page 4] + +RFC 6385 Gen-ART October 2011 + + + The Last Call assignments are done on a fairly strict round-robin + basis to ensure a fair workload amongst all the reviewers. Reviewers + that are unavailable (vacations, etc.) during the review period + timeframe obviously are excluded from that round of assignments, but + remain in the same queue position for the next round. The order is + occasionally modified to avoid assigning an editor/author or WG chair + their own I-Ds. A reviewer may also NACK an assignment if they feel + they may have some bias (although corporate affiliations are not + considered to be sources of bias) or they don't feel they can review + the I-D in a timely manner. + + The assignment process is completely manual, although a spreadsheet + tremendously facilitates the process. The details are described in + Section 5. Ideally, this process could be automated. However, + manual intervention would still be required to maintain the + appropriate available reviewer list (unless reviewers took on the + task of maintaining their data in some sort of database). Further + details on the tools necessary to automate the entire process are + provided in Section 8. + +4.2. IESG Telechat Review Process + + The process for reviewing I-Ds when they appear on the IESG agenda is + as follows: + + o The "nearly final" IESG meeting agenda generally appears on + Thursday night, less than one week before the IESG telechat. The + Gen-ART secretary uses this as the input for the assignment + process. + + o For I-Ds reviewed at IETF Last Call, a new review is only asked + for if the I-D is revised. In this case the reviewer, typically + the person who did the Last Call review, only needs to check that + any open issues were resolved. Often the draft will not have + changed between IETF LC and the IESG telechat review. Section 4.4 + provides the step-by-step telechat review assignment process, with + specific details on the maintenance of the review assignment data, + which is in turn maintained in review spreadsheets (Section 5). + +4.3. Form of Review + + Rather than invent new guidelines, the Gen-ART requirements for the + form of a review stole liberally from "Careful Additional Review of + Documents (CARD) by Senior IETF Reviewers (SIRS)" [SIRS], making + adaptations for the special "late, quick review" case and the nature + of the General Area's concerns. + + + + + +Barnes, et al. Informational [Page 5] + +RFC 6385 Gen-ART October 2011 + + + Each review must start with a summary statement chosen from or + adapted from the following list: + + o This draft is ready for publication as a [type] RFC, where [type] + is Informational, Experimental, etc. (In some cases, the review + might recommend publication as a different [type] than requested + by the author.) + + o This draft is basically ready for publication, but has nits that + should be fixed before publication. + + o This draft is on the right track but has open issues, described in + the review. + + o This draft has serious issues, described in the review, and needs + to be rethought. + + o This draft has very fundamental issues, described in the review, + and further work is not recommended. + + o Unfortunately, I don't have the expertise to review this draft. + + The length of a review can vary greatly according to circumstances, + and it is considered acceptable for purely editorial comments to be + sent privately if it's obvious that the I-D needs substantial + revision. All substantive comments, however, must be included in the + public review. Wherever possible, comments should be written as + suggestions for improvement rather than as simple criticism. + Explicit references to prior work and prior IETF discussion should be + given whenever possible. + + Reviewers are asked to review for all kinds of problems, such as + basic architectural or security issues, Internet-wide impact, + technical nits, problems of form and format (such as IANA + Considerations or incorrect references), and editorial issues. Since + these reviews are on I-Ds that are supposed to be finished, the + review should consider "no issue too small" -- but should cover the + whole range, from the general architectural level to the editorial + level. + + All reviews should apply generally agreed-upon IETF criteria, + such as: + + o [RFC1958]: "Architectural Principles of the Internet" + + o [RFC3426]: "General Architectural and Policy Considerations" + + o [RFC3439]: "Some Internet Architectural Guidelines and Philosophy" + + + +Barnes, et al. Informational [Page 6] + +RFC 6385 Gen-ART October 2011 + + + o ID-Checklist: The "ID checklist" document maintained by the IESG + + o [RFC2223bis]: "Instructions to Request for Comments (RFC) Authors" + as updated by [RFC-STYLE]: "RFC Document Style" + + o [RFC5226]: BCP 26 - "Guidelines for Writing an IANA Considerations + Section in RFCs" + + o [RFC3552]: BCP 72 - "Guidelines for Writing RFC Text on Security + Considerations" + + o Any other applicable architectural or procedural documents. It is + considered important that reviews give precise references to such + criteria when relevant to a comment. + + Of special interest to the General area, because they do not fall + under any other area, are: + + o A clear description of why the I-D or protocol is useful to the + Internet. + + o Adherence to IETF formalities, such as capitalized "must", + "should", etc. in normative statements, per the ID-Checklist. + + o Useful and reasonable IANA considerations. Ensure that all + necessary registries are defined/referenced, and ensure definition + and compliance with IANA assignment criteria. + + o Correct dependencies for normative references. + + o That the I-D is written in reasonably clear English. + + o Checking the updates/obsoletes information. + + o Running idnits and checking the output. + + o Checking that things imported by reference, especially from other + RFCs, make sense (notably definitions of terms, security + considerations, and lists of criteria) and ensuring they are used + as intended by the referenced document. + + o That examples (e.g., Fully Qualified Domain Names (FQDNs), + telephone numbers, IP addresses) are taken from the right spaces. + + + + + + + + +Barnes, et al. Informational [Page 7] + +RFC 6385 Gen-ART October 2011 + + +4.4. Gen-ART Process Overview + + The following provides a general overview of the Gen-ART process + along with some basic rules associated with assignments. The very + precise details of the secretary's process are provided in Section 5. + + o The availability of reviewers and the order of assignments for the + next round of Last Call document assignments are updated weekly + and are available on the directory where all the assignments and + reviews are cached. + + o At telechat assignment time, all previously reviewed I-Ds are + assigned to the reviewer who reviewed them previously, assuming + that reviewer is available. Otherwise, these I-Ds are assigned to + a new person in the process described below. + + o The secretary attempts to avoid assigning I-Ds that might conflict + with other IETF roles such as WG chairs, other directorates, etc. + However, in the cases where the secretary doesn't note the + conflict, the reviewer should notify the secretary and Gen-ART + mailing list so another reviewer may be assigned. + + o It should be emphasized that assignment is never made according to + a reviewer's technical specialty. Even though it happens, when, + for example, routing I-Ds fall on routing experts or MIB documents + fall on MIB doctors, it is coincidental. To the reviewer, the + choice looks random. + + o There is an attempt to evenly distribute I-Ds amongst reviewers at + LC time by using a round-robin process, starting from where the + previous week's assignments stopped. + + o Typically, there is no attempt made to actually equalize the load, + as the length and complexity of the I-Ds are not taken into + account in this process. (Thus, a reviewer could end up with a + couple of hundred-page I-Ds, but this is statistically rare.) + However, in the case of a reviewer that might receive more than + one new LC I-D at one time, the secretary does try to ensure that + both are not large I-Ds. + + o Once the assignments are made, the web pages that list the reviews + and the assignments are posted. Since the telechat agenda is not + published until the end of the day on the Thursdays prior to the + telechats (i.e., one week prior), the secretary needs to complete + the assignments on that Thursday evening. This often requires + working later in the evening and also requires an Internet + connection even when traveling. + + + + +Barnes, et al. Informational [Page 8] + +RFC 6385 Gen-ART October 2011 + + + o If the reviewers notice any problems or conflict of interest, a + bargaining process, shifting I-Ds from one reviewer to another, + takes place. The secretary updates the assignment files with any + new assignments. + + o Once the review has been completed, the reviewer sends the review + to the Gen-ART list, ideally using the template provided in the + review assignment emails. Typically, reviews are also sent to + authors, the responsible AD, and the WG chairs/document shepherd. + The only case where this might not be done is when there are no + issues found for a re-review and none had been found on an initial + review. Sending the review to the authors, ADs, and/or WG chairs/ + Proto Shepherds was originally voluntary but is now considered + standard practice. Reviewers may also send the reviews to the + IETF discussion list, but that is entirely at the discretion of + the reviewer, in which case the author must be copied on the + review to ensure they see any follow-up discussion. Reviewers may + also send the comments to the WG; however, this typically causes + the review to end up in the moderation queue, as most reviewers do + not want to subscribe to the WG lists for the I-Ds they review. + Thus, it is expected that any of the original recipients (i.e., + authors, WG chairs/document shepherd, or responsible AD) may + forward the review to the WG mailing list if they believe it is + necessary. In the past, sending these reviews resulted in + confusion among the authors, who may not have been expecting a + Gen-ART review and may not be familiar with Gen-ART. Thus, + reviewers are reminded to prepend to the email the description of + Gen-ART and the purpose of the review. This information is part + of the standard template provided in the review assignment emails. + + o The secretary gathers the reviews, sometimes edits them for + format, and records the review in the spreadsheet on the web + pages, including the synopsis. This is typically done on + Thursday. This is one aspect of the process that can be easily + delegated such that one volunteer uploads all the reviews and then + the secretary need only update the fields in the spreadsheet. If + the reviewer has not provided a synopsis ("Summary" field in the + template), the secretary makes a best guess based on the review + details. Note that in most cases the reviewers do include a + synopsis. + + o Ideally, the reviews should be posted to the Gen-ART mailing list + by close of business on the East coast on Tuesday. This is + necessary to allow the General AD time to consider the reviews + prior to the telechat. If the reviews are received after Tuesday, + the review may not be read by the AD before the IESG telechat. + Due to time constraints, the spreadsheets containing review + summaries/assignments are only updated on Thursday evenings when + + + +Barnes, et al. Informational [Page 9] + +RFC 6385 Gen-ART October 2011 + + + the new LC assignments and upcoming telechat assignments are done. + Ideally, the reviews would get uploaded on the Tuesdays prior to + the telechat along with the updated spreadsheets. + + o If the AD concludes that the concerns raised by the reviewer + warrant placing a DISCUSS comment on the I-D, the AD will do so, + and the DISCUSS must be resolved before the I-D advances. + Usually, the reviewer will be involved in the resolution process, + but the responsibility for the DISCUSS rests with the AD. + +5. Secretarial Process + + This section summarizes the details of managing the review materials, + including the spreadsheet used to track all reviews and the HTML + files containing the review assignments. Please note that these + details represent a snapshot of a process that has been implemented + -- the details are very likely to change over time, in particular as + the needed improvements highlighted in Section 8 are carried out. + +5.1. Maintaining Review Spreadsheet + + A spreadsheet is used to enter all the I-Ds at the time of assignment + and to capture all the reviews. For IETF LC assignments, the + assignments are completed before adding the I-Ds to the spreadsheet + as described in Section 5.2. For telechat assignments, I-Ds are + obviously only added in the cases where there is no previous LC + assignment. For the other I-Ds, the appropriate fields are updated + as described in Section 5.3. + + All the reviews can be accessed from the spreadsheet via hyperlinks + from specific fields, as summarized below. The following information + is maintained in the spreadsheet (in the order listed): + + 1. "Chat/LC Date": Indicates either the date on which the LC review + is due or the date of the telechat. + + 2. "Document": Filename for the I-D, which includes a hyperlink to + the IETF I-D tracker. + + 3. "Assigned": Name of the reviewer assigned to that I-D. + + 4. "Category": This field contains one of the following self- + explanatory values: "Doc - WG", "Doc - Ind/AD", or "IETF LC". + Note that Gen-ART does not review I-Ds submitted directly to the + RFC Editor. The "IETF LC" value is of course entered for all + I-Ds at LC time. It is changed to one of the other appropriate + values, based on the information in the telechat agenda. + + + + +Barnes, et al. Informational [Page 10] + +RFC 6385 Gen-ART October 2011 + + + 5. "Previous Review": This includes a link to any previous reviews. + For example, when a doc appears on a telechat agenda, if an IETF + LC review was done, this field is updated with the review summary + for the LC review (i.e., the information from the "Current Review + Summary" as described below is copied to this column). The field + is set to "New" when an I-D is first assigned/added to the + spreadsheet. In the case of returns, this field has a value of + "Return" or "Return/IETF-LC" for I-Ds for which there is an LC + review. It should be noted that since Gen-ART started doing + reviews at LC time, there seem to be far fewer returns on the + agenda. + + 6. "Current Review Summary": This field includes the review type and + version number of the document that is to be reviewed or has been + reviewed (e.g., LC: -02). When the field also contains a review + summary after the review type/version number (e.g., Telechat: -06 + Ready), the active hyperlink points to the review. Occasionally, + a reviewer will re-review an I-D prior to its telechat + assignment, in which case it is added to the spreadsheet, but the + date does not change to maintain consistency in the date field, + since the reviews themselves contain the review date. + + The following summarizes the steps to add a new I-D to the + spreadsheet: + + 1. In order to optimize steps, blank rows are first inserted for the + number of new I-Ds to be added. + + 2. To minimize data entry, a row with default fields (including the + hyperlinks) is kept at the end of the file. There is a separate + default row for IETF LC versus telechat assignments. This row is + copied into each of the new blank rows. The dates are then + entered (this allows double-checking that all I-Ds from the + review assignments are accounted for, especially LC). + + 3. The I-D name is then copied to the name field as well as being + appended to the hyperlink for the "Review Summary" field. The + hyperlink is included as part of the default row. This minimizes + the steps to enter the reviews in the spreadsheet. + + 4. The data is also sorted by "Chat/LC Date", "Assigned", and + "Document". The file is then saved and closed. + + 5. The file is then reopened and saved as HTML. + + 6. The file is opened a second time and sorted by "Assigned", + "Chat/LC Date", and "Document" to provide the I-D reviewers an + easy way to find any outstanding assignments. + + + +Barnes, et al. Informational [Page 11] + +RFC 6385 Gen-ART October 2011 + + +5.2. Last Call Assignment Procedure + + The secretary can cache the Last Call assignments as they are + announced and/or check the IETF announcement mailing list archives. + The assignments are done on Thursday evening, along with any telechat + assignments. This optimizes the process in terms of batch changes to + files. + + The assignments are listed in an HTML file. The following are the + steps in creating that file: + + 1. The order of assignment is actually created the week before, with + the details below. Thus, before starting the new assignments, + the current file is saved for editing for the following week. + The current file-naming convention is "reviewersyymmdd-lc.html" + (e.g., for July 8th, 2010, the file reviewers100708-lc.html was + created, and the file for the following week is named + reviewers100715-lc.html). + + 2. Since the file is already prepared with the appropriate ordering + of reviewers, the assignments are done in the order of due dates. + The link to the I-D in the Datatracker is copied into the + assignment file along with the intended RFC status for each of + the new LC I-Ds. + + 3. The "Due Date" paragraph from the Last Call announcement is + shortened as follows: "IETF LC ends on:", keeping the date. + + 4. Once the assignment file is complete, the new I-Ds are added to + the spreadsheet as described above. + + 5. The assignment file for the next week is then updated to reflect + the next reviewer in the round-robin process, by simply cutting + and pasting the names in the list in a block and removing any + "one doc per month" reviewers (annotated with an "*") that have + already received their monthly assignment. If the next round of + assignments occurs at the beginning of a new month, the "one doc + per month" reviewers are added back into the list (in the normal + order -- alphabetically by first name). + + 6. The assignment files and updated spreadsheets are then cached on + the Gen-ART server. + + 7. An email providing a link to the assignment file along with the + updated spreadsheets is sent to the Gen-ART mailing list. This + email has a standard form, such that the reviewers can simply cut + and paste the template to include the Gen-ART context statement + and link to the FAQ. + + + +Barnes, et al. Informational [Page 12] + +RFC 6385 Gen-ART October 2011 + + +5.3. Telechat Assignment Procedure + + Since LC assignments are now the starting point for Gen-ART I-D + reviews, the telechat assignments are generally straightforward, as + the majority of the I-Ds are already in the spreadsheet. The + following details the steps: + + 1. The telechat agenda is typically available around 6PM PDT. In + order to create the assignment HTML file, the agenda is created + from the email announcing the upcoming telechat agenda. The + filename has the following format, with the date corresponding to + the telechat date (versus the date of assignment, as is the case + for Last Call assignments): "reviewersyymmdd.html". + + 2. Rows are added to the agenda for the reviewers' names. + + 3. The reviewers' names are then added to the weekly assignment + file. + + 4. As each reviewer is added to the assignment file, the review + spreadsheet is updated as follows: + + * "Chat/LC Date" is changed to the telechat date. + + * The link to the LC review, if available, is copied as the link + for the "Previous Review" column. + + * The "text" for the "Current Review" is updated to reflect the + new review type (i.e., Telechat) and version. + + 5. In the case of an I-D that did not go through IETF LC, a reviewer + is assigned using the order in the file to be used for Last Call + assignments for the next week. + + 6. Once the reviewer(s) have been determined, the LC assignment file + for the next week is updated. + + 7. Any new I-Ds are then added to the spreadsheet (and the updates + saved) per the steps described in Section 5.1. + + 8. The assignment files and updated spreadsheets are then cached on + the Gen-ART server. + + 9. An email providing a link to the assignment file along with the + updated spreadsheets is sent to the Gen-ART mailing list. This + email has a standard form, such that the reviewers can simply cut + and paste the template to include the Gen-ART context statement + and link to the FAQ. + + + +Barnes, et al. Informational [Page 13] + +RFC 6385 Gen-ART October 2011 + + +5.4. Capturing Reviews + + As noted in Section 4.4, the spreadsheet is typically updated with + the review summaries on Thursday evenings just prior to entering the + data for that week's LC and any telechat assignments. The following + summarizes the steps to capture the reviews: + + 1. Currently, an additional volunteer is assisting the secretary in + caching the email reviews as they arrive. + + 2. In the cases where the review is included inline in the body of + the email, the review is cut and pasted into a text file and + saved with the reviewer's last name appended to the filename -- + e.g., draft-ietf-xyz-00-smith.txt. + + 3. In the case where the review is included as an attachment to the + email, the file can be directly saved and uploaded. + + 4. The volunteer uploads the reviews by around 5PM CST on Thursdays; + thus, they are available to the secretary at the time that week's + assignments are done. This sequence is necessary to ensure the + information for I-Ds on the upcoming telechat is up to date. + + 5. The review summary is entered into the "Current Summary" field. + Note that the hyperlink to the review (added at assignment time) + will automatically work when the file is uploaded. + + 6. Once all the reviews have been entered and the spreadsheets + formatted, the review spreadsheet is saved and files uploaded per + the last three steps in Section 5.1. + +6. Results + + Over the past 7 years, the Gen-ART has provided reviewing services to + 3 ADs and has done around two thousand publicly available reviews. + The reviews have been executed with a team of around a dozen full- + time reviewers and other reviewers receiving one I-D assignment each + month. There are currently 9 reviewers in the latter category. The + full-time reviewers receive 2-3 assignments each month. In terms of + improving quality, the number of I-Ds that are now "Ready" at the + time of the telechat (since the reviews are now initiated at LC time) + has increased. The review term "Ready" means the reviewer believes + that the document has no outstanding editorial or technical issues. + Based on the data from 2007, there were over 250 I-Ds assigned at LC + time that went through IESG review. Of those 250 I-Ds, 82% of the LC + reviews (205 I-Ds) were completed. Of the completed reviews, 70% + (144 I-Ds) were "Ready" at the time of the telechat. Of those 144 + I-Ds, roughly 1/4 had been deemed "Ready" (with no nits) at LC time + + + +Barnes, et al. Informational [Page 14] + +RFC 6385 Gen-ART October 2011 + + + (based on a sample of 50 reviews). For the I-Ds that were not + reviewed at LC time, only about 1/4 of those were deemed "Ready" when + they were reviewed for the telechat. So, doing the Gen-ART reviews + at Last Call time does seem to improve the quality of the I-Ds for + the telechat. + +7. Impressions + + This section is divided into 3 subsections: the impressions as + gathered from the Gen-ART, the impressions of the ADs for whom they + worked, and the impressions of the secretaries of Gen-ART. + +7.1. Reviewers' Impressions + + The following list of comments are excerpted and edited from comments + sent in by the reviewers of Gen-ART in response to the request: + + "We'd like to ask you each to write a few lines about your personal + experience and lessons learned as a Gen-ART reviewer". + + o We really do find problems, but we don't find problems with + most I-Ds. + + o Comments seem to be in three areas: editorial/grammar, editorial/ + what-the-heck-does-this-mean, and actual problems. I'm seeing + fewer reviews in the first category, which is a good thing. + + o It is becoming rarer that we hear back "these guys have suffered + enough; I'm voting no objection" (I'm remembering an LDAP I-D that + had been around so long it had 2119 referenced AS A DRAFT -- some + people suffered a lot). + + o The direct assignment of reviews is necessary and effective. It + does not matter much as far as I can tell what scheme is used to + actually do the assignment. + + o Folks are very open to the reviews that come out of Gen-ART. This + somewhat surprised me, because I have seen resistance to outside + reviews in other cases. + + o The improvements that have come about (for example, one of my + latest, an I-D about the SIPPING conference) have made a big + difference to the comprehensibility and usability of the I-Ds -- + and provide a useful incentive to keep going. + + o Some form of review like this is desperately needed. While most + of the stuff we see is good, every once in a while really bad + errors have made their way all the way to IESG vote. + + + +Barnes, et al. Informational [Page 15] + +RFC 6385 Gen-ART October 2011 + + + o Reading this stuff is interesting. I like having a reason to read + a wide range of materials. + + o I am more than convinced that this can be, and is, a valuable + process. It is, in my opinion, a pity that Senior IETF Reviewers + (SIRS) and so on did not take off, because this late-stage + reviewing is a poor substitute for doing the same thing at a much + earlier stage. Very few of the drafts that have come past my + screen are truly fully ready for IESG review. It is actually a + joy to find the occasional nugget that is both well written and is + a proper technical job, such that the reviewer really can say + "This is ready". + + o I have certainly found the process intellectually stimulating! It + encourages me to take a wider interest in what is going on in the + IETF, but consumes a fair bit of time to do a proper job, and + requires a very wide knowledge to be able to properly catch the + cross-area implications: I hope (believe!) that this is something + that one gets better at with experience and doing a few of these + reviews. + + o There is probably a very limited pool of people who have both the + time and the inclination to keep on doing these reviews. It does + require a fair bit of dedication. + + o It is difficult to avoid correcting the English, even if that is + not really the point: Often, really bad English (whether as a + result of non-mother-tongue authors with limited grasp or mother- + tongue authors using informal language) obscures/corrupts what is + being said or just makes it impossible to read. + + o Mostly authors welcome the comments: I think most of them + understand the concept of "ego-free reviewing", and we have + generally been constructive rather than destructive. + + o Part of the job of Gen-ART is to think the unthinkable from + another point of view, to challenge (apparently undocumented) + assumptions, and apply experience from other fields. + + + + + + + + + + + + + +Barnes, et al. Informational [Page 16] + +RFC 6385 Gen-ART October 2011 + + +7.2. General Area Directors' Impressions + + It should be noted that these impressions are from multiple General + Area Directors; thus, the "I"s are not necessarily associated with a + specific AD. + + o It's essential. The reviewing load for the IESG DOES NOT + SCALE. + + o Without Gen-ART, I would be a much less effective General AD. + + o On a single fortnight example, the IESG had 21 drafts on the + agenda. It is just impossible (to conscientiously review all the + documents), and no wonder we sometimes miss serious issues. + + o So I think a distributed review team with about 30 trusted + reviewers needs to be institutionalized. I suspect that will need + to be formalized in a BCP sooner or later -- with their reviews + having a formal position in the standards process, and the + expectation that the whole IESG truly reviews all I-Ds being + relaxed. + + o We've learned that polite, well reasoned, constructive reviews are + very positively received by authors and WGs. Dismissive reviews + are counter-productive. And reviews sent in private eventually + show up in public, so it's better to go public at the start. + + o Normally, LC reviews are available in good time for the draft to + be revised before reaching the IESG agenda. It is important that + this happens, except for an emergency situation where the + responsible AD has good reason to place the draft on the agenda + immediately. In that case, it would be preferable for the AD to + inform the Gen-ART, so that the review can be expedited. + + o The other problem is a big detail -- between late Thursday or + early Friday when the secretary sends out the assignments, and + Wednesday when the General AD likes to start filling in ballots + based on the reviews received by close of business on Tuesday, + there are only three work days (plus possible volunteer time over + the weekend). Now even with only one I-D to review, that may be a + real challenge. Sometimes, a lucky reviewer will get 130 pages + (e.g., draft-ietf-nntpext-base-27). That doesn't compute. + + + + + + + + + +Barnes, et al. Informational [Page 17] + +RFC 6385 Gen-ART October 2011 + + + o There are some mechanical issues. The process followed is far too + manual. Everything needs to be robotic except for the judgment + calls about which reviewer gets which draft. Similarly, the + reviewer should be able to just paste the review into a web form, + click, and it's sent off to everyone appropriate and posted to the + review site. + +7.3. Gen-ART Secretaries' Impressions + + Serving as the secretary of Gen-ART is a worthwhile experience. From + a personal point of view, it gives the secretary an easy way to track + all of the work going through the IESG review process and see how the + work flowed through that process. Also, by reviewing and sometimes + creating the one-line abstracts that go on the review web page, the + secretary has an opportunity to really get a survey of the work being + approved by the IETF. + + The nature of these reviews is informal, and originally the reviews + were only intended for the General AD, though they were made public. + During 2004, there was little if any interaction between authors and + reviewers. There was some discussion during 2004 about trying to + expand the role of Gen-ART to a more formal, early-review model, + i.e., to evolve it into a form of SIRS. The original Gen-ART + secretary was against such a transformation, because she felt it + would put at risk something that worked. She believed that there + were risks inherent in formalizing the reviews and adding mechanisms + for standardizing the resultant review process. Another concern + involves the interaction between reviewers and authors. As discussed + above, it has become the practice to send reviews to the authors with + an explanation about the nature of Gen-ART reviews. While it is + clear that this has resulted in improved RFCs, it has also resulted + in increased workload for the reviewers. + + The secretary thinks that Gen-ART is an experiment that works well, + but she also believes it is fragile. The secretary is often + concerned about overburdening reviewers, and feels it is her + responsibility to keep them from burning out. Adding additional + reviewers to the review team would help to alleviate this concern. + In terms of the process, adding additional reviewers has minimal + impact. + +8. Needed Improvements + + The current size of the review team introduces a fairly heavy + workload for the individual reviewers that are not on the "one doc + per month" assignment cycle. Additional reviewers would be really + helpful to alleviate this workload. It is also important to note + that having additional reviewers adds minimal workload to the + + + +Barnes, et al. Informational [Page 18] + +RFC 6385 Gen-ART October 2011 + + + secretary's process; thus, the only blocking point is finding the + right folks that are interested in this type of volunteer role. As + noted in Section 7.2, 30 would be a good size for the review team. + This would cut the workload for an individual reviewer in half (given + the current model of 9 reviewers on the "one doc per month" + assignment cycle). + + Obviously, automation of the process would be a good thing. However, + Gen-ART secretaries are not necessarily highly motivated to + transition to a more automated approach until a significant part of + the process is automated. In more recent consideration of this + situation, it likely would be best to first automate the process of + entering the reviews, as that benefits the review team as a whole. + This automation should allow the reviewers to enter the reviews via a + web interface that would automatically generate the appropriate + emails -- quite similar to how the draft "Upload" tool currently + works. Also, given consistent naming conventions for the review + forms, this step would automate some of the process for the + secretary, as the reviews would automatically appear via the + spreadsheet hyperlinks, although there would still be a need to + manually enter the summary. But this would eliminate the need to + edit/normalize and upload files and, hopefully, eliminate the problem + encountered with unflowed text in emails and getting the review + properly formatted using some text editors. + + Section 5 was written to facilitate the process of determining tools + requirements, by providing the very detailed steps currently applied + to the process. As noted above, automating the upload of the reviews + could be a good first step. This is somewhat starting at the end of + the process. However, it seems that by automating in this direction, + we may have optimal results; since one of the earliest steps in the + process is the task of assigning reviewers, it likely needs the most + manual intervention, even with tools available. + + The current SecDir secretary does use some tools for assignments and + generating assignment emails. These tools could be considered for + use by the Gen-ART secretary. Since the SecDir reviews are not + cached and the information maintained for those reviews is less + detailed, there would be no reusability of that aspect. However, if + the Gen-ART spreadsheet can be automatically populated (with + assignments and completed reviews), the SecDir may be able to make + use of that same tool. + + + + + + + + + +Barnes, et al. Informational [Page 19] + +RFC 6385 Gen-ART October 2011 + + +9. Applicability + + As implemented today, the process has no formal role in the IETF + standards process. But as trust in the review team has built, and as + the team itself has learned to deliver reviews that are generally + well received, they have had a significant impact on I-D quality and + on timeliness. Rather than becoming a roadblock, they have (in + general) allowed the General AD to feel more confident in reaching + decisions and be more precise in resolving issues. Since reviews now + typically appear during IETF Last Call, the reviews, like the SecDir + reviews, are now generally expected. So, the role of the team has + evolved to be more formal than in the past (i.e., when this document + was first drafted in 2005). However, the handling of the reviews + remains entirely within the scope of the ADs, document shepherds, WG + chairs, and authors as they deem appropriate. + +10. Security Considerations + + Since this is an informational I-D about an open process, the + security considerations are specific to the process and users + involved in the process. The primary concern would be to limit the + people that have the ability to create and update the Gen-ART data/ + files to ensure that the integrity of the data is maintained. For + example, each Gen-ART reviewer should have a unique user name/ + password, just as folks do to access any other IETF-maintained data, + as appropriate. + +11. Acknowledgments + + Initial comments were received from the members of the Gen-ART, and + the experiences discussed in this document were derived from their + hard work over the last 7+ years. We thank the past reviewers of the + Gen-ART: + + Mark Allman + Harald Alvestrand (creator of Gen-ART) + Ron Bonica + Scott Brim + Gonzalo Camarillo + Sharon Chisholm + Spencer Dawkins + Lakshminath Dondeti + Avri Doria (past secretary) + Pasi Eronen + Dorothy Gellert + Eric Gray + Avashalom Houri + Glenn Kowack + + + +Barnes, et al. Informational [Page 20] + +RFC 6385 Gen-ART October 2011 + + + John Loughney + Lucy Lynch + Enrico Marocco + Michael Patton + Stefan Santesson + Robert Sparks + Tom Taylor + Sean Turner + Christian Vogt + Suzanne Woolf + + We also thank the current team of reviewers/secretary: + + Mary Barnes (past secretary, 2005-2010) + Richard Barnes + David Black + Ben Campbell + Brian Carpenter (past General AD) + Elwyn Davies + Francis Dupont + Roni Even + Miguel-Angel Garcia + Vijay Gurbani (assisting secretary to upload reviews) + Wassim Haddad + Joel Halpern + Suresh Krishnan + Peter McCann + Jean Mahoney (secretary as of Jan. 2011) + Alexey Melnikov + Kathleen Moriarty + +12. Informative References + + [RFC1958] Carpenter, B., Ed., "Architectural Principles of the + Internet", RFC 1958, June 1996. + + [RFC2223bis] Reynolds, J., Ed., and R. Braden, Ed., "Instructions to + Request for Comments (RFC) Authors", Work in Progress, + August 2004. + + [RFC-STYLE] Braden, R., Ginoza, S., and A. Hagens, "RFC Document + Style", September 2009, + . + + [RFC3426] Floyd, S., "General Architectural and Policy + Considerations", RFC 3426, November 2002. + + + + + +Barnes, et al. Informational [Page 21] + +RFC 6385 Gen-ART October 2011 + + + [RFC3439] Bush, R. and D. Meyer, "Some Internet Architectural + Guidelines and Philosophy", RFC 3439, December 2002. + + [RFC3552] Rescorla, E. and B. Korver, "Guidelines for Writing RFC + Text on Security Considerations", BCP 72, RFC 3552, + July 2003. + + [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing + an IANA Considerations Section in RFCs", BCP 26, + RFC 5226, May 2008. + + [SIRS] Carpenter, B. and D. Crocker, "Careful Additional + Review of Documents (CARD) by Senior IETF Reviewers + (SIRS)", Work in Progress, June 2003. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Barnes, et al. Informational [Page 22] + +RFC 6385 Gen-ART October 2011 + + +Authors' Addresses + + Mary Barnes + Polycom + TX + USA + + EMail: mary.ietf.barnes@gmail.com + + + Avri Doria + Research Consultant + Providence, RI + USA + + EMail: avri@acm.org + + + Harald Alvestrand + Google + Kungsbron 2 + 11122 Stockholm + SE + + EMail: harald@alvestrand.no + + + Brian E. Carpenter + University of Auckland + PB 92019 + Auckland, 1142 + New Zealand + + Phone: + EMail: brian.e.carpenter@gmail.com + + + + + + + + + + + + + + + + +Barnes, et al. Informational [Page 23] + -- cgit v1.2.3