summaryrefslogtreecommitdiff
path: root/doc/rfc/rfc6385.txt
diff options
context:
space:
mode:
Diffstat (limited to 'doc/rfc/rfc6385.txt')
-rw-r--r--doc/rfc/rfc6385.txt1291
1 files changed, 1291 insertions, 0 deletions
diff --git a/doc/rfc/rfc6385.txt b/doc/rfc/rfc6385.txt
new file mode 100644
index 0000000..722131a
--- /dev/null
+++ b/doc/rfc/rfc6385.txt
@@ -0,0 +1,1291 @@
+
+
+
+
+
+
+Internet Engineering Task Force (IETF) M. Barnes
+Request for Comments: 6385 Polycom
+Category: Informational A. Doria
+ISSN: 2070-1721 Research Consultant
+ H. Alvestrand
+ Google
+ B. Carpenter
+ University of Auckland
+ October 2011
+
+
+ General Area Review Team (Gen-ART) Experiences
+
+Abstract
+
+ The General Area Review Team (Gen-ART) has been doing reviews of
+ Internet-Drafts (I-Ds) since 2004. This document discusses the
+ experience and the lessons learned over the past 7 years of this
+ process. The review team initially reviewed the I-Ds before each of
+ the IESG telechats. Since late 2005, review team members have been
+ assigned to review I-Ds during IETF Last Call, unless no IETF Last
+ Call is necessary for the I-D. The same reviewer then reviews any
+ updates when the I-D is placed on an IESG telechat agenda.
+
+Status of This Memo
+
+ This document is not an Internet Standards Track specification; it is
+ published for informational purposes.
+
+ This document is a product of the Internet Engineering Task Force
+ (IETF). It represents the consensus of the IETF community. It has
+ received public review and has been approved for publication by the
+ Internet Engineering Steering Group (IESG). Not all documents
+ approved by the IESG are a candidate for any level of Internet
+ Standard; see Section 2 of RFC 5741.
+
+ Information about the current status of this document, any errata,
+ and how to provide feedback on it may be obtained at
+ http://www.rfc-editor.org/info/rfc6385.
+
+
+
+
+
+
+
+
+
+
+
+
+Barnes, et al. Informational [Page 1]
+
+RFC 6385 Gen-ART October 2011
+
+
+Copyright Notice
+
+ Copyright (c) 2011 IETF Trust and the persons identified as the
+ document authors. All rights reserved.
+
+ This document is subject to BCP 78 and the IETF Trust's Legal
+ Provisions Relating to IETF Documents
+ (http://trustee.ietf.org/license-info) in effect on the date of
+ publication of this document. Please review these documents
+ carefully, as they describe your rights and restrictions with respect
+ to this document. Code Components extracted from this document must
+ include Simplified BSD License text as described in Section 4.e of
+ the Trust Legal Provisions and are provided without warranty as
+ described in the Simplified BSD License.
+
+Table of Contents
+
+ 1. Introduction ....................................................2
+ 2. Who Are the Gen-ART Members? ....................................3
+ 3. Goals of Gen-ART ................................................3
+ 4. Gen-ART Reviews .................................................4
+ 4.1. IETF Last Call Review Process ..............................4
+ 4.2. IESG Telechat Review Process ...............................5
+ 4.3. Form of Review .............................................5
+ 4.4. Gen-ART Process Overview ...................................8
+ 5. Secretarial Process ............................................10
+ 5.1. Maintaining Review Spreadsheet ............................10
+ 5.2. Last Call Assignment Procedure ............................12
+ 5.3. Telechat Assignment Procedure .............................13
+ 5.4. Capturing Reviews .........................................14
+ 6. Results ........................................................14
+ 7. Impressions ....................................................15
+ 7.1. Reviewers' Impressions ....................................15
+ 7.2. General Area Directors' Impressions .......................17
+ 7.3. Gen-ART Secretaries' Impressions ..........................18
+ 8. Needed Improvements ............................................18
+ 9. Applicability ..................................................20
+ 10. Security Considerations .......................................20
+ 11. Acknowledgments ...............................................20
+ 12. Informative References ........................................21
+
+1. Introduction
+
+ The General Area Review Team (Gen-ART) was created personally by the
+ General Area Director in 2004. This document discusses the
+ experiences and the lessons learned as the process has evolved over
+ the past 7 years. The process described in this document reflects
+ that which was in place at the time this document was published.
+
+
+
+Barnes, et al. Informational [Page 2]
+
+RFC 6385 Gen-ART October 2011
+
+
+ This process is likely to continue to change over time. The review
+ team has been retained by subsequent General Area Directors. It has
+ no official role in the IETF standards process, except as a set of
+ individuals entitled, like everyone, to comment on Internet-Drafts
+ (I-Ds). Its volunteers, including a secretary and the team of
+ reviewers, serve at the invitation of the General AD. Both the
+ reviews and the reviewer names are public.
+
+2. Who Are the Gen-ART Members?
+
+ The reviewers are typically individuals that have a fair amount of
+ experience within various IETF Working Groups (WGs), have authored WG
+ I-Ds and RFCs, and are often considered to be subject matter experts
+ (SMEs) in their particular areas of work. The current review team is
+ comprised of such technical experts, including several WG chairs as
+ well as past and current Internet Architecture Board (IAB) members.
+ Several past and current ADs have served as reviewers. Two past
+ General ADs have also served as reviewers, with one currently
+ serving.
+
+ Members of the review team sometimes excuse themselves from the team
+ for various reasons, typically due to "day job" demands. However,
+ they often rejoin (for periods of time) as their schedules allow.
+ Also, some reviewers remain on the team, while their review workload
+ is decreased by assigning them just one I-D (at Last Call time) to
+ review each month. Section 11 provides a list of currently active
+ reviewers, along with those who have served on the review team in the
+ past.
+
+3. Goals of Gen-ART
+
+ The original and continuing goal of the Gen-ART was, and is, to
+ offload from the General AD some of the burden of IESG reviews. The
+ load for the bi-weekly IESG reviews is often quite large;
+ occasionally, there are more than 20 I-Ds scheduled for discussion in
+ a single telechat. Thus, ADs also have less than a week's notice for
+ many of the I-Ds on the telechat agenda.
+
+ Gen-ART was based on a model that had proved productive in the
+ Operations (OPS) Directorate: quick review close to telechat time, to
+ advise the AD on issues that remain serious. By having a trusted
+ group of reviewers read and evaluate the I-Ds, the General AD would
+ be able to concentrate on those I-Ds where there was a concern
+ expressed by the reviewer. The reviewers are expected to provide
+ feedback based on a whole set of criteria, including the criteria
+
+
+
+
+
+
+Barnes, et al. Informational [Page 3]
+
+RFC 6385 Gen-ART October 2011
+
+
+ summarized in Section 4.3. The overall objective is to ensure that
+ the I-Ds are well structured; can be easily understood, at least at a
+ high level; and provide a reasonable basis for implementation (for
+ I-Ds intended for the Standards Track).
+
+ While other area (and WG) directorates/review teams existed prior to
+ Gen-ART and more have been established since Gen-ART, the roles of
+ each are fairly distinct. Thus, there is little overlap between the
+ goals and review criteria for the various review teams. It is also
+ very valuable for these other review teams to operate independently.
+ For example, when both Gen-ART reviews and Security Directorate
+ (SecDir) reviews raise the same sorts of concerns, it's a clear red
+ flag that the I-D needs more work before progressing. In addition,
+ due to the typical thoroughness (and objectiveness) of the various
+ review teams' reviews, the sponsoring AD and document shepherd are
+ often able to work with the editors/WG (and vice versa, depending
+ upon area and WG structure) to improve the overall quality of the
+ final I-D. It should also be noted that some ADs take the Gen-ART
+ reviews into consideration when preparing their own evaluations.
+
+ Statistics from the Gen-ART reviews over the past 6+ years show a
+ trend of increased quality and readiness for progression of I-Ds by
+ the time they are placed on the telechat agenda. Additional
+ statistics are discussed in Section 6.
+
+4. Gen-ART Reviews
+
+4.1. IETF Last Call Review Process
+
+ While the original process was meant only for reviews just before the
+ IESG telechat, the decision was made to include IETF Last Call (LC)
+ reviews in early 2005. Over time the latter has proven to be quite
+ effective. Assigning the I-Ds at IETF LC time typically gives a
+ reviewer more time to review an I-D. And, in some cases, the IETF LC
+ version is the one to appear on the telechat. Thus, by the time I-Ds
+ are added to the telechat agenda, a majority (typically at least 70%)
+ have already been reviewed. For those I-Ds that have been
+ up-versioned, the amount of time dedicated to re-review depends upon
+ the review summary for the IETF LC review.
+
+ The assignments at IETF LC time evolved to minimize the gap between
+ LC announcements and assignment time, with the secretary doing LC
+ assignments every Thursday night. This typically allows the reviewer
+ at least one week and sometimes two to three weeks to complete the
+ review. The reviews are obviously most helpful when done on or
+ before the end of the IETF LC.
+
+
+
+
+
+Barnes, et al. Informational [Page 4]
+
+RFC 6385 Gen-ART October 2011
+
+
+ The Last Call assignments are done on a fairly strict round-robin
+ basis to ensure a fair workload amongst all the reviewers. Reviewers
+ that are unavailable (vacations, etc.) during the review period
+ timeframe obviously are excluded from that round of assignments, but
+ remain in the same queue position for the next round. The order is
+ occasionally modified to avoid assigning an editor/author or WG chair
+ their own I-Ds. A reviewer may also NACK an assignment if they feel
+ they may have some bias (although corporate affiliations are not
+ considered to be sources of bias) or they don't feel they can review
+ the I-D in a timely manner.
+
+ The assignment process is completely manual, although a spreadsheet
+ tremendously facilitates the process. The details are described in
+ Section 5. Ideally, this process could be automated. However,
+ manual intervention would still be required to maintain the
+ appropriate available reviewer list (unless reviewers took on the
+ task of maintaining their data in some sort of database). Further
+ details on the tools necessary to automate the entire process are
+ provided in Section 8.
+
+4.2. IESG Telechat Review Process
+
+ The process for reviewing I-Ds when they appear on the IESG agenda is
+ as follows:
+
+ o The "nearly final" IESG meeting agenda generally appears on
+ Thursday night, less than one week before the IESG telechat. The
+ Gen-ART secretary uses this as the input for the assignment
+ process.
+
+ o For I-Ds reviewed at IETF Last Call, a new review is only asked
+ for if the I-D is revised. In this case the reviewer, typically
+ the person who did the Last Call review, only needs to check that
+ any open issues were resolved. Often the draft will not have
+ changed between IETF LC and the IESG telechat review. Section 4.4
+ provides the step-by-step telechat review assignment process, with
+ specific details on the maintenance of the review assignment data,
+ which is in turn maintained in review spreadsheets (Section 5).
+
+4.3. Form of Review
+
+ Rather than invent new guidelines, the Gen-ART requirements for the
+ form of a review stole liberally from "Careful Additional Review of
+ Documents (CARD) by Senior IETF Reviewers (SIRS)" [SIRS], making
+ adaptations for the special "late, quick review" case and the nature
+ of the General Area's concerns.
+
+
+
+
+
+Barnes, et al. Informational [Page 5]
+
+RFC 6385 Gen-ART October 2011
+
+
+ Each review must start with a summary statement chosen from or
+ adapted from the following list:
+
+ o This draft is ready for publication as a [type] RFC, where [type]
+ is Informational, Experimental, etc. (In some cases, the review
+ might recommend publication as a different [type] than requested
+ by the author.)
+
+ o This draft is basically ready for publication, but has nits that
+ should be fixed before publication.
+
+ o This draft is on the right track but has open issues, described in
+ the review.
+
+ o This draft has serious issues, described in the review, and needs
+ to be rethought.
+
+ o This draft has very fundamental issues, described in the review,
+ and further work is not recommended.
+
+ o Unfortunately, I don't have the expertise to review this draft.
+
+ The length of a review can vary greatly according to circumstances,
+ and it is considered acceptable for purely editorial comments to be
+ sent privately if it's obvious that the I-D needs substantial
+ revision. All substantive comments, however, must be included in the
+ public review. Wherever possible, comments should be written as
+ suggestions for improvement rather than as simple criticism.
+ Explicit references to prior work and prior IETF discussion should be
+ given whenever possible.
+
+ Reviewers are asked to review for all kinds of problems, such as
+ basic architectural or security issues, Internet-wide impact,
+ technical nits, problems of form and format (such as IANA
+ Considerations or incorrect references), and editorial issues. Since
+ these reviews are on I-Ds that are supposed to be finished, the
+ review should consider "no issue too small" -- but should cover the
+ whole range, from the general architectural level to the editorial
+ level.
+
+ All reviews should apply generally agreed-upon IETF criteria,
+ such as:
+
+ o [RFC1958]: "Architectural Principles of the Internet"
+
+ o [RFC3426]: "General Architectural and Policy Considerations"
+
+ o [RFC3439]: "Some Internet Architectural Guidelines and Philosophy"
+
+
+
+Barnes, et al. Informational [Page 6]
+
+RFC 6385 Gen-ART October 2011
+
+
+ o ID-Checklist: The "ID checklist" document maintained by the IESG
+
+ o [RFC2223bis]: "Instructions to Request for Comments (RFC) Authors"
+ as updated by [RFC-STYLE]: "RFC Document Style"
+
+ o [RFC5226]: BCP 26 - "Guidelines for Writing an IANA Considerations
+ Section in RFCs"
+
+ o [RFC3552]: BCP 72 - "Guidelines for Writing RFC Text on Security
+ Considerations"
+
+ o Any other applicable architectural or procedural documents. It is
+ considered important that reviews give precise references to such
+ criteria when relevant to a comment.
+
+ Of special interest to the General area, because they do not fall
+ under any other area, are:
+
+ o A clear description of why the I-D or protocol is useful to the
+ Internet.
+
+ o Adherence to IETF formalities, such as capitalized "must",
+ "should", etc. in normative statements, per the ID-Checklist.
+
+ o Useful and reasonable IANA considerations. Ensure that all
+ necessary registries are defined/referenced, and ensure definition
+ and compliance with IANA assignment criteria.
+
+ o Correct dependencies for normative references.
+
+ o That the I-D is written in reasonably clear English.
+
+ o Checking the updates/obsoletes information.
+
+ o Running idnits and checking the output.
+
+ o Checking that things imported by reference, especially from other
+ RFCs, make sense (notably definitions of terms, security
+ considerations, and lists of criteria) and ensuring they are used
+ as intended by the referenced document.
+
+ o That examples (e.g., Fully Qualified Domain Names (FQDNs),
+ telephone numbers, IP addresses) are taken from the right spaces.
+
+
+
+
+
+
+
+
+Barnes, et al. Informational [Page 7]
+
+RFC 6385 Gen-ART October 2011
+
+
+4.4. Gen-ART Process Overview
+
+ The following provides a general overview of the Gen-ART process
+ along with some basic rules associated with assignments. The very
+ precise details of the secretary's process are provided in Section 5.
+
+ o The availability of reviewers and the order of assignments for the
+ next round of Last Call document assignments are updated weekly
+ and are available on the directory where all the assignments and
+ reviews are cached.
+
+ o At telechat assignment time, all previously reviewed I-Ds are
+ assigned to the reviewer who reviewed them previously, assuming
+ that reviewer is available. Otherwise, these I-Ds are assigned to
+ a new person in the process described below.
+
+ o The secretary attempts to avoid assigning I-Ds that might conflict
+ with other IETF roles such as WG chairs, other directorates, etc.
+ However, in the cases where the secretary doesn't note the
+ conflict, the reviewer should notify the secretary and Gen-ART
+ mailing list so another reviewer may be assigned.
+
+ o It should be emphasized that assignment is never made according to
+ a reviewer's technical specialty. Even though it happens, when,
+ for example, routing I-Ds fall on routing experts or MIB documents
+ fall on MIB doctors, it is coincidental. To the reviewer, the
+ choice looks random.
+
+ o There is an attempt to evenly distribute I-Ds amongst reviewers at
+ LC time by using a round-robin process, starting from where the
+ previous week's assignments stopped.
+
+ o Typically, there is no attempt made to actually equalize the load,
+ as the length and complexity of the I-Ds are not taken into
+ account in this process. (Thus, a reviewer could end up with a
+ couple of hundred-page I-Ds, but this is statistically rare.)
+ However, in the case of a reviewer that might receive more than
+ one new LC I-D at one time, the secretary does try to ensure that
+ both are not large I-Ds.
+
+ o Once the assignments are made, the web pages that list the reviews
+ and the assignments are posted. Since the telechat agenda is not
+ published until the end of the day on the Thursdays prior to the
+ telechats (i.e., one week prior), the secretary needs to complete
+ the assignments on that Thursday evening. This often requires
+ working later in the evening and also requires an Internet
+ connection even when traveling.
+
+
+
+
+Barnes, et al. Informational [Page 8]
+
+RFC 6385 Gen-ART October 2011
+
+
+ o If the reviewers notice any problems or conflict of interest, a
+ bargaining process, shifting I-Ds from one reviewer to another,
+ takes place. The secretary updates the assignment files with any
+ new assignments.
+
+ o Once the review has been completed, the reviewer sends the review
+ to the Gen-ART list, ideally using the template provided in the
+ review assignment emails. Typically, reviews are also sent to
+ authors, the responsible AD, and the WG chairs/document shepherd.
+ The only case where this might not be done is when there are no
+ issues found for a re-review and none had been found on an initial
+ review. Sending the review to the authors, ADs, and/or WG chairs/
+ Proto Shepherds was originally voluntary but is now considered
+ standard practice. Reviewers may also send the reviews to the
+ IETF discussion list, but that is entirely at the discretion of
+ the reviewer, in which case the author must be copied on the
+ review to ensure they see any follow-up discussion. Reviewers may
+ also send the comments to the WG; however, this typically causes
+ the review to end up in the moderation queue, as most reviewers do
+ not want to subscribe to the WG lists for the I-Ds they review.
+ Thus, it is expected that any of the original recipients (i.e.,
+ authors, WG chairs/document shepherd, or responsible AD) may
+ forward the review to the WG mailing list if they believe it is
+ necessary. In the past, sending these reviews resulted in
+ confusion among the authors, who may not have been expecting a
+ Gen-ART review and may not be familiar with Gen-ART. Thus,
+ reviewers are reminded to prepend to the email the description of
+ Gen-ART and the purpose of the review. This information is part
+ of the standard template provided in the review assignment emails.
+
+ o The secretary gathers the reviews, sometimes edits them for
+ format, and records the review in the spreadsheet on the web
+ pages, including the synopsis. This is typically done on
+ Thursday. This is one aspect of the process that can be easily
+ delegated such that one volunteer uploads all the reviews and then
+ the secretary need only update the fields in the spreadsheet. If
+ the reviewer has not provided a synopsis ("Summary" field in the
+ template), the secretary makes a best guess based on the review
+ details. Note that in most cases the reviewers do include a
+ synopsis.
+
+ o Ideally, the reviews should be posted to the Gen-ART mailing list
+ by close of business on the East coast on Tuesday. This is
+ necessary to allow the General AD time to consider the reviews
+ prior to the telechat. If the reviews are received after Tuesday,
+ the review may not be read by the AD before the IESG telechat.
+ Due to time constraints, the spreadsheets containing review
+ summaries/assignments are only updated on Thursday evenings when
+
+
+
+Barnes, et al. Informational [Page 9]
+
+RFC 6385 Gen-ART October 2011
+
+
+ the new LC assignments and upcoming telechat assignments are done.
+ Ideally, the reviews would get uploaded on the Tuesdays prior to
+ the telechat along with the updated spreadsheets.
+
+ o If the AD concludes that the concerns raised by the reviewer
+ warrant placing a DISCUSS comment on the I-D, the AD will do so,
+ and the DISCUSS must be resolved before the I-D advances.
+ Usually, the reviewer will be involved in the resolution process,
+ but the responsibility for the DISCUSS rests with the AD.
+
+5. Secretarial Process
+
+ This section summarizes the details of managing the review materials,
+ including the spreadsheet used to track all reviews and the HTML
+ files containing the review assignments. Please note that these
+ details represent a snapshot of a process that has been implemented
+ -- the details are very likely to change over time, in particular as
+ the needed improvements highlighted in Section 8 are carried out.
+
+5.1. Maintaining Review Spreadsheet
+
+ A spreadsheet is used to enter all the I-Ds at the time of assignment
+ and to capture all the reviews. For IETF LC assignments, the
+ assignments are completed before adding the I-Ds to the spreadsheet
+ as described in Section 5.2. For telechat assignments, I-Ds are
+ obviously only added in the cases where there is no previous LC
+ assignment. For the other I-Ds, the appropriate fields are updated
+ as described in Section 5.3.
+
+ All the reviews can be accessed from the spreadsheet via hyperlinks
+ from specific fields, as summarized below. The following information
+ is maintained in the spreadsheet (in the order listed):
+
+ 1. "Chat/LC Date": Indicates either the date on which the LC review
+ is due or the date of the telechat.
+
+ 2. "Document": Filename for the I-D, which includes a hyperlink to
+ the IETF I-D tracker.
+
+ 3. "Assigned": Name of the reviewer assigned to that I-D.
+
+ 4. "Category": This field contains one of the following self-
+ explanatory values: "Doc - WG", "Doc - Ind/AD", or "IETF LC".
+ Note that Gen-ART does not review I-Ds submitted directly to the
+ RFC Editor. The "IETF LC" value is of course entered for all
+ I-Ds at LC time. It is changed to one of the other appropriate
+ values, based on the information in the telechat agenda.
+
+
+
+
+Barnes, et al. Informational [Page 10]
+
+RFC 6385 Gen-ART October 2011
+
+
+ 5. "Previous Review": This includes a link to any previous reviews.
+ For example, when a doc appears on a telechat agenda, if an IETF
+ LC review was done, this field is updated with the review summary
+ for the LC review (i.e., the information from the "Current Review
+ Summary" as described below is copied to this column). The field
+ is set to "New" when an I-D is first assigned/added to the
+ spreadsheet. In the case of returns, this field has a value of
+ "Return" or "Return/IETF-LC" for I-Ds for which there is an LC
+ review. It should be noted that since Gen-ART started doing
+ reviews at LC time, there seem to be far fewer returns on the
+ agenda.
+
+ 6. "Current Review Summary": This field includes the review type and
+ version number of the document that is to be reviewed or has been
+ reviewed (e.g., LC: -02). When the field also contains a review
+ summary after the review type/version number (e.g., Telechat: -06
+ Ready), the active hyperlink points to the review. Occasionally,
+ a reviewer will re-review an I-D prior to its telechat
+ assignment, in which case it is added to the spreadsheet, but the
+ date does not change to maintain consistency in the date field,
+ since the reviews themselves contain the review date.
+
+ The following summarizes the steps to add a new I-D to the
+ spreadsheet:
+
+ 1. In order to optimize steps, blank rows are first inserted for the
+ number of new I-Ds to be added.
+
+ 2. To minimize data entry, a row with default fields (including the
+ hyperlinks) is kept at the end of the file. There is a separate
+ default row for IETF LC versus telechat assignments. This row is
+ copied into each of the new blank rows. The dates are then
+ entered (this allows double-checking that all I-Ds from the
+ review assignments are accounted for, especially LC).
+
+ 3. The I-D name is then copied to the name field as well as being
+ appended to the hyperlink for the "Review Summary" field. The
+ hyperlink is included as part of the default row. This minimizes
+ the steps to enter the reviews in the spreadsheet.
+
+ 4. The data is also sorted by "Chat/LC Date", "Assigned", and
+ "Document". The file is then saved and closed.
+
+ 5. The file is then reopened and saved as HTML.
+
+ 6. The file is opened a second time and sorted by "Assigned",
+ "Chat/LC Date", and "Document" to provide the I-D reviewers an
+ easy way to find any outstanding assignments.
+
+
+
+Barnes, et al. Informational [Page 11]
+
+RFC 6385 Gen-ART October 2011
+
+
+5.2. Last Call Assignment Procedure
+
+ The secretary can cache the Last Call assignments as they are
+ announced and/or check the IETF announcement mailing list archives.
+ The assignments are done on Thursday evening, along with any telechat
+ assignments. This optimizes the process in terms of batch changes to
+ files.
+
+ The assignments are listed in an HTML file. The following are the
+ steps in creating that file:
+
+ 1. The order of assignment is actually created the week before, with
+ the details below. Thus, before starting the new assignments,
+ the current file is saved for editing for the following week.
+ The current file-naming convention is "reviewersyymmdd-lc.html"
+ (e.g., for July 8th, 2010, the file reviewers100708-lc.html was
+ created, and the file for the following week is named
+ reviewers100715-lc.html).
+
+ 2. Since the file is already prepared with the appropriate ordering
+ of reviewers, the assignments are done in the order of due dates.
+ The link to the I-D in the Datatracker is copied into the
+ assignment file along with the intended RFC status for each of
+ the new LC I-Ds.
+
+ 3. The "Due Date" paragraph from the Last Call announcement is
+ shortened as follows: "IETF LC ends on:", keeping the date.
+
+ 4. Once the assignment file is complete, the new I-Ds are added to
+ the spreadsheet as described above.
+
+ 5. The assignment file for the next week is then updated to reflect
+ the next reviewer in the round-robin process, by simply cutting
+ and pasting the names in the list in a block and removing any
+ "one doc per month" reviewers (annotated with an "*") that have
+ already received their monthly assignment. If the next round of
+ assignments occurs at the beginning of a new month, the "one doc
+ per month" reviewers are added back into the list (in the normal
+ order -- alphabetically by first name).
+
+ 6. The assignment files and updated spreadsheets are then cached on
+ the Gen-ART server.
+
+ 7. An email providing a link to the assignment file along with the
+ updated spreadsheets is sent to the Gen-ART mailing list. This
+ email has a standard form, such that the reviewers can simply cut
+ and paste the template to include the Gen-ART context statement
+ and link to the FAQ.
+
+
+
+Barnes, et al. Informational [Page 12]
+
+RFC 6385 Gen-ART October 2011
+
+
+5.3. Telechat Assignment Procedure
+
+ Since LC assignments are now the starting point for Gen-ART I-D
+ reviews, the telechat assignments are generally straightforward, as
+ the majority of the I-Ds are already in the spreadsheet. The
+ following details the steps:
+
+ 1. The telechat agenda is typically available around 6PM PDT. In
+ order to create the assignment HTML file, the agenda is created
+ from the email announcing the upcoming telechat agenda. The
+ filename has the following format, with the date corresponding to
+ the telechat date (versus the date of assignment, as is the case
+ for Last Call assignments): "reviewersyymmdd.html".
+
+ 2. Rows are added to the agenda for the reviewers' names.
+
+ 3. The reviewers' names are then added to the weekly assignment
+ file.
+
+ 4. As each reviewer is added to the assignment file, the review
+ spreadsheet is updated as follows:
+
+ * "Chat/LC Date" is changed to the telechat date.
+
+ * The link to the LC review, if available, is copied as the link
+ for the "Previous Review" column.
+
+ * The "text" for the "Current Review" is updated to reflect the
+ new review type (i.e., Telechat) and version.
+
+ 5. In the case of an I-D that did not go through IETF LC, a reviewer
+ is assigned using the order in the file to be used for Last Call
+ assignments for the next week.
+
+ 6. Once the reviewer(s) have been determined, the LC assignment file
+ for the next week is updated.
+
+ 7. Any new I-Ds are then added to the spreadsheet (and the updates
+ saved) per the steps described in Section 5.1.
+
+ 8. The assignment files and updated spreadsheets are then cached on
+ the Gen-ART server.
+
+ 9. An email providing a link to the assignment file along with the
+ updated spreadsheets is sent to the Gen-ART mailing list. This
+ email has a standard form, such that the reviewers can simply cut
+ and paste the template to include the Gen-ART context statement
+ and link to the FAQ.
+
+
+
+Barnes, et al. Informational [Page 13]
+
+RFC 6385 Gen-ART October 2011
+
+
+5.4. Capturing Reviews
+
+ As noted in Section 4.4, the spreadsheet is typically updated with
+ the review summaries on Thursday evenings just prior to entering the
+ data for that week's LC and any telechat assignments. The following
+ summarizes the steps to capture the reviews:
+
+ 1. Currently, an additional volunteer is assisting the secretary in
+ caching the email reviews as they arrive.
+
+ 2. In the cases where the review is included inline in the body of
+ the email, the review is cut and pasted into a text file and
+ saved with the reviewer's last name appended to the filename --
+ e.g., draft-ietf-xyz-00-smith.txt.
+
+ 3. In the case where the review is included as an attachment to the
+ email, the file can be directly saved and uploaded.
+
+ 4. The volunteer uploads the reviews by around 5PM CST on Thursdays;
+ thus, they are available to the secretary at the time that week's
+ assignments are done. This sequence is necessary to ensure the
+ information for I-Ds on the upcoming telechat is up to date.
+
+ 5. The review summary is entered into the "Current Summary" field.
+ Note that the hyperlink to the review (added at assignment time)
+ will automatically work when the file is uploaded.
+
+ 6. Once all the reviews have been entered and the spreadsheets
+ formatted, the review spreadsheet is saved and files uploaded per
+ the last three steps in Section 5.1.
+
+6. Results
+
+ Over the past 7 years, the Gen-ART has provided reviewing services to
+ 3 ADs and has done around two thousand publicly available reviews.
+ The reviews have been executed with a team of around a dozen full-
+ time reviewers and other reviewers receiving one I-D assignment each
+ month. There are currently 9 reviewers in the latter category. The
+ full-time reviewers receive 2-3 assignments each month. In terms of
+ improving quality, the number of I-Ds that are now "Ready" at the
+ time of the telechat (since the reviews are now initiated at LC time)
+ has increased. The review term "Ready" means the reviewer believes
+ that the document has no outstanding editorial or technical issues.
+ Based on the data from 2007, there were over 250 I-Ds assigned at LC
+ time that went through IESG review. Of those 250 I-Ds, 82% of the LC
+ reviews (205 I-Ds) were completed. Of the completed reviews, 70%
+ (144 I-Ds) were "Ready" at the time of the telechat. Of those 144
+ I-Ds, roughly 1/4 had been deemed "Ready" (with no nits) at LC time
+
+
+
+Barnes, et al. Informational [Page 14]
+
+RFC 6385 Gen-ART October 2011
+
+
+ (based on a sample of 50 reviews). For the I-Ds that were not
+ reviewed at LC time, only about 1/4 of those were deemed "Ready" when
+ they were reviewed for the telechat. So, doing the Gen-ART reviews
+ at Last Call time does seem to improve the quality of the I-Ds for
+ the telechat.
+
+7. Impressions
+
+ This section is divided into 3 subsections: the impressions as
+ gathered from the Gen-ART, the impressions of the ADs for whom they
+ worked, and the impressions of the secretaries of Gen-ART.
+
+7.1. Reviewers' Impressions
+
+ The following list of comments are excerpted and edited from comments
+ sent in by the reviewers of Gen-ART in response to the request:
+
+ "We'd like to ask you each to write a few lines about your personal
+ experience and lessons learned as a Gen-ART reviewer".
+
+ o We really do find problems, but we don't find problems with
+ most I-Ds.
+
+ o Comments seem to be in three areas: editorial/grammar, editorial/
+ what-the-heck-does-this-mean, and actual problems. I'm seeing
+ fewer reviews in the first category, which is a good thing.
+
+ o It is becoming rarer that we hear back "these guys have suffered
+ enough; I'm voting no objection" (I'm remembering an LDAP I-D that
+ had been around so long it had 2119 referenced AS A DRAFT -- some
+ people suffered a lot).
+
+ o The direct assignment of reviews is necessary and effective. It
+ does not matter much as far as I can tell what scheme is used to
+ actually do the assignment.
+
+ o Folks are very open to the reviews that come out of Gen-ART. This
+ somewhat surprised me, because I have seen resistance to outside
+ reviews in other cases.
+
+ o The improvements that have come about (for example, one of my
+ latest, an I-D about the SIPPING conference) have made a big
+ difference to the comprehensibility and usability of the I-Ds --
+ and provide a useful incentive to keep going.
+
+ o Some form of review like this is desperately needed. While most
+ of the stuff we see is good, every once in a while really bad
+ errors have made their way all the way to IESG vote.
+
+
+
+Barnes, et al. Informational [Page 15]
+
+RFC 6385 Gen-ART October 2011
+
+
+ o Reading this stuff is interesting. I like having a reason to read
+ a wide range of materials.
+
+ o I am more than convinced that this can be, and is, a valuable
+ process. It is, in my opinion, a pity that Senior IETF Reviewers
+ (SIRS) and so on did not take off, because this late-stage
+ reviewing is a poor substitute for doing the same thing at a much
+ earlier stage. Very few of the drafts that have come past my
+ screen are truly fully ready for IESG review. It is actually a
+ joy to find the occasional nugget that is both well written and is
+ a proper technical job, such that the reviewer really can say
+ "This is ready".
+
+ o I have certainly found the process intellectually stimulating! It
+ encourages me to take a wider interest in what is going on in the
+ IETF, but consumes a fair bit of time to do a proper job, and
+ requires a very wide knowledge to be able to properly catch the
+ cross-area implications: I hope (believe!) that this is something
+ that one gets better at with experience and doing a few of these
+ reviews.
+
+ o There is probably a very limited pool of people who have both the
+ time and the inclination to keep on doing these reviews. It does
+ require a fair bit of dedication.
+
+ o It is difficult to avoid correcting the English, even if that is
+ not really the point: Often, really bad English (whether as a
+ result of non-mother-tongue authors with limited grasp or mother-
+ tongue authors using informal language) obscures/corrupts what is
+ being said or just makes it impossible to read.
+
+ o Mostly authors welcome the comments: I think most of them
+ understand the concept of "ego-free reviewing", and we have
+ generally been constructive rather than destructive.
+
+ o Part of the job of Gen-ART is to think the unthinkable from
+ another point of view, to challenge (apparently undocumented)
+ assumptions, and apply experience from other fields.
+
+
+
+
+
+
+
+
+
+
+
+
+
+Barnes, et al. Informational [Page 16]
+
+RFC 6385 Gen-ART October 2011
+
+
+7.2. General Area Directors' Impressions
+
+ It should be noted that these impressions are from multiple General
+ Area Directors; thus, the "I"s are not necessarily associated with a
+ specific AD.
+
+ o It's essential. The reviewing load for the IESG <shout>DOES NOT
+ SCALE</shout>.
+
+ o Without Gen-ART, I would be a much less effective General AD.
+
+ o On a single fortnight example, the IESG had 21 drafts on the
+ agenda. It is just impossible (to conscientiously review all the
+ documents), and no wonder we sometimes miss serious issues.
+
+ o So I think a distributed review team with about 30 trusted
+ reviewers needs to be institutionalized. I suspect that will need
+ to be formalized in a BCP sooner or later -- with their reviews
+ having a formal position in the standards process, and the
+ expectation that the whole IESG truly reviews all I-Ds being
+ relaxed.
+
+ o We've learned that polite, well reasoned, constructive reviews are
+ very positively received by authors and WGs. Dismissive reviews
+ are counter-productive. And reviews sent in private eventually
+ show up in public, so it's better to go public at the start.
+
+ o Normally, LC reviews are available in good time for the draft to
+ be revised before reaching the IESG agenda. It is important that
+ this happens, except for an emergency situation where the
+ responsible AD has good reason to place the draft on the agenda
+ immediately. In that case, it would be preferable for the AD to
+ inform the Gen-ART, so that the review can be expedited.
+
+ o The other problem is a big detail -- between late Thursday or
+ early Friday when the secretary sends out the assignments, and
+ Wednesday when the General AD likes to start filling in ballots
+ based on the reviews received by close of business on Tuesday,
+ there are only three work days (plus possible volunteer time over
+ the weekend). Now even with only one I-D to review, that may be a
+ real challenge. Sometimes, a lucky reviewer will get 130 pages
+ (e.g., draft-ietf-nntpext-base-27). That doesn't compute.
+
+
+
+
+
+
+
+
+
+Barnes, et al. Informational [Page 17]
+
+RFC 6385 Gen-ART October 2011
+
+
+ o There are some mechanical issues. The process followed is far too
+ manual. Everything needs to be robotic except for the judgment
+ calls about which reviewer gets which draft. Similarly, the
+ reviewer should be able to just paste the review into a web form,
+ click, and it's sent off to everyone appropriate and posted to the
+ review site.
+
+7.3. Gen-ART Secretaries' Impressions
+
+ Serving as the secretary of Gen-ART is a worthwhile experience. From
+ a personal point of view, it gives the secretary an easy way to track
+ all of the work going through the IESG review process and see how the
+ work flowed through that process. Also, by reviewing and sometimes
+ creating the one-line abstracts that go on the review web page, the
+ secretary has an opportunity to really get a survey of the work being
+ approved by the IETF.
+
+ The nature of these reviews is informal, and originally the reviews
+ were only intended for the General AD, though they were made public.
+ During 2004, there was little if any interaction between authors and
+ reviewers. There was some discussion during 2004 about trying to
+ expand the role of Gen-ART to a more formal, early-review model,
+ i.e., to evolve it into a form of SIRS. The original Gen-ART
+ secretary was against such a transformation, because she felt it
+ would put at risk something that worked. She believed that there
+ were risks inherent in formalizing the reviews and adding mechanisms
+ for standardizing the resultant review process. Another concern
+ involves the interaction between reviewers and authors. As discussed
+ above, it has become the practice to send reviews to the authors with
+ an explanation about the nature of Gen-ART reviews. While it is
+ clear that this has resulted in improved RFCs, it has also resulted
+ in increased workload for the reviewers.
+
+ The secretary thinks that Gen-ART is an experiment that works well,
+ but she also believes it is fragile. The secretary is often
+ concerned about overburdening reviewers, and feels it is her
+ responsibility to keep them from burning out. Adding additional
+ reviewers to the review team would help to alleviate this concern.
+ In terms of the process, adding additional reviewers has minimal
+ impact.
+
+8. Needed Improvements
+
+ The current size of the review team introduces a fairly heavy
+ workload for the individual reviewers that are not on the "one doc
+ per month" assignment cycle. Additional reviewers would be really
+ helpful to alleviate this workload. It is also important to note
+ that having additional reviewers adds minimal workload to the
+
+
+
+Barnes, et al. Informational [Page 18]
+
+RFC 6385 Gen-ART October 2011
+
+
+ secretary's process; thus, the only blocking point is finding the
+ right folks that are interested in this type of volunteer role. As
+ noted in Section 7.2, 30 would be a good size for the review team.
+ This would cut the workload for an individual reviewer in half (given
+ the current model of 9 reviewers on the "one doc per month"
+ assignment cycle).
+
+ Obviously, automation of the process would be a good thing. However,
+ Gen-ART secretaries are not necessarily highly motivated to
+ transition to a more automated approach until a significant part of
+ the process is automated. In more recent consideration of this
+ situation, it likely would be best to first automate the process of
+ entering the reviews, as that benefits the review team as a whole.
+ This automation should allow the reviewers to enter the reviews via a
+ web interface that would automatically generate the appropriate
+ emails -- quite similar to how the draft "Upload" tool currently
+ works. Also, given consistent naming conventions for the review
+ forms, this step would automate some of the process for the
+ secretary, as the reviews would automatically appear via the
+ spreadsheet hyperlinks, although there would still be a need to
+ manually enter the summary. But this would eliminate the need to
+ edit/normalize and upload files and, hopefully, eliminate the problem
+ encountered with unflowed text in emails and getting the review
+ properly formatted using some text editors.
+
+ Section 5 was written to facilitate the process of determining tools
+ requirements, by providing the very detailed steps currently applied
+ to the process. As noted above, automating the upload of the reviews
+ could be a good first step. This is somewhat starting at the end of
+ the process. However, it seems that by automating in this direction,
+ we may have optimal results; since one of the earliest steps in the
+ process is the task of assigning reviewers, it likely needs the most
+ manual intervention, even with tools available.
+
+ The current SecDir secretary does use some tools for assignments and
+ generating assignment emails. These tools could be considered for
+ use by the Gen-ART secretary. Since the SecDir reviews are not
+ cached and the information maintained for those reviews is less
+ detailed, there would be no reusability of that aspect. However, if
+ the Gen-ART spreadsheet can be automatically populated (with
+ assignments and completed reviews), the SecDir may be able to make
+ use of that same tool.
+
+
+
+
+
+
+
+
+
+Barnes, et al. Informational [Page 19]
+
+RFC 6385 Gen-ART October 2011
+
+
+9. Applicability
+
+ As implemented today, the process has no formal role in the IETF
+ standards process. But as trust in the review team has built, and as
+ the team itself has learned to deliver reviews that are generally
+ well received, they have had a significant impact on I-D quality and
+ on timeliness. Rather than becoming a roadblock, they have (in
+ general) allowed the General AD to feel more confident in reaching
+ decisions and be more precise in resolving issues. Since reviews now
+ typically appear during IETF Last Call, the reviews, like the SecDir
+ reviews, are now generally expected. So, the role of the team has
+ evolved to be more formal than in the past (i.e., when this document
+ was first drafted in 2005). However, the handling of the reviews
+ remains entirely within the scope of the ADs, document shepherds, WG
+ chairs, and authors as they deem appropriate.
+
+10. Security Considerations
+
+ Since this is an informational I-D about an open process, the
+ security considerations are specific to the process and users
+ involved in the process. The primary concern would be to limit the
+ people that have the ability to create and update the Gen-ART data/
+ files to ensure that the integrity of the data is maintained. For
+ example, each Gen-ART reviewer should have a unique user name/
+ password, just as folks do to access any other IETF-maintained data,
+ as appropriate.
+
+11. Acknowledgments
+
+ Initial comments were received from the members of the Gen-ART, and
+ the experiences discussed in this document were derived from their
+ hard work over the last 7+ years. We thank the past reviewers of the
+ Gen-ART:
+
+ Mark Allman
+ Harald Alvestrand (creator of Gen-ART)
+ Ron Bonica
+ Scott Brim
+ Gonzalo Camarillo
+ Sharon Chisholm
+ Spencer Dawkins
+ Lakshminath Dondeti
+ Avri Doria (past secretary)
+ Pasi Eronen
+ Dorothy Gellert
+ Eric Gray
+ Avashalom Houri
+ Glenn Kowack
+
+
+
+Barnes, et al. Informational [Page 20]
+
+RFC 6385 Gen-ART October 2011
+
+
+ John Loughney
+ Lucy Lynch
+ Enrico Marocco
+ Michael Patton
+ Stefan Santesson
+ Robert Sparks
+ Tom Taylor
+ Sean Turner
+ Christian Vogt
+ Suzanne Woolf
+
+ We also thank the current team of reviewers/secretary:
+
+ Mary Barnes (past secretary, 2005-2010)
+ Richard Barnes
+ David Black
+ Ben Campbell
+ Brian Carpenter (past General AD)
+ Elwyn Davies
+ Francis Dupont
+ Roni Even
+ Miguel-Angel Garcia
+ Vijay Gurbani (assisting secretary to upload reviews)
+ Wassim Haddad
+ Joel Halpern
+ Suresh Krishnan
+ Peter McCann
+ Jean Mahoney (secretary as of Jan. 2011)
+ Alexey Melnikov
+ Kathleen Moriarty
+
+12. Informative References
+
+ [RFC1958] Carpenter, B., Ed., "Architectural Principles of the
+ Internet", RFC 1958, June 1996.
+
+ [RFC2223bis] Reynolds, J., Ed., and R. Braden, Ed., "Instructions to
+ Request for Comments (RFC) Authors", Work in Progress,
+ August 2004.
+
+ [RFC-STYLE] Braden, R., Ginoza, S., and A. Hagens, "RFC Document
+ Style", September 2009,
+ <http://www.rfc-editor.org/rfc-style-guide/rfc-style>.
+
+ [RFC3426] Floyd, S., "General Architectural and Policy
+ Considerations", RFC 3426, November 2002.
+
+
+
+
+
+Barnes, et al. Informational [Page 21]
+
+RFC 6385 Gen-ART October 2011
+
+
+ [RFC3439] Bush, R. and D. Meyer, "Some Internet Architectural
+ Guidelines and Philosophy", RFC 3439, December 2002.
+
+ [RFC3552] Rescorla, E. and B. Korver, "Guidelines for Writing RFC
+ Text on Security Considerations", BCP 72, RFC 3552,
+ July 2003.
+
+ [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing
+ an IANA Considerations Section in RFCs", BCP 26,
+ RFC 5226, May 2008.
+
+ [SIRS] Carpenter, B. and D. Crocker, "Careful Additional
+ Review of Documents (CARD) by Senior IETF Reviewers
+ (SIRS)", Work in Progress, June 2003.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Barnes, et al. Informational [Page 22]
+
+RFC 6385 Gen-ART October 2011
+
+
+Authors' Addresses
+
+ Mary Barnes
+ Polycom
+ TX
+ USA
+
+ EMail: mary.ietf.barnes@gmail.com
+
+
+ Avri Doria
+ Research Consultant
+ Providence, RI
+ USA
+
+ EMail: avri@acm.org
+
+
+ Harald Alvestrand
+ Google
+ Kungsbron 2
+ 11122 Stockholm
+ SE
+
+ EMail: harald@alvestrand.no
+
+
+ Brian E. Carpenter
+ University of Auckland
+ PB 92019
+ Auckland, 1142
+ New Zealand
+
+ Phone:
+ EMail: brian.e.carpenter@gmail.com
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Barnes, et al. Informational [Page 23]
+