doc: Add RFC documents

1 files changed, 1291 insertions, 0 deletions

diff --git a/doc/rfc/rfc6385.txt b/doc/rfc/rfc6385.txt
new file mode 100644
index 0000000..722131a
--- /dev/null
+++ b/doc/rfc/rfc6385.txt
@@ -0,0 +1,1291 @@
+
+
+
+
+
+
+Internet Engineering Task Force (IETF)                         M. Barnes
+Request for Comments: 6385                                       Polycom
+Category: Informational                                         A. Doria
+ISSN: 2070-1721                                      Research Consultant
+                                                           H. Alvestrand
+                                                                  Google
+                                                            B. Carpenter
+                                                  University of Auckland
+                                                            October 2011
+
+
+             General Area Review Team (Gen-ART) Experiences
+
+Abstract
+
+   The General Area Review Team (Gen-ART) has been doing reviews of
+   Internet-Drafts (I-Ds) since 2004.  This document discusses the
+   experience and the lessons learned over the past 7 years of this
+   process.  The review team initially reviewed the I-Ds before each of
+   the IESG telechats.  Since late 2005, review team members have been
+   assigned to review I-Ds during IETF Last Call, unless no IETF Last
+   Call is necessary for the I-D.  The same reviewer then reviews any
+   updates when the I-D is placed on an IESG telechat agenda.
+
+Status of This Memo
+
+   This document is not an Internet Standards Track specification; it is
+   published for informational purposes.
+
+   This document is a product of the Internet Engineering Task Force
+   (IETF).  It represents the consensus of the IETF community.  It has
+   received public review and has been approved for publication by the
+   Internet Engineering Steering Group (IESG).  Not all documents
+   approved by the IESG are a candidate for any level of Internet
+   Standard; see Section 2 of RFC 5741.
+
+   Information about the current status of this document, any errata,
+   and how to provide feedback on it may be obtained at
+   http://www.rfc-editor.org/info/rfc6385.
+
+
+
+
+
+
+
+
+
+
+
+
+Barnes, et al.                Informational                     [Page 1]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+Copyright Notice
+
+   Copyright (c) 2011 IETF Trust and the persons identified as the
+   document authors.  All rights reserved.
+
+   This document is subject to BCP 78 and the IETF Trust's Legal
+   Provisions Relating to IETF Documents
+   (http://trustee.ietf.org/license-info) in effect on the date of
+   publication of this document.  Please review these documents
+   carefully, as they describe your rights and restrictions with respect
+   to this document.  Code Components extracted from this document must
+   include Simplified BSD License text as described in Section 4.e of
+   the Trust Legal Provisions and are provided without warranty as
+   described in the Simplified BSD License.
+
+Table of Contents
+
+   1. Introduction ....................................................2
+   2. Who Are the Gen-ART Members? ....................................3
+   3. Goals of Gen-ART ................................................3
+   4. Gen-ART Reviews .................................................4
+      4.1. IETF Last Call Review Process ..............................4
+      4.2. IESG Telechat Review Process ...............................5
+      4.3. Form of Review .............................................5
+      4.4. Gen-ART Process Overview ...................................8
+   5. Secretarial Process ............................................10
+      5.1. Maintaining Review Spreadsheet ............................10
+      5.2. Last Call Assignment Procedure ............................12
+      5.3. Telechat Assignment Procedure .............................13
+      5.4. Capturing Reviews .........................................14
+   6. Results ........................................................14
+   7. Impressions ....................................................15
+      7.1. Reviewers' Impressions ....................................15
+      7.2. General Area Directors' Impressions .......................17
+      7.3. Gen-ART Secretaries' Impressions ..........................18
+   8. Needed Improvements ............................................18
+   9. Applicability ..................................................20
+   10. Security Considerations .......................................20
+   11. Acknowledgments ...............................................20
+   12. Informative References ........................................21
+
+1.  Introduction
+
+   The General Area Review Team (Gen-ART) was created personally by the
+   General Area Director in 2004.  This document discusses the
+   experiences and the lessons learned as the process has evolved over
+   the past 7 years.  The process described in this document reflects
+   that which was in place at the time this document was published.
+
+
+
+Barnes, et al.                Informational                     [Page 2]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+   This process is likely to continue to change over time.  The review
+   team has been retained by subsequent General Area Directors.  It has
+   no official role in the IETF standards process, except as a set of
+   individuals entitled, like everyone, to comment on Internet-Drafts
+   (I-Ds).  Its volunteers, including a secretary and the team of
+   reviewers, serve at the invitation of the General AD.  Both the
+   reviews and the reviewer names are public.
+
+2.  Who Are the Gen-ART Members?
+
+   The reviewers are typically individuals that have a fair amount of
+   experience within various IETF Working Groups (WGs), have authored WG
+   I-Ds and RFCs, and are often considered to be subject matter experts
+   (SMEs) in their particular areas of work.  The current review team is
+   comprised of such technical experts, including several WG chairs as
+   well as past and current Internet Architecture Board (IAB) members.
+   Several past and current ADs have served as reviewers.  Two past
+   General ADs have also served as reviewers, with one currently
+   serving.
+
+   Members of the review team sometimes excuse themselves from the team
+   for various reasons, typically due to "day job" demands.  However,
+   they often rejoin (for periods of time) as their schedules allow.
+   Also, some reviewers remain on the team, while their review workload
+   is decreased by assigning them just one I-D (at Last Call time) to
+   review each month.  Section 11 provides a list of currently active
+   reviewers, along with those who have served on the review team in the
+   past.
+
+3.  Goals of Gen-ART
+
+   The original and continuing goal of the Gen-ART was, and is, to
+   offload from the General AD some of the burden of IESG reviews.  The
+   load for the bi-weekly IESG reviews is often quite large;
+   occasionally, there are more than 20 I-Ds scheduled for discussion in
+   a single telechat.  Thus, ADs also have less than a week's notice for
+   many of the I-Ds on the telechat agenda.
+
+   Gen-ART was based on a model that had proved productive in the
+   Operations (OPS) Directorate: quick review close to telechat time, to
+   advise the AD on issues that remain serious.  By having a trusted
+   group of reviewers read and evaluate the I-Ds, the General AD would
+   be able to concentrate on those I-Ds where there was a concern
+   expressed by the reviewer.  The reviewers are expected to provide
+   feedback based on a whole set of criteria, including the criteria
+
+
+
+
+
+
+Barnes, et al.                Informational                     [Page 3]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+   summarized in Section 4.3.  The overall objective is to ensure that
+   the I-Ds are well structured; can be easily understood, at least at a
+   high level; and provide a reasonable basis for implementation (for
+   I-Ds intended for the Standards Track).
+
+   While other area (and WG) directorates/review teams existed prior to
+   Gen-ART and more have been established since Gen-ART, the roles of
+   each are fairly distinct.  Thus, there is little overlap between the
+   goals and review criteria for the various review teams.  It is also
+   very valuable for these other review teams to operate independently.
+   For example, when both Gen-ART reviews and Security Directorate
+   (SecDir) reviews raise the same sorts of concerns, it's a clear red
+   flag that the I-D needs more work before progressing.  In addition,
+   due to the typical thoroughness (and objectiveness) of the various
+   review teams' reviews, the sponsoring AD and document shepherd are
+   often able to work with the editors/WG (and vice versa, depending
+   upon area and WG structure) to improve the overall quality of the
+   final I-D.  It should also be noted that some ADs take the Gen-ART
+   reviews into consideration when preparing their own evaluations.
+
+   Statistics from the Gen-ART reviews over the past 6+ years show a
+   trend of increased quality and readiness for progression of I-Ds by
+   the time they are placed on the telechat agenda.  Additional
+   statistics are discussed in Section 6.
+
+4.  Gen-ART Reviews
+
+4.1.  IETF Last Call Review Process
+
+   While the original process was meant only for reviews just before the
+   IESG telechat, the decision was made to include IETF Last Call (LC)
+   reviews in early 2005.  Over time the latter has proven to be quite
+   effective.  Assigning the I-Ds at IETF LC time typically gives a
+   reviewer more time to review an I-D.  And, in some cases, the IETF LC
+   version is the one to appear on the telechat.  Thus, by the time I-Ds
+   are added to the telechat agenda, a majority (typically at least 70%)
+   have already been reviewed.  For those I-Ds that have been
+   up-versioned, the amount of time dedicated to re-review depends upon
+   the review summary for the IETF LC review.
+
+   The assignments at IETF LC time evolved to minimize the gap between
+   LC announcements and assignment time, with the secretary doing LC
+   assignments every Thursday night.  This typically allows the reviewer
+   at least one week and sometimes two to three weeks to complete the
+   review.  The reviews are obviously most helpful when done on or
+   before the end of the IETF LC.
+
+
+
+
+
+Barnes, et al.                Informational                     [Page 4]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+   The Last Call assignments are done on a fairly strict round-robin
+   basis to ensure a fair workload amongst all the reviewers.  Reviewers
+   that are unavailable (vacations, etc.) during the review period
+   timeframe obviously are excluded from that round of assignments, but
+   remain in the same queue position for the next round.  The order is
+   occasionally modified to avoid assigning an editor/author or WG chair
+   their own I-Ds.  A reviewer may also NACK an assignment if they feel
+   they may have some bias (although corporate affiliations are not
+   considered to be sources of bias) or they don't feel they can review
+   the I-D in a timely manner.
+
+   The assignment process is completely manual, although a spreadsheet
+   tremendously facilitates the process.  The details are described in
+   Section 5.  Ideally, this process could be automated.  However,
+   manual intervention would still be required to maintain the
+   appropriate available reviewer list (unless reviewers took on the
+   task of maintaining their data in some sort of database).  Further
+   details on the tools necessary to automate the entire process are
+   provided in Section 8.
+
+4.2.  IESG Telechat Review Process
+
+   The process for reviewing I-Ds when they appear on the IESG agenda is
+   as follows:
+
+   o  The "nearly final" IESG meeting agenda generally appears on
+      Thursday night, less than one week before the IESG telechat.  The
+      Gen-ART secretary uses this as the input for the assignment
+      process.
+
+   o  For I-Ds reviewed at IETF Last Call, a new review is only asked
+      for if the I-D is revised.  In this case the reviewer, typically
+      the person who did the Last Call review, only needs to check that
+      any open issues were resolved.  Often the draft will not have
+      changed between IETF LC and the IESG telechat review.  Section 4.4
+      provides the step-by-step telechat review assignment process, with
+      specific details on the maintenance of the review assignment data,
+      which is in turn maintained in review spreadsheets (Section 5).
+
+4.3.  Form of Review
+
+   Rather than invent new guidelines, the Gen-ART requirements for the
+   form of a review stole liberally from "Careful Additional Review of
+   Documents (CARD) by Senior IETF Reviewers (SIRS)" [SIRS], making
+   adaptations for the special "late, quick review" case and the nature
+   of the General Area's concerns.
+
+
+
+
+
+Barnes, et al.                Informational                     [Page 5]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+   Each review must start with a summary statement chosen from or
+   adapted from the following list:
+
+   o  This draft is ready for publication as a [type] RFC, where [type]
+      is Informational, Experimental, etc.  (In some cases, the review
+      might recommend publication as a different [type] than requested
+      by the author.)
+
+   o  This draft is basically ready for publication, but has nits that
+      should be fixed before publication.
+
+   o  This draft is on the right track but has open issues, described in
+      the review.
+
+   o  This draft has serious issues, described in the review, and needs
+      to be rethought.
+
+   o  This draft has very fundamental issues, described in the review,
+      and further work is not recommended.
+
+   o  Unfortunately, I don't have the expertise to review this draft.
+
+   The length of a review can vary greatly according to circumstances,
+   and it is considered acceptable for purely editorial comments to be
+   sent privately if it's obvious that the I-D needs substantial
+   revision.  All substantive comments, however, must be included in the
+   public review.  Wherever possible, comments should be written as
+   suggestions for improvement rather than as simple criticism.
+   Explicit references to prior work and prior IETF discussion should be
+   given whenever possible.
+
+   Reviewers are asked to review for all kinds of problems, such as
+   basic architectural or security issues, Internet-wide impact,
+   technical nits, problems of form and format (such as IANA
+   Considerations or incorrect references), and editorial issues.  Since
+   these reviews are on I-Ds that are supposed to be finished, the
+   review should consider "no issue too small" -- but should cover the
+   whole range, from the general architectural level to the editorial
+   level.
+
+   All reviews should apply generally agreed-upon IETF criteria,
+   such as:
+
+   o  [RFC1958]: "Architectural Principles of the Internet"
+
+   o  [RFC3426]: "General Architectural and Policy Considerations"
+
+   o  [RFC3439]: "Some Internet Architectural Guidelines and Philosophy"
+
+
+
+Barnes, et al.                Informational                     [Page 6]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+   o  ID-Checklist: The "ID checklist" document maintained by the IESG
+
+   o  [RFC2223bis]: "Instructions to Request for Comments (RFC) Authors"
+      as updated by [RFC-STYLE]: "RFC Document Style"
+
+   o  [RFC5226]: BCP 26 - "Guidelines for Writing an IANA Considerations
+      Section in RFCs"
+
+   o  [RFC3552]: BCP 72 - "Guidelines for Writing RFC Text on Security
+      Considerations"
+
+   o  Any other applicable architectural or procedural documents.  It is
+      considered important that reviews give precise references to such
+      criteria when relevant to a comment.
+
+   Of special interest to the General area, because they do not fall
+   under any other area, are:
+
+   o  A clear description of why the I-D or protocol is useful to the
+      Internet.
+
+   o  Adherence to IETF formalities, such as capitalized "must",
+      "should", etc. in normative statements, per the ID-Checklist.
+
+   o  Useful and reasonable IANA considerations.  Ensure that all
+      necessary registries are defined/referenced, and ensure definition
+      and compliance with IANA assignment criteria.
+
+   o  Correct dependencies for normative references.
+
+   o  That the I-D is written in reasonably clear English.
+
+   o  Checking the updates/obsoletes information.
+
+   o  Running idnits and checking the output.
+
+   o  Checking that things imported by reference, especially from other
+      RFCs, make sense (notably definitions of terms, security
+      considerations, and lists of criteria) and ensuring they are used
+      as intended by the referenced document.
+
+   o  That examples (e.g., Fully Qualified Domain Names (FQDNs),
+      telephone numbers, IP addresses) are taken from the right spaces.
+
+
+
+
+
+
+
+
+Barnes, et al.                Informational                     [Page 7]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+4.4.  Gen-ART Process Overview
+
+   The following provides a general overview of the Gen-ART process
+   along with some basic rules associated with assignments.  The very
+   precise details of the secretary's process are provided in Section 5.
+
+   o  The availability of reviewers and the order of assignments for the
+      next round of Last Call document assignments are updated weekly
+      and are available on the directory where all the assignments and
+      reviews are cached.
+
+   o  At telechat assignment time, all previously reviewed I-Ds are
+      assigned to the reviewer who reviewed them previously, assuming
+      that reviewer is available.  Otherwise, these I-Ds are assigned to
+      a new person in the process described below.
+
+   o  The secretary attempts to avoid assigning I-Ds that might conflict
+      with other IETF roles such as WG chairs, other directorates, etc.
+      However, in the cases where the secretary doesn't note the
+      conflict, the reviewer should notify the secretary and Gen-ART
+      mailing list so another reviewer may be assigned.
+
+   o  It should be emphasized that assignment is never made according to
+      a reviewer's technical specialty.  Even though it happens, when,
+      for example, routing I-Ds fall on routing experts or MIB documents
+      fall on MIB doctors, it is coincidental.  To the reviewer, the
+      choice looks random.
+
+   o  There is an attempt to evenly distribute I-Ds amongst reviewers at
+      LC time by using a round-robin process, starting from where the
+      previous week's assignments stopped.
+
+   o  Typically, there is no attempt made to actually equalize the load,
+      as the length and complexity of the I-Ds are not taken into
+      account in this process.  (Thus, a reviewer could end up with a
+      couple of hundred-page I-Ds, but this is statistically rare.)
+      However, in the case of a reviewer that might receive more than
+      one new LC I-D at one time, the secretary does try to ensure that
+      both are not large I-Ds.
+
+   o  Once the assignments are made, the web pages that list the reviews
+      and the assignments are posted.  Since the telechat agenda is not
+      published until the end of the day on the Thursdays prior to the
+      telechats (i.e., one week prior), the secretary needs to complete
+      the assignments on that Thursday evening.  This often requires
+      working later in the evening and also requires an Internet
+      connection even when traveling.
+
+
+
+
+Barnes, et al.                Informational                     [Page 8]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+   o  If the reviewers notice any problems or conflict of interest, a
+      bargaining process, shifting I-Ds from one reviewer to another,
+      takes place.  The secretary updates the assignment files with any
+      new assignments.
+
+   o  Once the review has been completed, the reviewer sends the review
+      to the Gen-ART list, ideally using the template provided in the
+      review assignment emails.  Typically, reviews are also sent to
+      authors, the responsible AD, and the WG chairs/document shepherd.
+      The only case where this might not be done is when there are no
+      issues found for a re-review and none had been found on an initial
+      review.  Sending the review to the authors, ADs, and/or WG chairs/
+      Proto Shepherds was originally voluntary but is now considered
+      standard practice.  Reviewers may also send the reviews to the
+      IETF discussion list, but that is entirely at the discretion of
+      the reviewer, in which case the author must be copied on the
+      review to ensure they see any follow-up discussion.  Reviewers may
+      also send the comments to the WG; however, this typically causes
+      the review to end up in the moderation queue, as most reviewers do
+      not want to subscribe to the WG lists for the I-Ds they review.
+      Thus, it is expected that any of the original recipients (i.e.,
+      authors, WG chairs/document shepherd, or responsible AD) may
+      forward the review to the WG mailing list if they believe it is
+      necessary.  In the past, sending these reviews resulted in
+      confusion among the authors, who may not have been expecting a
+      Gen-ART review and may not be familiar with Gen-ART.  Thus,
+      reviewers are reminded to prepend to the email the description of
+      Gen-ART and the purpose of the review.  This information is part
+      of the standard template provided in the review assignment emails.
+
+   o  The secretary gathers the reviews, sometimes edits them for
+      format, and records the review in the spreadsheet on the web
+      pages, including the synopsis.  This is typically done on
+      Thursday.  This is one aspect of the process that can be easily
+      delegated such that one volunteer uploads all the reviews and then
+      the secretary need only update the fields in the spreadsheet.  If
+      the reviewer has not provided a synopsis ("Summary" field in the
+      template), the secretary makes a best guess based on the review
+      details.  Note that in most cases the reviewers do include a
+      synopsis.
+
+   o  Ideally, the reviews should be posted to the Gen-ART mailing list
+      by close of business on the East coast on Tuesday.  This is
+      necessary to allow the General AD time to consider the reviews
+      prior to the telechat.  If the reviews are received after Tuesday,
+      the review may not be read by the AD before the IESG telechat.
+      Due to time constraints, the spreadsheets containing review
+      summaries/assignments are only updated on Thursday evenings when
+
+
+
+Barnes, et al.                Informational                     [Page 9]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+      the new LC assignments and upcoming telechat assignments are done.
+      Ideally, the reviews would get uploaded on the Tuesdays prior to
+      the telechat along with the updated spreadsheets.
+
+   o  If the AD concludes that the concerns raised by the reviewer
+      warrant placing a DISCUSS comment on the I-D, the AD will do so,
+      and the DISCUSS must be resolved before the I-D advances.
+      Usually, the reviewer will be involved in the resolution process,
+      but the responsibility for the DISCUSS rests with the AD.
+
+5.  Secretarial Process
+
+   This section summarizes the details of managing the review materials,
+   including the spreadsheet used to track all reviews and the HTML
+   files containing the review assignments.  Please note that these
+   details represent a snapshot of a process that has been implemented
+   -- the details are very likely to change over time, in particular as
+   the needed improvements highlighted in Section 8 are carried out.
+
+5.1.  Maintaining Review Spreadsheet
+
+   A spreadsheet is used to enter all the I-Ds at the time of assignment
+   and to capture all the reviews.  For IETF LC assignments, the
+   assignments are completed before adding the I-Ds to the spreadsheet
+   as described in Section 5.2.  For telechat assignments, I-Ds are
+   obviously only added in the cases where there is no previous LC
+   assignment.  For the other I-Ds, the appropriate fields are updated
+   as described in Section 5.3.
+
+   All the reviews can be accessed from the spreadsheet via hyperlinks
+   from specific fields, as summarized below.  The following information
+   is maintained in the spreadsheet (in the order listed):
+
+   1.  "Chat/LC Date": Indicates either the date on which the LC review
+       is due or the date of the telechat.
+
+   2.  "Document": Filename for the I-D, which includes a hyperlink to
+       the IETF I-D tracker.
+
+   3.  "Assigned": Name of the reviewer assigned to that I-D.
+
+   4.  "Category": This field contains one of the following self-
+       explanatory values: "Doc - WG", "Doc - Ind/AD", or "IETF LC".
+       Note that Gen-ART does not review I-Ds submitted directly to the
+       RFC Editor.  The "IETF LC" value is of course entered for all
+       I-Ds at LC time.  It is changed to one of the other appropriate
+       values, based on the information in the telechat agenda.
+
+
+
+
+Barnes, et al.                Informational                    [Page 10]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+   5.  "Previous Review": This includes a link to any previous reviews.
+       For example, when a doc appears on a telechat agenda, if an IETF
+       LC review was done, this field is updated with the review summary
+       for the LC review (i.e., the information from the "Current Review
+       Summary" as described below is copied to this column).  The field
+       is set to "New" when an I-D is first assigned/added to the
+       spreadsheet.  In the case of returns, this field has a value of
+       "Return" or "Return/IETF-LC" for I-Ds for which there is an LC
+       review.  It should be noted that since Gen-ART started doing
+       reviews at LC time, there seem to be far fewer returns on the
+       agenda.
+
+   6.  "Current Review Summary": This field includes the review type and
+       version number of the document that is to be reviewed or has been
+       reviewed (e.g., LC: -02).  When the field also contains a review
+       summary after the review type/version number (e.g., Telechat: -06
+       Ready), the active hyperlink points to the review.  Occasionally,
+       a reviewer will re-review an I-D prior to its telechat
+       assignment, in which case it is added to the spreadsheet, but the
+       date does not change to maintain consistency in the date field,
+       since the reviews themselves contain the review date.
+
+   The following summarizes the steps to add a new I-D to the
+   spreadsheet:
+
+   1.  In order to optimize steps, blank rows are first inserted for the
+       number of new I-Ds to be added.
+
+   2.  To minimize data entry, a row with default fields (including the
+       hyperlinks) is kept at the end of the file.  There is a separate
+       default row for IETF LC versus telechat assignments.  This row is
+       copied into each of the new blank rows.  The dates are then
+       entered (this allows double-checking that all I-Ds from the
+       review assignments are accounted for, especially LC).
+
+   3.  The I-D name is then copied to the name field as well as being
+       appended to the hyperlink for the "Review Summary" field.  The
+       hyperlink is included as part of the default row.  This minimizes
+       the steps to enter the reviews in the spreadsheet.
+
+   4.  The data is also sorted by "Chat/LC Date", "Assigned", and
+       "Document".  The file is then saved and closed.
+
+   5.  The file is then reopened and saved as HTML.
+
+   6.  The file is opened a second time and sorted by "Assigned",
+       "Chat/LC Date", and "Document" to provide the I-D reviewers an
+       easy way to find any outstanding assignments.
+
+
+
+Barnes, et al.                Informational                    [Page 11]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+5.2.  Last Call Assignment Procedure
+
+   The secretary can cache the Last Call assignments as they are
+   announced and/or check the IETF announcement mailing list archives.
+   The assignments are done on Thursday evening, along with any telechat
+   assignments.  This optimizes the process in terms of batch changes to
+   files.
+
+   The assignments are listed in an HTML file.  The following are the
+   steps in creating that file:
+
+   1.  The order of assignment is actually created the week before, with
+       the details below.  Thus, before starting the new assignments,
+       the current file is saved for editing for the following week.
+       The current file-naming convention is "reviewersyymmdd-lc.html"
+       (e.g., for July 8th, 2010, the file reviewers100708-lc.html was
+       created, and the file for the following week is named
+       reviewers100715-lc.html).
+
+   2.  Since the file is already prepared with the appropriate ordering
+       of reviewers, the assignments are done in the order of due dates.
+       The link to the I-D in the Datatracker is copied into the
+       assignment file along with the intended RFC status for each of
+       the new LC I-Ds.
+
+   3.  The "Due Date" paragraph from the Last Call announcement is
+       shortened as follows: "IETF LC ends on:", keeping the date.
+
+   4.  Once the assignment file is complete, the new I-Ds are added to
+       the spreadsheet as described above.
+
+   5.  The assignment file for the next week is then updated to reflect
+       the next reviewer in the round-robin process, by simply cutting
+       and pasting the names in the list in a block and removing any
+       "one doc per month" reviewers (annotated with an "*") that have
+       already received their monthly assignment.  If the next round of
+       assignments occurs at the beginning of a new month, the "one doc
+       per month" reviewers are added back into the list (in the normal
+       order -- alphabetically by first name).
+
+   6.  The assignment files and updated spreadsheets are then cached on
+       the Gen-ART server.
+
+   7.  An email providing a link to the assignment file along with the
+       updated spreadsheets is sent to the Gen-ART mailing list.  This
+       email has a standard form, such that the reviewers can simply cut
+       and paste the template to include the Gen-ART context statement
+       and link to the FAQ.
+
+
+
+Barnes, et al.                Informational                    [Page 12]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+5.3.  Telechat Assignment Procedure
+
+   Since LC assignments are now the starting point for Gen-ART I-D
+   reviews, the telechat assignments are generally straightforward, as
+   the majority of the I-Ds are already in the spreadsheet.  The
+   following details the steps:
+
+   1.  The telechat agenda is typically available around 6PM PDT.  In
+       order to create the assignment HTML file, the agenda is created
+       from the email announcing the upcoming telechat agenda.  The
+       filename has the following format, with the date corresponding to
+       the telechat date (versus the date of assignment, as is the case
+       for Last Call assignments): "reviewersyymmdd.html".
+
+   2.  Rows are added to the agenda for the reviewers' names.
+
+   3.  The reviewers' names are then added to the weekly assignment
+       file.
+
+   4.  As each reviewer is added to the assignment file, the review
+       spreadsheet is updated as follows:
+
+       *  "Chat/LC Date" is changed to the telechat date.
+
+       *  The link to the LC review, if available, is copied as the link
+          for the "Previous Review" column.
+
+       *  The "text" for the "Current Review" is updated to reflect the
+          new review type (i.e., Telechat) and version.
+
+   5.  In the case of an I-D that did not go through IETF LC, a reviewer
+       is assigned using the order in the file to be used for Last Call
+       assignments for the next week.
+
+   6.  Once the reviewer(s) have been determined, the LC assignment file
+       for the next week is updated.
+
+   7.  Any new I-Ds are then added to the spreadsheet (and the updates
+       saved) per the steps described in Section 5.1.
+
+   8.  The assignment files and updated spreadsheets are then cached on
+       the Gen-ART server.
+
+   9.  An email providing a link to the assignment file along with the
+       updated spreadsheets is sent to the Gen-ART mailing list.  This
+       email has a standard form, such that the reviewers can simply cut
+       and paste the template to include the Gen-ART context statement
+       and link to the FAQ.
+
+
+
+Barnes, et al.                Informational                    [Page 13]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+5.4.  Capturing Reviews
+
+   As noted in Section 4.4, the spreadsheet is typically updated with
+   the review summaries on Thursday evenings just prior to entering the
+   data for that week's LC and any telechat assignments.  The following
+   summarizes the steps to capture the reviews:
+
+   1.  Currently, an additional volunteer is assisting the secretary in
+       caching the email reviews as they arrive.
+
+   2.  In the cases where the review is included inline in the body of
+       the email, the review is cut and pasted into a text file and
+       saved with the reviewer's last name appended to the filename --
+       e.g., draft-ietf-xyz-00-smith.txt.
+
+   3.  In the case where the review is included as an attachment to the
+       email, the file can be directly saved and uploaded.
+
+   4.  The volunteer uploads the reviews by around 5PM CST on Thursdays;
+       thus, they are available to the secretary at the time that week's
+       assignments are done.  This sequence is necessary to ensure the
+       information for I-Ds on the upcoming telechat is up to date.
+
+   5.  The review summary is entered into the "Current Summary" field.
+       Note that the hyperlink to the review (added at assignment time)
+       will automatically work when the file is uploaded.
+
+   6.  Once all the reviews have been entered and the spreadsheets
+       formatted, the review spreadsheet is saved and files uploaded per
+       the last three steps in Section 5.1.
+
+6.  Results
+
+   Over the past 7 years, the Gen-ART has provided reviewing services to
+   3 ADs and has done around two thousand publicly available reviews.
+   The reviews have been executed with a team of around a dozen full-
+   time reviewers and other reviewers receiving one I-D assignment each
+   month.  There are currently 9 reviewers in the latter category.  The
+   full-time reviewers receive 2-3 assignments each month.  In terms of
+   improving quality, the number of I-Ds that are now "Ready" at the
+   time of the telechat (since the reviews are now initiated at LC time)
+   has increased.  The review term "Ready" means the reviewer believes
+   that the document has no outstanding editorial or technical issues.
+   Based on the data from 2007, there were over 250 I-Ds assigned at LC
+   time that went through IESG review.  Of those 250 I-Ds, 82% of the LC
+   reviews (205 I-Ds) were completed.  Of the completed reviews, 70%
+   (144 I-Ds) were "Ready" at the time of the telechat.  Of those 144
+   I-Ds, roughly 1/4 had been deemed "Ready" (with no nits) at LC time
+
+
+
+Barnes, et al.                Informational                    [Page 14]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+   (based on a sample of 50 reviews).  For the I-Ds that were not
+   reviewed at LC time, only about 1/4 of those were deemed "Ready" when
+   they were reviewed for the telechat.  So, doing the Gen-ART reviews
+   at Last Call time does seem to improve the quality of the I-Ds for
+   the telechat.
+
+7.  Impressions
+
+   This section is divided into 3 subsections: the impressions as
+   gathered from the Gen-ART, the impressions of the ADs for whom they
+   worked, and the impressions of the secretaries of Gen-ART.
+
+7.1.  Reviewers' Impressions
+
+   The following list of comments are excerpted and edited from comments
+   sent in by the reviewers of Gen-ART in response to the request:
+
+   "We'd like to ask you each to write a few lines about your personal
+   experience and lessons learned as a Gen-ART reviewer".
+
+   o  We really do find problems, but we don't find problems with
+      most I-Ds.
+
+   o  Comments seem to be in three areas: editorial/grammar, editorial/
+      what-the-heck-does-this-mean, and actual problems.  I'm seeing
+      fewer reviews in the first category, which is a good thing.
+
+   o  It is becoming rarer that we hear back "these guys have suffered
+      enough; I'm voting no objection" (I'm remembering an LDAP I-D that
+      had been around so long it had 2119 referenced AS A DRAFT -- some
+      people suffered a lot).
+
+   o  The direct assignment of reviews is necessary and effective.  It
+      does not matter much as far as I can tell what scheme is used to
+      actually do the assignment.
+
+   o  Folks are very open to the reviews that come out of Gen-ART.  This
+      somewhat surprised me, because I have seen resistance to outside
+      reviews in other cases.
+
+   o  The improvements that have come about (for example, one of my
+      latest, an I-D about the SIPPING conference) have made a big
+      difference to the comprehensibility and usability of the I-Ds --
+      and provide a useful incentive to keep going.
+
+   o  Some form of review like this is desperately needed.  While most
+      of the stuff we see is good, every once in a while really bad
+      errors have made their way all the way to IESG vote.
+
+
+
+Barnes, et al.                Informational                    [Page 15]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+   o  Reading this stuff is interesting.  I like having a reason to read
+      a wide range of materials.
+
+   o  I am more than convinced that this can be, and is, a valuable
+      process.  It is, in my opinion, a pity that Senior IETF Reviewers
+      (SIRS) and so on did not take off, because this late-stage
+      reviewing is a poor substitute for doing the same thing at a much
+      earlier stage.  Very few of the drafts that have come past my
+      screen are truly fully ready for IESG review.  It is actually a
+      joy to find the occasional nugget that is both well written and is
+      a proper technical job, such that the reviewer really can say
+      "This is ready".
+
+   o  I have certainly found the process intellectually stimulating!  It
+      encourages me to take a wider interest in what is going on in the
+      IETF, but consumes a fair bit of time to do a proper job, and
+      requires a very wide knowledge to be able to properly catch the
+      cross-area implications: I hope (believe!) that this is something
+      that one gets better at with experience and doing a few of these
+      reviews.
+
+   o  There is probably a very limited pool of people who have both the
+      time and the inclination to keep on doing these reviews.  It does
+      require a fair bit of dedication.
+
+   o  It is difficult to avoid correcting the English, even if that is
+      not really the point: Often, really bad English (whether as a
+      result of non-mother-tongue authors with limited grasp or mother-
+      tongue authors using informal language) obscures/corrupts what is
+      being said or just makes it impossible to read.
+
+   o  Mostly authors welcome the comments: I think most of them
+      understand the concept of "ego-free reviewing", and we have
+      generally been constructive rather than destructive.
+
+   o  Part of the job of Gen-ART is to think the unthinkable from
+      another point of view, to challenge (apparently undocumented)
+      assumptions, and apply experience from other fields.
+
+
+
+
+
+
+
+
+
+
+
+
+
+Barnes, et al.                Informational                    [Page 16]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+7.2.  General Area Directors' Impressions
+
+   It should be noted that these impressions are from multiple General
+   Area Directors; thus, the "I"s are not necessarily associated with a
+   specific AD.
+
+   o  It's essential.  The reviewing load for the IESG <shout>DOES NOT
+      SCALE</shout>.
+
+   o  Without Gen-ART, I would be a much less effective General AD.
+
+   o  On a single fortnight example, the IESG had 21 drafts on the
+      agenda.  It is just impossible (to conscientiously review all the
+      documents), and no wonder we sometimes miss serious issues.
+
+   o  So I think a distributed review team with about 30 trusted
+      reviewers needs to be institutionalized.  I suspect that will need
+      to be formalized in a BCP sooner or later -- with their reviews
+      having a formal position in the standards process, and the
+      expectation that the whole IESG truly reviews all I-Ds being
+      relaxed.
+
+   o  We've learned that polite, well reasoned, constructive reviews are
+      very positively received by authors and WGs.  Dismissive reviews
+      are counter-productive.  And reviews sent in private eventually
+      show up in public, so it's better to go public at the start.
+
+   o  Normally, LC reviews are available in good time for the draft to
+      be revised before reaching the IESG agenda.  It is important that
+      this happens, except for an emergency situation where the
+      responsible AD has good reason to place the draft on the agenda
+      immediately.  In that case, it would be preferable for the AD to
+      inform the Gen-ART, so that the review can be expedited.
+
+   o  The other problem is a big detail -- between late Thursday or
+      early Friday when the secretary sends out the assignments, and
+      Wednesday when the General AD likes to start filling in ballots
+      based on the reviews received by close of business on Tuesday,
+      there are only three work days (plus possible volunteer time over
+      the weekend).  Now even with only one I-D to review, that may be a
+      real challenge.  Sometimes, a lucky reviewer will get 130 pages
+      (e.g., draft-ietf-nntpext-base-27).  That doesn't compute.
+
+
+
+
+
+
+
+
+
+Barnes, et al.                Informational                    [Page 17]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+   o  There are some mechanical issues.  The process followed is far too
+      manual.  Everything needs to be robotic except for the judgment
+      calls about which reviewer gets which draft.  Similarly, the
+      reviewer should be able to just paste the review into a web form,
+      click, and it's sent off to everyone appropriate and posted to the
+      review site.
+
+7.3.  Gen-ART Secretaries' Impressions
+
+   Serving as the secretary of Gen-ART is a worthwhile experience.  From
+   a personal point of view, it gives the secretary an easy way to track
+   all of the work going through the IESG review process and see how the
+   work flowed through that process.  Also, by reviewing and sometimes
+   creating the one-line abstracts that go on the review web page, the
+   secretary has an opportunity to really get a survey of the work being
+   approved by the IETF.
+
+   The nature of these reviews is informal, and originally the reviews
+   were only intended for the General AD, though they were made public.
+   During 2004, there was little if any interaction between authors and
+   reviewers.  There was some discussion during 2004 about trying to
+   expand the role of Gen-ART to a more formal, early-review model,
+   i.e., to evolve it into a form of SIRS.  The original Gen-ART
+   secretary was against such a transformation, because she felt it
+   would put at risk something that worked.  She believed that there
+   were risks inherent in formalizing the reviews and adding mechanisms
+   for standardizing the resultant review process.  Another concern
+   involves the interaction between reviewers and authors.  As discussed
+   above, it has become the practice to send reviews to the authors with
+   an explanation about the nature of Gen-ART reviews.  While it is
+   clear that this has resulted in improved RFCs, it has also resulted
+   in increased workload for the reviewers.
+
+   The secretary thinks that Gen-ART is an experiment that works well,
+   but she also believes it is fragile.  The secretary is often
+   concerned about overburdening reviewers, and feels it is her
+   responsibility to keep them from burning out.  Adding additional
+   reviewers to the review team would help to alleviate this concern.
+   In terms of the process, adding additional reviewers has minimal
+   impact.
+
+8.  Needed Improvements
+
+   The current size of the review team introduces a fairly heavy
+   workload for the individual reviewers that are not on the "one doc
+   per month" assignment cycle.  Additional reviewers would be really
+   helpful to alleviate this workload.  It is also important to note
+   that having additional reviewers adds minimal workload to the
+
+
+
+Barnes, et al.                Informational                    [Page 18]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+   secretary's process; thus, the only blocking point is finding the
+   right folks that are interested in this type of volunteer role.  As
+   noted in Section 7.2, 30 would be a good size for the review team.
+   This would cut the workload for an individual reviewer in half (given
+   the current model of 9 reviewers on the "one doc per month"
+   assignment cycle).
+
+   Obviously, automation of the process would be a good thing.  However,
+   Gen-ART secretaries are not necessarily highly motivated to
+   transition to a more automated approach until a significant part of
+   the process is automated.  In more recent consideration of this
+   situation, it likely would be best to first automate the process of
+   entering the reviews, as that benefits the review team as a whole.
+   This automation should allow the reviewers to enter the reviews via a
+   web interface that would automatically generate the appropriate
+   emails -- quite similar to how the draft "Upload" tool currently
+   works.  Also, given consistent naming conventions for the review
+   forms, this step would automate some of the process for the
+   secretary, as the reviews would automatically appear via the
+   spreadsheet hyperlinks, although there would still be a need to
+   manually enter the summary.  But this would eliminate the need to
+   edit/normalize and upload files and, hopefully, eliminate the problem
+   encountered with unflowed text in emails and getting the review
+   properly formatted using some text editors.
+
+   Section 5 was written to facilitate the process of determining tools
+   requirements, by providing the very detailed steps currently applied
+   to the process.  As noted above, automating the upload of the reviews
+   could be a good first step.  This is somewhat starting at the end of
+   the process.  However, it seems that by automating in this direction,
+   we may have optimal results; since one of the earliest steps in the
+   process is the task of assigning reviewers, it likely needs the most
+   manual intervention, even with tools available.
+
+   The current SecDir secretary does use some tools for assignments and
+   generating assignment emails.  These tools could be considered for
+   use by the Gen-ART secretary.  Since the SecDir reviews are not
+   cached and the information maintained for those reviews is less
+   detailed, there would be no reusability of that aspect.  However, if
+   the Gen-ART spreadsheet can be automatically populated (with
+   assignments and completed reviews), the SecDir may be able to make
+   use of that same tool.
+
+
+
+
+
+
+
+
+
+Barnes, et al.                Informational                    [Page 19]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+9.  Applicability
+
+   As implemented today, the process has no formal role in the IETF
+   standards process.  But as trust in the review team has built, and as
+   the team itself has learned to deliver reviews that are generally
+   well received, they have had a significant impact on I-D quality and
+   on timeliness.  Rather than becoming a roadblock, they have (in
+   general) allowed the General AD to feel more confident in reaching
+   decisions and be more precise in resolving issues.  Since reviews now
+   typically appear during IETF Last Call, the reviews, like the SecDir
+   reviews, are now generally expected.  So, the role of the team has
+   evolved to be more formal than in the past (i.e., when this document
+   was first drafted in 2005).  However, the handling of the reviews
+   remains entirely within the scope of the ADs, document shepherds, WG
+   chairs, and authors as they deem appropriate.
+
+10.  Security Considerations
+
+   Since this is an informational I-D about an open process, the
+   security considerations are specific to the process and users
+   involved in the process.  The primary concern would be to limit the
+   people that have the ability to create and update the Gen-ART data/
+   files to ensure that the integrity of the data is maintained.  For
+   example, each Gen-ART reviewer should have a unique user name/
+   password, just as folks do to access any other IETF-maintained data,
+   as appropriate.
+
+11.  Acknowledgments
+
+   Initial comments were received from the members of the Gen-ART, and
+   the experiences discussed in this document were derived from their
+   hard work over the last 7+ years.  We thank the past reviewers of the
+   Gen-ART:
+
+      Mark Allman
+      Harald Alvestrand (creator of Gen-ART)
+      Ron Bonica
+      Scott Brim
+      Gonzalo Camarillo
+      Sharon Chisholm
+      Spencer Dawkins
+      Lakshminath Dondeti
+      Avri Doria (past secretary)
+      Pasi Eronen
+      Dorothy Gellert
+      Eric Gray
+      Avashalom Houri
+      Glenn Kowack
+
+
+
+Barnes, et al.                Informational                    [Page 20]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+      John Loughney
+      Lucy Lynch
+      Enrico Marocco
+      Michael Patton
+      Stefan Santesson
+      Robert Sparks
+      Tom Taylor
+      Sean Turner
+      Christian Vogt
+      Suzanne Woolf
+
+   We also thank the current team of reviewers/secretary:
+
+      Mary Barnes (past secretary, 2005-2010)
+      Richard Barnes
+      David Black
+      Ben Campbell
+      Brian Carpenter (past General AD)
+      Elwyn Davies
+      Francis Dupont
+      Roni Even
+      Miguel-Angel Garcia
+      Vijay Gurbani (assisting secretary to upload reviews)
+      Wassim Haddad
+      Joel Halpern
+      Suresh Krishnan
+      Peter McCann
+      Jean Mahoney (secretary as of Jan. 2011)
+      Alexey Melnikov
+      Kathleen Moriarty
+
+12.  Informative References
+
+   [RFC1958]     Carpenter, B., Ed., "Architectural Principles of the
+                 Internet", RFC 1958, June 1996.
+
+   [RFC2223bis]  Reynolds, J., Ed., and R. Braden, Ed., "Instructions to
+                 Request for Comments (RFC) Authors", Work in Progress,
+                 August 2004.
+
+   [RFC-STYLE]   Braden, R., Ginoza, S., and A. Hagens, "RFC Document
+                 Style", September 2009,
+                 <http://www.rfc-editor.org/rfc-style-guide/rfc-style>.
+
+   [RFC3426]     Floyd, S., "General Architectural and Policy
+                 Considerations", RFC 3426, November 2002.
+
+
+
+
+
+Barnes, et al.                Informational                    [Page 21]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+   [RFC3439]     Bush, R. and D. Meyer, "Some Internet Architectural
+                 Guidelines and Philosophy", RFC 3439, December 2002.
+
+   [RFC3552]     Rescorla, E. and B. Korver, "Guidelines for Writing RFC
+                 Text on Security Considerations", BCP 72, RFC 3552,
+                 July 2003.
+
+   [RFC5226]     Narten, T. and H. Alvestrand, "Guidelines for Writing
+                 an IANA Considerations Section in RFCs", BCP 26,
+                 RFC 5226, May 2008.
+
+   [SIRS]        Carpenter, B. and D. Crocker, "Careful Additional
+                 Review of Documents (CARD) by Senior IETF Reviewers
+                 (SIRS)", Work in Progress, June 2003.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Barnes, et al.                Informational                    [Page 22]
+
+RFC 6385                         Gen-ART                    October 2011
+
+
+Authors' Addresses
+
+   Mary Barnes
+   Polycom
+   TX
+   USA
+
+   EMail: mary.ietf.barnes@gmail.com
+
+
+   Avri Doria
+   Research Consultant
+   Providence, RI
+   USA
+
+   EMail: avri@acm.org
+
+
+   Harald Alvestrand
+   Google
+   Kungsbron 2
+   11122 Stockholm
+   SE
+
+   EMail: harald@alvestrand.no
+
+
+   Brian E. Carpenter
+   University of Auckland
+   PB 92019
+   Auckland, 1142
+   New Zealand
+
+   Phone:
+   EMail: brian.e.carpenter@gmail.com
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Barnes, et al.                Informational                    [Page 23]
+