summaryrefslogtreecommitdiff
path: root/doc/rfc/rfc5657.txt
diff options
context:
space:
mode:
Diffstat (limited to 'doc/rfc/rfc5657.txt')
-rw-r--r--doc/rfc/rfc5657.txt675
1 files changed, 675 insertions, 0 deletions
diff --git a/doc/rfc/rfc5657.txt b/doc/rfc/rfc5657.txt
new file mode 100644
index 0000000..83267a1
--- /dev/null
+++ b/doc/rfc/rfc5657.txt
@@ -0,0 +1,675 @@
+
+
+
+
+
+
+Network Working Group L. Dusseault
+Request for Comments: 5657 Messaging Architects
+BCP: 9 R. Sparks
+Updates: 2026 Tekelec
+Category: Best Current Practice September 2009
+
+
+ Guidance on Interoperation and Implementation Reports
+ for Advancement to Draft Standard
+
+Abstract
+
+ Advancing a protocol to Draft Standard requires documentation of the
+ interoperation and implementation of the protocol. Historic reports
+ have varied widely in form and level of content and there is little
+ guidance available to new report preparers. This document updates
+ the existing processes and provides more detail on what is
+ appropriate in an interoperability and implementation report.
+
+Status of This Memo
+
+ This document specifies an Internet Best Current Practices for the
+ Internet Community, and requests discussion and suggestions for
+ improvements. Distribution of this memo is unlimited.
+
+Copyright and License Notice
+
+ Copyright (c) 2009 IETF Trust and the persons identified as the
+ document authors. All rights reserved.
+
+ This document is subject to BCP 78 and the IETF Trust's Legal
+ Provisions Relating to IETF Documents
+ (http://trustee.ietf.org/license-info) in effect on the date of
+ publication of this document. Please review these documents
+ carefully, as they describe your rights and restrictions with respect
+ to this document. Code Components extracted from this document must
+ include Simplified BSD License text as described in Section 4.e of
+ the Trust Legal Provisions and are provided without warranty as
+ described in the BSD License.
+
+
+
+
+
+
+
+
+
+
+
+
+Dusseault & Sparks Best Current Practice [Page 1]
+
+RFC 5657 Implementation Report Guidance September 2009
+
+
+Table of Contents
+
+ 1. Introduction ....................................................2
+ 2. Content Requirements ............................................4
+ 3. Format ..........................................................5
+ 4. Feature Coverage ................................................6
+ 5. Special Cases ...................................................8
+ 5.1. Deployed Protocols .........................................8
+ 5.2. Undeployed Protocols .......................................8
+ 5.3. Schemas, Languages, and Formats ............................8
+ 5.4. Multiple Contributors, Multiple Implementation Reports .....9
+ 5.5. Test Suites ................................................9
+ 5.6. Optional Features, Extensibility Features .................10
+ 6. Examples .......................................................10
+ 6.1. Minimal Implementation Report .............................11
+ 6.2. Covering Exceptions .......................................11
+ 7. Security Considerations ........................................11
+ 8. References .....................................................12
+ 8.1. Normative References ......................................12
+ 8.2. Informative References ....................................12
+
+1. Introduction
+
+ The Draft Standard level, and requirements for standards to meet it,
+ are described in [RFC2026]. For Draft Standard, not only must two
+ implementations interoperate, but also documentation (the report)
+ must be provided to the IETF. The entire paragraph covering this
+ documentation reads:
+
+ The Working Group chair is responsible for documenting the
+ specific implementations which qualify the specification for Draft
+ or Internet Standard status along with documentation about testing
+ of the interoperation of these implementations. The documentation
+ must include information about the support of each of the
+ individual options and features. This documentation should be
+ submitted to the Area Director with the protocol action request.
+ (see Section 6)
+
+ Moving documents along the standards track can be an important signal
+ to the user and implementor communities, and the process of
+ submitting a standard for advancement can help improve that standard
+ or the quality of implementations that participate. However, the
+ barriers seem to be high for advancement to Draft Standard, or at the
+ very least confusing. This memo may help in guiding people through
+ one part of advancing specifications to Draft Standard. It also
+ changes some of the requirements made in RFC 2026 in ways that are
+ intended to maintain or improve the quality of reports while reducing
+ the burden of creating them.
+
+
+
+Dusseault & Sparks Best Current Practice [Page 2]
+
+RFC 5657 Implementation Report Guidance September 2009
+
+
+ Having and demonstrating sufficient interoperability is a gating
+ requirement for advancing a protocol to Draft Standard. Thus, the
+ primary goal of an implementation report is to convince the IETF and
+ the IESG that the protocol is ready for Draft Standard. This goal
+ can be met by summarizing the interoperability characteristics and by
+ providing just enough detail to support that conclusion. Side
+ benefits may accrue to the community creating the report in the form
+ of bugs found or fixed in tested implementations, documentation that
+ can help future implementors, or ideas for other documents or future
+ revisions of the protocol being tested.
+
+ Different kinds of documentation are appropriate for widely deployed
+ standards than for standards that are not yet deployed. Different
+ test approaches are appropriate for standards that are not typical
+ protocols: languages, formats, schemas, etc. This memo discusses how
+ reports for these standards may vary in Section 5.
+
+ Implementation should naturally focus on the final version of the
+ RFC. If there's any evidence that implementations are interoperating
+ based on Internet-Drafts or earlier versions of the specification, or
+ if interoperability was greatly aided by mailing list clarifications,
+ this should be noted in the report.
+
+ The level of detail in reports accepted in the past has varied
+ widely. An example of a submitted report that is not sufficient for
+ demonstrating interoperability is (in its entirety): "A partial list
+ of implementations include: Cray SGI Netstar IBM HP Network Systems
+ Convex". This report does not state how it is known that these
+ implementations interoperate (was it through public lab testing?
+ internal lab testing? deployment?). Nor does it capture whether
+ implementors are aware of, or were asked about, any features that
+ proved to be problematic. At a different extreme, reports have been
+ submitted that contain a great amount of detail about the test
+ methodology, but relatively little information about what worked and
+ what failed to work.
+
+ This memo is intended to clarify what an implementation report should
+ contain and to suggest a reasonable form for most implementation
+ reports. It is not intended to rule out good ideas. For example,
+ this memo can't take into account all process variations such as
+ documents going to Draft Standard twice, nor can it consider all
+ types of standards. Whenever the situation varies significantly from
+ what's described here, the IESG uses judgement in determining whether
+ an implementation report meets the goals above.
+
+ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+ "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
+ document are to be interpreted as described in BCP 14 [RFC2119].
+
+
+
+Dusseault & Sparks Best Current Practice [Page 3]
+
+RFC 5657 Implementation Report Guidance September 2009
+
+
+2. Content Requirements
+
+ The implementation report MUST identify the author of the report, who
+ is responsible for characterizing the interoperability quality of the
+ protocol. The report MAY identify other contributors (testers, those
+ who answered surveys, or those who contributed information) to share
+ credit or blame. The report MAY provide a list of report reviewers
+ who corroborate the characterization of interoperability quality, or
+ name an active working group (WG) that reviewed the report.
+
+ Some of the requirements of RFC 2026 are relaxed with this update:
+
+ o The report MAY name exactly which implementations were tested. A
+ requirement to name implementations was implied by the description
+ of the responsibility for "documenting the specific
+ implementations" in RFC 2026. However, note that usually
+ identifying implementations will help meet the goals of
+ implementation reports. If a subset of implementations was tested
+ or surveyed, it would also help to explain how that subset was
+ chosen or self-selected. See also the note on implementation
+ independence below.
+
+ o The report author MAY choose an appropriate level of detail to
+ document feature interoperability, rather than document each
+ individual feature. See note on granularity of features below.
+
+ o A contributor other than a WG chair MAY submit an implementation
+ report to an Area Director (AD).
+
+ o Optional features that are not implemented, but are important and
+ do not harm interoperability, MAY, exceptionally and with approval
+ of the IESG, be left in a protocol at Draft Standard. See
+ Section 5.6 for documentation requirements and an example of where
+ this is needed.
+
+ Note: Independence of implementations is mentioned in the RFC 2026
+ requirements for Draft Standard status. Independent
+ implementations should be written by different people at
+ different organizations using different code and protocol
+ libraries. If it's necessary to relax this definition, it can
+ be relaxed as long as there is evidence to show that success is
+ due more to the quality of the protocol than to out-of-band
+ understandings or common code. If there are only two
+ implementations of an undeployed protocol, the report SHOULD
+ identify the implementations and their "genealogy" (which
+ libraries were used or where the codebase came from). If there
+ are many more implementations, or the protocol is in broad
+ deployment, it is not necessary to call out which two of the
+
+
+
+Dusseault & Sparks Best Current Practice [Page 4]
+
+RFC 5657 Implementation Report Guidance September 2009
+
+
+ implementations demonstrated interoperability of each given
+ feature -- a reader may conclude that at least some of the
+ implementations of that feature are independent.
+
+ Note: The granularity of features described in a specification is
+ necessarily very detailed. In contrast, the granularity of an
+ implementation report need not be as detailed. A report need
+ not list every "MAY", "SHOULD", and "MUST" in a complete matrix
+ across implementations. A more effective approach might be to
+ characterize the interoperability quality and testing approach,
+ then call out any known problems in either testing or
+ interoperability.
+
+3. Format
+
+ The format of implementation and interoperability reports MUST be
+ ASCII text with line breaks for readability. As with Internet-
+ Drafts, no 8-bit characters are currently allowed. It is acceptable,
+ but not necessary, for a report to be formatted as an Internet-Draft.
+
+ Here is a simple outline that an implementation report MAY follow in
+ part or in full:
+
+ Title: Titles of implementation reports are strongly RECOMMENDED to
+ contain one or more RFC number for consistent lookup in a simple
+ archive. In addition, the name or a common mnemonic of the
+ standard should be in the title. An example might look like
+ "Implementation Report for the Example Name of Some Protocol
+ (ENSP) RFC XXXX".
+
+ Author: Identify the author of the report.
+
+ Summary: Attest that the standard meets the requirements for Draft
+ Standard and name who is attesting it. Describe how many
+ implementations were tested or surveyed. Quickly characterize the
+ deployment level and where the standard can be found in
+ deployment. Call out, and if possible, briefly describe any
+ notably difficult or poorly interoperable features and explain why
+ these still meet the requirement. Assert any derivative
+ conclusions: if a high-level system is tested and shown to work,
+ then we may conclude that the normative requirements of that
+ system (all sub-system or lower-layer protocols, to the extent
+ that a range of features is used) have also been shown to work.
+
+ Methodology: Describe how the information in the report was
+ obtained. This should be no longer than the summary.
+
+
+
+
+
+Dusseault & Sparks Best Current Practice [Page 5]
+
+RFC 5657 Implementation Report Guidance September 2009
+
+
+ Exceptions: This section might read "Every feature was implemented,
+ tested, and widely interoperable without exception and without
+ question". If that statement is not true, then this section
+ should cover whether any features were thought to be problematic.
+ Problematic features need not disqualify a protocol from Draft
+ Standard, but this section should explain why they do not (e.g.,
+ optional, untestable, trace, or extension features). See the
+ example in Section 6.2.
+
+ Detail sections: Any other justifying or background information can
+ be included here. In particular, any information that would have
+ made the summary or methodology sections more than a few
+ paragraphs long may be created as a detail section and referenced.
+
+ In this section, it would be good to discuss how the various
+ considerations sections played out. Were the security
+ considerations accurate and dealt with appropriately in
+ implementations? Was real internationalization experience found
+ among the tested implementations? Did the implementations have
+ any common monitoring or management functionality (although note
+ that documenting the interoperability of a management standard
+ might be separate from documenting the interoperability of the
+ protocol itself)? Did the IANA registries or registrations, if
+ any, work as intended?
+
+ Appendix sections: It's not necessary to archive test material such
+ as test suites, test documents, questionnaire text, or
+ questionnaire responses. However, if it's easy to preserve this
+ information, appendix sections allow readers to skip over it if
+ they are not interested. Preserving detailed test information can
+ help people doing similar or follow-on implementation reports, and
+ can also help new implementors.
+
+4. Feature Coverage
+
+ What constitutes a "feature" for the purposes of an interoperability
+ report has been frequently debated. Good judgement is required in
+ finding a level of detail that adequately demonstrates coverage of
+ the requirements. Statements made at too high a level will result in
+ a document that can't be verified and hasn't adequately challenged
+ that the testing accidentally missed an important failure to
+ interoperate. On the other hand, statements at too fine a level
+ result in an exponentially exploding matrix of requirement
+ interaction that overburdens the testers and report writers. The
+ important information in the resulting report would likely be hard to
+ find in the sea of detail, making it difficult to evaluate whether
+ the important points of interoperability have been addressed.
+
+
+
+
+Dusseault & Sparks Best Current Practice [Page 6]
+
+RFC 5657 Implementation Report Guidance September 2009
+
+
+ The best interoperability reports will organize statements of
+ interoperability at a level of detail just sufficient to convince the
+ reader that testing has covered the full set of requirements and in
+ particular that the testing was sufficient to uncover any places
+ where interoperability does not exist. Reports similar to that for
+ RTP/RTCP (an excerpt appears below) are more useful than an
+ exhaustive checklist of every normative statement in the
+ specification.
+
+ 10. Interoperable exchange of receiver report packets.
+
+ o PASS: Many implementations, tested UCL rat with vat,
+ Cisco IP/TV with vat/vic.
+
+ 11. Interoperable exchange of receiver report packets when
+ not receiving data (ie: the empty receiver report
+ which has to be sent first in each compound RTCP packet
+ when no-participants are transmitting data).
+
+ o PASS: Many implementations, tested UCL rat with vat,
+ Cisco IP/TV with vat/vic.
+
+ ...
+
+ 8. Interoperable transport of RTP via TCP using the
+ encapsulation defined in the audio/video profile
+
+ o FAIL: no known implementations. This has been
+ removed from the audio/video profile.
+
+
+ Excerpts from
+ http://www.ietf.org/iesg/implementation/report-avt-rtp-rtcp.txt
+
+ Consensus can be a good tool to help determine the appropriate level
+ for such feature descriptions. A working group can make a strong
+ statement by documenting its consensus that a report sufficiently
+ covers a specification and that interoperability has been
+ demonstrated.
+
+
+
+
+
+
+
+
+
+
+
+
+Dusseault & Sparks Best Current Practice [Page 7]
+
+RFC 5657 Implementation Report Guidance September 2009
+
+
+5. Special Cases
+
+5.1. Deployed Protocols
+
+ When a protocol is deployed, results obtained from laboratory testing
+ are not as useful to the IETF as learning what is actually working in
+ deployment. To this end, it may be more informative to survey
+ implementors or operators. A questionnaire or interview can elicit
+ information from a wider number of sources. As long as it is known
+ that independent implementations can work in deployment, it is more
+ useful to discover what problems exist, rather than gather long and
+ detailed checklists of features and options.
+
+5.2. Undeployed Protocols
+
+ It is appropriate to provide finer-grained detail in reports for
+ protocols that do not yet have a wealth of experience gained through
+ deployment. In particular, some complicated, flexible or powerful
+ features might show interoperability problems when testers start to
+ probe outside the core use cases. RFC 2026 requires "sufficient
+ successful operational experience" before progressing a standard to
+ Draft, and notes that:
+
+ Draft Standard may still require additional or more widespread
+ field experience, since it is possible for implementations based
+ on Draft Standard specifications to demonstrate unforeseen
+ behavior when subjected to large-scale use in production
+ environments.
+
+ When possible, reports for protocols without much deployment
+ experience should anticipate common operational considerations. For
+ example, it would be appropriate to put additional emphasis on
+ overload or congestion management features the protocol may have.
+
+5.3. Schemas, Languages, and Formats
+
+ Standards that are not on-the-wire protocols may be special cases for
+ implementation reports. The IESG SHOULD use judgement in what kind
+ of implementation information is acceptable for these kinds of
+ standards. ABNF (RFC 4234) is an example of a language for which an
+ implementation report was filed: it is interoperable in that
+ protocols are specified using ABNF and these protocols can be
+ successfully implemented and syntax verified. Implementations of
+ ABNF include the RFCs that use it as well as ABNF checking software.
+ Management Information Base (MIB, [RFC3410]) modules are sometimes
+ documented in implementation reports, and examples of that can be
+ found in the archive of implementation reports.
+
+
+
+
+Dusseault & Sparks Best Current Practice [Page 8]
+
+RFC 5657 Implementation Report Guidance September 2009
+
+
+ The interoperability reporting requirements for some classes of
+ documents may be discussed in separate documents. See [METRICSTEST]
+ for example.
+
+5.4. Multiple Contributors, Multiple Implementation Reports
+
+ If it's easiest to divide up the work of implementation reports by
+ implementation, the result -- multiple implementation reports -- MAY
+ be submitted to the sponsoring Area Director one-by-one. Each report
+ might cover one implementation, including:
+
+ identification of the implementation;
+
+ an affirmation that the implementation works in testing (or
+ better, in deployment);
+
+ whether any features are known to interoperate poorly with other
+ implementations;
+
+ which optional or required features are not implemented (note that
+ there are no protocol police to punish this disclosure, we should
+ instead thank implementors who point out unimplemented or
+ unimplementable features especially if they can explain why); and
+
+ who is submitting this report for this implementation.
+
+ These SHOULD be collated into one document for archiving under one
+ title, but can be concatenated trivially even if the result has
+ several summary sections or introductions.
+
+5.5. Test Suites
+
+ Some automated tests, such as automated test clients, do not test
+ interoperability directly. When specialized test implementations are
+ necessary, tests can at least be constructed from real-world protocol
+ or document examples. For example:
+
+ - ABNF [RFC4234] itself was tested by combining real-world examples
+ -- uses of ABNF found in well-known RFCs -- and feeding those
+ real-world examples into ABNF checkers. As the well-known RFCs
+ were themselves interoperable and in broad deployment, this served
+ as both a deployment proof and an interoperability proof.
+ [RFC4234] progressed from Proposed Standard through Draft Standard
+ to Standard and is obsoleted by [RFC5234].
+
+
+
+
+
+
+
+Dusseault & Sparks Best Current Practice [Page 9]
+
+RFC 5657 Implementation Report Guidance September 2009
+
+
+ - Atom [RFC4287] clients might be tested by finding that they
+ consistently display the information in a test Atom feed,
+ constructed from real-world examples that cover all the required
+ and optional features.
+
+ - MIB modules can be tested with generic MIB browsers, to confirm
+ that different implementations return the same values for objects
+ under similar conditions.
+
+ As a counter-example, the automated WWW Distributed Authoring and
+ Versioning (WebDAV) test client Litmus
+ (http://www.webdav.org/neon/litmus/) is of limited use in
+ demonstrating interoperability for WebDAV because it tests
+ completeness of server implementations and simple test cases. It
+ does not test real-world use or whether any real WebDAV clients
+ implement a feature properly or at all.
+
+5.6. Optional Features, Extensibility Features
+
+ Optional features need not be shown to be implemented everywhere.
+ However, they do need to be implemented somewhere, and more than one
+ independent implementation is required. If an optional feature does
+ not meet this requirement, the implementation report must say so and
+ explain why the feature must be kept anyway versus being evidence of
+ a poor-quality standard.
+
+ Extensibility points and versioning features are particularly likely
+ to need this kind of treatment. When a protocol version 1 is
+ released, the protocol version field itself is likely to be unused.
+ Before any other versions exist, it can't really be demonstrated that
+ this particular field or option is implemented.
+
+6. Examples
+
+ Some good, extremely brief, examples of implementation reports can be
+ found in the archives:
+
+ http://www.ietf.org/iesg/implementation/report-ppp-lcp-ext.html
+
+ http://www.ietf.org/iesg/implementation/report-otp.html
+
+ In some cases, perfectly good implementation reports are longer than
+ necessary, but may preserve helpful information:
+
+ http://www.ietf.org/iesg/implementation/report-rfc2329.txt
+
+ http://www.ietf.org/iesg/implementation/report-rfc4234.txt
+
+
+
+
+Dusseault & Sparks Best Current Practice [Page 10]
+
+RFC 5657 Implementation Report Guidance September 2009
+
+
+6.1. Minimal Implementation Report
+
+ A large number of SMTP implementations support SMTP pipelining,
+ including: (1) Innosoft's PMDF and Sun's SIMS. (2) ISODE/
+ MessagingDirect's PP. (3) ISOCOR's nPlex. (4) software.com's
+ post.office. (5) Zmailer. (6) Smail. (7) The SMTP server in
+ Windows 2000. SMTP pipelining has been widely deployed in these
+ and other implementations for some time, and there have been no
+ reported interoperability problems.
+
+ This implementation report can also be found at
+ http://www.ietf.org//iesg/implementation/report-smtp-pipelining.txt
+ but the entire report is already reproduced above. Since SMTP
+ pipelining had no interoperability problems, the implementation
+ report was able to provide all the key information in a very terse
+ format. The reader can infer from the different vendors and
+ platforms that the codebases must, by and in large, be independent.
+
+ This implementation report would only be slightly improved by a
+ positive affirmation that there have been probes or investigations
+ asking about interoperability problems rather than merely a lack of
+ problem reports, and by stating who provided this summary report.
+
+6.2. Covering Exceptions
+
+ The RFC2821bis (SMTP) implementation survey asked implementors what
+ features were not implemented. The VRFY and EXPN commands showed up
+ frequently in the responses as not implemented or disabled. That
+ implementation report might have followed the advice in this
+ document, had it already existed, by justifying the interoperability
+ of those features up front or in an "exceptions" section if the
+ outline defined in this memo were used:
+
+ VRFY and EXPN commands are often not implemented or are disabled.
+ This does not pose an interoperability problem for SMTP because
+ EXPN is an optional features and its support is never relied on.
+ VRFY is required, but in practice it is not relied on because
+ servers can legitimately reply with a non-response. These
+ commands should remain in the standard because they are sometimes
+ used by administrators within a domain under controlled
+ circumstances (e.g. authenticated query from within the domain).
+ Thus, the occasional utility argues for keeping these features,
+ while the lack of problems for end-users means that the
+ interoperability of SMTP in real use is not in the least degraded.
+
+7. Security Considerations
+
+ This memo introduces no new security considerations.
+
+
+
+Dusseault & Sparks Best Current Practice [Page 11]
+
+RFC 5657 Implementation Report Guidance September 2009
+
+
+8. References
+
+8.1. Normative References
+
+ [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
+ Requirement Levels", BCP 14, RFC 2119, March 1997.
+
+8.2. Informative References
+
+ [METRICSTEST] Bradner, S. and V. Paxson, "Advancement of metrics
+ specifications on the IETF Standards Track", Work
+ in Progress, July 2007.
+
+ [RFC2026] Bradner, S., "The Internet Standards Process --
+ Revision 3", BCP 9, RFC 2026, October 1996.
+
+ [RFC3410] Case, J., Mundy, R., Partain, D., and B. Stewart,
+ "Introduction and Applicability Statements for
+ Internet-Standard Management Framework", RFC 3410,
+ December 2002.
+
+ [RFC4234] Crocker, D., Ed. and P. Overell, "Augmented BNF for
+ Syntax Specifications: ABNF", RFC 4234, October 2005.
+
+ [RFC4287] Nottingham, M., Ed. and R. Sayre, Ed., "The Atom
+ Syndication Format", RFC 4287, December 2005.
+
+ [RFC5234] Crocker, D. and P. Overell, "Augmented BNF for Syntax
+ Specifications: ABNF", STD 68, RFC 5234, January 2008.
+
+Authors' Addresses
+
+ Lisa Dusseault
+ Messaging Architects
+
+ EMail: lisa.dusseault@gmail.com
+
+
+ Robert Sparks
+ Tekelec
+ 17210 Campbell Road
+ Suite 250
+ Dallas, Texas 75254-4203
+ USA
+
+ EMail: RjS@nostrum.com
+
+
+
+
+
+Dusseault & Sparks Best Current Practice [Page 12]
+