doc: Add RFC documents

1 files changed, 2411 insertions, 0 deletions

diff --git a/doc/rfc/rfc5894.txt b/doc/rfc/rfc5894.txt
new file mode 100644
index 0000000..1d664e1
--- /dev/null
+++ b/doc/rfc/rfc5894.txt
@@ -0,0 +1,2411 @@
+
+
+
+
+
+
+Internet Engineering Task Force (IETF)                        J. Klensin
+Request for Comments: 5894                                   August 2010
+Category: Informational
+ISSN: 2070-1721
+
+
+        Internationalized Domain Names for Applications (IDNA):
+                 Background, Explanation, and Rationale
+
+Abstract
+
+   Several years have passed since the original protocol for
+   Internationalized Domain Names (IDNs) was completed and deployed.
+   During that time, a number of issues have arisen, including the need
+   to update the system to deal with newer versions of Unicode.  Some of
+   these issues require tuning of the existing protocols and the tables
+   on which they depend.  This document provides an overview of a
+   revised system and provides explanatory material for its components.
+
+Status of This Memo
+
+   This document is not an Internet Standards Track specification; it is
+   published for informational purposes.
+
+   This document is a product of the Internet Engineering Task Force
+   (IETF).  It represents the consensus of the IETF community.  It has
+   received public review and has been approved for publication by the
+   Internet Engineering Steering Group (IESG).  Not all documents
+   approved by the IESG are a candidate for any level of Internet
+   Standard; see Section 2 of RFC 5741.
+
+   Information about the current status of this document, any errata,
+   and how to provide feedback on it may be obtained at
+   http://www.rfc-editor.org/info/rfc5894.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Klensin                       Informational                     [Page 1]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+Copyright Notice
+
+   Copyright (c) 2010 IETF Trust and the persons identified as the
+   document authors.  All rights reserved.
+
+   This document is subject to BCP 78 and the IETF Trust's Legal
+   Provisions Relating to IETF Documents
+   (http://trustee.ietf.org/license-info) in effect on the date of
+   publication of this document.  Please review these documents
+   carefully, as they describe your rights and restrictions with respect
+   to this document.  Code Components extracted from this document must
+   include Simplified BSD License text as described in Section 4.e of
+   the Trust Legal Provisions and are provided without warranty as
+   described in the Simplified BSD License.
+
+   This document may contain material from IETF Documents or IETF
+   Contributions published or made publicly available before November
+   10, 2008.  The person(s) controlling the copyright in some of this
+   material may not have granted the IETF Trust the right to allow
+   modifications of such material outside the IETF Standards Process.
+   Without obtaining an adequate license from the person(s) controlling
+   the copyright in such materials, this document may not be modified
+   outside the IETF Standards Process, and derivative works of it may
+   not be created outside the IETF Standards Process, except to format
+   it for publication as an RFC or to translate it into languages other
+   than English.
+
+Table of Contents
+
+   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  4
+     1.1.  Context and Overview . . . . . . . . . . . . . . . . . . .  4
+     1.2.  Terminology  . . . . . . . . . . . . . . . . . . . . . . .  5
+       1.2.1.  DNS "Name" Terminology . . . . . . . . . . . . . . . .  5
+       1.2.2.  New Terminology and Restrictions . . . . . . . . . . .  6
+     1.3.  Objectives . . . . . . . . . . . . . . . . . . . . . . . .  6
+     1.4.  Applicability and Function of IDNA . . . . . . . . . . . .  7
+     1.5.  Comprehensibility of IDNA Mechanisms and Processing  . . .  8
+   2.  Processing in IDNA2008 . . . . . . . . . . . . . . . . . . . .  9
+   3.  Permitted Characters: An Inclusion List  . . . . . . . . . . .  9
+     3.1.  A Tiered Model of Permitted Characters and Labels  . . . . 10
+       3.1.1.  PROTOCOL-VALID . . . . . . . . . . . . . . . . . . . . 10
+       3.1.2.  CONTEXTUAL RULE REQUIRED . . . . . . . . . . . . . . . 11
+         3.1.2.1.  Contextual Restrictions  . . . . . . . . . . . . . 11
+         3.1.2.2.  Rules and Their Application  . . . . . . . . . . . 12
+       3.1.3.  DISALLOWED . . . . . . . . . . . . . . . . . . . . . . 12
+       3.1.4.  UNASSIGNED . . . . . . . . . . . . . . . . . . . . . . 13
+     3.2.  Registration Policy  . . . . . . . . . . . . . . . . . . . 14
+
+
+
+
+Klensin                       Informational                     [Page 2]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+     3.3.  Layered Restrictions: Tables, Context, Registration, and
+           Applications . . . . . . . . . . . . . . . . . . . . . . . 15
+   4.  Application-Related Issues . . . . . . . . . . . . . . . . . . 15
+     4.1.  Display and Network Order  . . . . . . . . . . . . . . . . 15
+     4.2.  Entry and Display in Applications  . . . . . . . . . . . . 16
+     4.3.  Linguistic Expectations: Ligatures, Digraphs, and
+           Alternate Character Forms  . . . . . . . . . . . . . . . . 19
+     4.4.  Case Mapping and Related Issues  . . . . . . . . . . . . . 20
+     4.5.  Right-to-Left Text . . . . . . . . . . . . . . . . . . . . 21
+   5.  IDNs and the Robustness Principle  . . . . . . . . . . . . . . 22
+   6.  Front-end and User Interface Processing for Lookup . . . . . . 22
+   7.  Migration from IDNA2003 and Unicode Version Synchronization  . 25
+     7.1.  Design Criteria  . . . . . . . . . . . . . . . . . . . . . 25
+       7.1.1.  Summary and Discussion of IDNA Validity Criteria . . . 25
+       7.1.2.  Labels in Registration . . . . . . . . . . . . . . . . 26
+       7.1.3.  Labels in Lookup . . . . . . . . . . . . . . . . . . . 27
+     7.2.  Changes in Character Interpretations . . . . . . . . . . . 28
+       7.2.1.  Character Changes: Eszett and Final Sigma  . . . . . . 28
+       7.2.2.  Character Changes: Zero Width Joiner and Zero
+               Width Non-Joiner . . . . . . . . . . . . . . . . . . . 29
+       7.2.3.  Character Changes and the Need for Transition  . . . . 29
+       7.2.4.  Transition Strategies  . . . . . . . . . . . . . . . . 30
+     7.3.  Elimination of Character Mapping . . . . . . . . . . . . . 31
+     7.4.  The Question of Prefix Changes . . . . . . . . . . . . . . 31
+       7.4.1.  Conditions Requiring a Prefix Change . . . . . . . . . 31
+       7.4.2.  Conditions Not Requiring a Prefix Change . . . . . . . 32
+       7.4.3.  Implications of Prefix Changes . . . . . . . . . . . . 32
+     7.5.  Stringprep Changes and Compatibility . . . . . . . . . . . 33
+     7.6.  The Symbol Question  . . . . . . . . . . . . . . . . . . . 33
+     7.7.  Migration between Unicode Versions: Unassigned Code
+           Points . . . . . . . . . . . . . . . . . . . . . . . . . . 35
+     7.8.  Other Compatibility Issues . . . . . . . . . . . . . . . . 36
+   8.  Name Server Considerations . . . . . . . . . . . . . . . . . . 37
+     8.1.  Processing Non-ASCII Strings . . . . . . . . . . . . . . . 37
+     8.2.  Root and Other DNS Server Considerations . . . . . . . . . 37
+   9.  Internationalization Considerations  . . . . . . . . . . . . . 38
+   10. IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 38
+     10.1. IDNA Character Registry  . . . . . . . . . . . . . . . . . 38
+     10.2. IDNA Context Registry  . . . . . . . . . . . . . . . . . . 39
+     10.3. IANA Repository of IDN Practices of TLDs . . . . . . . . . 39
+   11. Security Considerations  . . . . . . . . . . . . . . . . . . . 39
+     11.1. General Security Issues with IDNA  . . . . . . . . . . . . 39
+   12. Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . . 39
+   13. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 40
+   14. References . . . . . . . . . . . . . . . . . . . . . . . . . . 40
+     14.1. Normative References . . . . . . . . . . . . . . . . . . . 40
+     14.2. Informative References . . . . . . . . . . . . . . . . . . 41
+
+
+
+
+Klensin                       Informational                     [Page 3]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+1.  Introduction
+
+1.1.  Context and Overview
+
+   Internationalized Domain Names in Applications (IDNA) is a collection
+   of standards that allow client applications to convert some mnemonic
+   strings expressed in Unicode to an ASCII-compatible encoding form
+   ("ACE") that is a valid DNS label containing only LDH syntax (see the
+   Definitions document [RFC5890]).  The specific form of ACE label used
+   by IDNA is called an "A-label".  A client can look up an exact
+   A-label in the existing DNS, so A-labels do not require any
+   extensions to DNS, upgrades of DNS servers, or updates to low-level
+   client libraries.  An A-label is recognizable from the prefix "xn--"
+   before the characters produced by the Punycode algorithm [RFC3492];
+   thus, a user application can identify an A-label and convert it into
+   Unicode (or some local coded character set) for display.
+
+   On the registry side, IDNA allows a registry to offer
+   Internationalized Domain Names (IDNs) for registration as A-labels.
+   A registry may offer any subset of valid IDNs, and may apply any
+   restrictions or bundling (grouping of similar labels together in one
+   registration) appropriate for the context of that registry.
+   Registration of labels is sometimes discussed separately from lookup,
+   and it is subject to a few specific requirements that do not apply to
+   lookup.
+
+   DNS clients and registries are subject to some differences in
+   requirements for handling IDNs.  In particular, registries are urged
+   to register only exact, valid A-labels, while clients might do some
+   mapping to get from otherwise-invalid user input to a valid A-label.
+
+   The first version of IDNA was published in 2003 and is referred to
+   here as IDNA2003 to contrast it with the current version, which is
+   known as IDNA2008 (after the year in which IETF work started on it).
+   IDNA2003 consists of four documents: the IDNA base specification
+   [RFC3490], Nameprep [RFC3491], Punycode [RFC3492], and Stringprep
+   [RFC3454].  The current set of documents, IDNA2008, is not dependent
+   on any of the IDNA2003 specifications other than the one for Punycode
+   encoding.  References to "IDNA2008", "these specifications", or
+   "these documents" are to the entire IDNA2008 set listed in a separate
+   Definitions document [RFC5890].  The characters that are valid in
+   A-labels are identified from rules listed in the Tables document
+   [RFC5892], but validity can be derived from the Unicode properties of
+   those characters with a very few exceptions.
+
+   Traditionally, DNS labels are matched case-insensitively (as
+   described in the DNS specifications [RFC1034][RFC1035]).  That
+   convention was preserved in IDNA2003 by a case-folding operation that
+
+
+
+Klensin                       Informational                     [Page 4]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   generally maps capital letters into lowercase ones.  However, if case
+   rules are enforced from one language, another language sometimes
+   loses the ability to treat two characters separately.  Case-
+   insensitivity is treated slightly differently in IDNA2008.
+
+   IDNA2003 used Unicode version 3.2 only.  In order to keep up with new
+   characters added in new versions of Unicode, IDNA2008 decouples its
+   rules from any particular version of Unicode.  Instead, the
+   attributes of new characters in Unicode, supplemented by a small
+   number of exception cases, determine how and whether the characters
+   can be used in IDNA labels.
+
+   This document provides informational context for IDNA2008, including
+   terminology, background, and policy discussions.  It contains no
+   normative material; specifications for conformance to the IDNA2008
+   protocols appears entirely in the other documents in the series.
+
+1.2.  Terminology
+
+   Terminology for IDNA2008 appears in the Definitions document
+   [RFC5890].  That document also contains a road map to the IDNA2008
+   document collection.  No attempt should be made to understand this
+   document without the definitions and concepts that appear there.
+
+1.2.1.  DNS "Name" Terminology
+
+   In the context of IDNs, the DNS term "name" has introduced some
+   confusion as people speak of DNS labels in terms of the words or
+   phrases of various natural languages.  Historically, many of the
+   "names" in the DNS have been mnemonics to identify some particular
+   concept, object, or organization.  They are typically rooted in some
+   language because most people think in language-based ways.  But,
+   because they are mnemonics, they need not obey the orthographic
+   conventions of any language: it is not a requirement that it be
+   possible for them to be "words".
+
+   This distinction is important because the reasonable goal of an IDN
+   effort is not to be able to write the great Klingon (or language of
+   one's choice) novel in DNS labels but to be able to form a usefully
+   broad range of mnemonics in ways that are as natural as possible in a
+   very broad range of scripts.
+
+
+
+
+
+
+
+
+
+
+Klensin                       Informational                     [Page 5]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+1.2.2.  New Terminology and Restrictions
+
+   IDNA2008 introduces new terminology.  Precise definitions are
+   provided in the Definitions document for the terms U-label, A-Label,
+   LDH label (to which all valid pre-IDNA hostnames conformed), Reserved
+   LDH label (R-LDH label), XN-label, Fake A-label, and Non-Reserved LDH
+   label (NR-LDH label).
+
+   In addition, the term "putative label" has been adopted to refer to a
+   label that may appear to meet certain definitional constraints but
+   has not yet been sufficiently tested for validity.
+
+   These definitions are also illustrated in Figure 1 of the Definitions
+   document.  R-LDH labels contain "--" in the third and fourth
+   character positions from the beginning of the label.  In IDNA-aware
+   applications, only a subset of these reserved labels is permitted to
+   be used, namely the A-label subset.  A-labels are a subset of the
+   R-LDH labels that begin with the case-insensitive string "xn--".
+   Labels that bear this prefix but that are not otherwise valid fall
+   into the "Fake A-label" category.  The Non-Reserved labels (NR-LDH
+   labels) are implicitly valid since they do not bear any resemblance
+   to the labels specified by IDNA.
+
+   The creation of the Reserved-LDH category is required for three
+   reasons:
+
+   o  to prevent confusion with pre-IDNA coding forms;
+
+   o  to permit future extensions that would require changing the
+      prefix, no matter how unlikely those might be (see Section 7.4);
+      and
+
+   o  to reduce the opportunities for attacks via the Punycode encoding
+      algorithm itself.
+
+   As with other documents in the IDNA2008 set, this document uses the
+   term "registry" to describe any zone in the DNS.  That term, and the
+   terms "zone" or "zone administration", are interchangeable.
+
+1.3.  Objectives
+
+   These are the main objectives in revising IDNA.
+
+   o  Use a more recent version of Unicode and allow IDNA to be
+      independent of Unicode versions, so that IDNA2008 need not be
+      updated for implementations to adopt code points from new Unicode
+      versions.
+
+
+
+
+Klensin                       Informational                     [Page 6]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   o  Fix a very small number of code point categorizations that have
+      turned out to cause problems in the communities that use those
+      code points.
+
+   o  Reduce the dependency on mapping, in favor of valid A-labels.
+      This will result in pre-mapped forms that are not valid IDNA
+      labels appearing less often in various contexts.
+
+   o  Fix some details in the bidirectional code point handling
+      algorithms.
+
+1.4.  Applicability and Function of IDNA
+
+   The IDNA specification solves the problem of extending the repertoire
+   of characters that can be used in domain names to include a large
+   subset of the Unicode repertoire.
+
+   IDNA does not extend DNS.  Instead, the applications (and, by
+   implication, the users) continue to see an exact-match lookup
+   service.  Either there is a single name that matches exactly (subject
+   to the base DNS requirement of case-insensitive ASCII matching) or
+   there is no match.  This model has served the existing applications
+   well, but it requires, with or without internationalized domain
+   names, that users know the exact spelling of the domain names that
+   are to be typed into applications such as web browsers and mail user
+   agents.  The introduction of the larger repertoire of characters
+   potentially makes the set of misspellings larger, especially given
+   that in some cases the same appearance, for example on a business
+   card, might visually match several Unicode code points or several
+   sequences of code points.
+
+   The IDNA standard does not require any applications to conform to it,
+   nor does it retroactively change those applications.  An application
+   can elect to use IDNA in order to support IDNs while maintaining
+   interoperability with existing infrastructure.  For applications that
+   want to use non-ASCII characters in public DNS domain names, IDNA is
+   the only option that is defined at the time this specification is
+   published.  Adding IDNA support to an existing application entails
+   changes to the application only, and leaves room for flexibility in
+   front-end processing and more specifically in the user interface (see
+   Section 6).
+
+   A great deal of the discussion of IDN solutions has focused on
+   transition issues and how IDNs will work in a world where not all of
+   the components have been updated.  Proposals that were not chosen by
+   the original IDN Working Group would have depended on updating user
+   applications, DNS resolvers, and DNS servers in order for a user to
+   apply an internationalized domain name in any form or coding
+
+
+
+Klensin                       Informational                     [Page 7]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   acceptable under that method.  While processing must be performed
+   prior to or after access to the DNS, IDNA requires no changes to the
+   DNS protocol, any DNS servers, or the resolvers on users' computers.
+
+   IDNA allows the graceful introduction of IDNs not only by avoiding
+   upgrades to existing infrastructure (such as DNS servers and mail
+   transport agents), but also by allowing some limited use of IDNs in
+   applications by using the ASCII-encoded representation of the labels
+   containing non-ASCII characters.  While such names are user-
+   unfriendly to read and type, and hence not optimal for user input,
+   they can be used as a last resort to allow rudimentary IDN usage.
+   For example, they might be the best choice for display if it were
+   known that relevant fonts were not available on the user's computer.
+   In order to allow user-friendly input and output of the IDNs and
+   acceptance of some characters as equivalent to those to be processed
+   according to the protocol, the applications need to be modified to
+   conform to this specification.
+
+   This version of IDNA uses the Unicode character repertoire for
+   continuity with the original version of IDNA.
+
+1.5.  Comprehensibility of IDNA Mechanisms and Processing
+
+   One goal of IDNA2008, which is aided by the main goal of reducing the
+   dependency on mapping, is to improve the general understanding of how
+   IDNA works and what characters are permitted and what happens to
+   them.  Comprehensibility and predictability to users and registrants
+   are important design goals for this effort.  End-user applications
+   have an important role to play in increasing this comprehensibility.
+
+   Any system that tries to handle international characters encounters
+   some common problems.  For example, a User Interface (UI) cannot
+   display a character if no font containing that character is
+   available.  In some cases, internationalization enables effective
+   localization while maintaining some global uniformity but losing some
+   universality.
+
+   It is difficult to even make suggestions as to how end-user
+   applications should cope when characters and fonts are not available.
+   Because display functions are rarely controlled by the types of
+   applications that would call upon IDNA, such suggestions will rarely
+   be very effective.
+
+   Conversion between local character sets and normalized Unicode, if
+   needed, is part of this set of user interface issues.  Those
+   conversions introduce complexity in a system that does not use
+   Unicode as its primary (or only) internal character coding system.
+   If a label is converted to a local character set that does not have
+
+
+
+Klensin                       Informational                     [Page 8]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   all the needed characters, or that uses different character-coding
+   principles, the user interface program may have to add special logic
+   to avoid or reduce loss of information.
+
+   The major difficulty may lie in accurately identifying the incoming
+   character set and applying the correct conversion routine.  Even more
+   difficult, the local character coding system could be based on
+   conceptually different assumptions than those used by Unicode (e.g.,
+   choice of font encodings used for publications in some Indic
+   scripts).  Those differences may not easily yield unambiguous
+   conversions or interpretations even if each coding system is
+   internally consistent and adequate to represent the local language
+   and script.
+
+   IDNA2008 shifts responsibility for character mapping and other
+   adjustments from the protocol (where it was located in IDNA2003) to
+   pre-processing before invoking IDNA itself.  The intent is that this
+   change will lead to greater usage of fully-valid A-Labels or U-labels
+   in display, transit, and storage, which should aid comprehensibility
+   and predictability.  A careful look at pre-processing raises issues
+   about what that pre-processing should do and at what point
+   pre-processing becomes harmful; how universally consistent
+   pre-processing algorithms can be; and how to be compatible with
+   labels prepared in an IDNA2003 context.  Those issues are discussed
+   in Section 6 and in the Mapping document [IDNA2008-Mapping].
+
+2.  Processing in IDNA2008
+
+   IDNA2008 separates Domain Name Registration and Lookup in the
+   protocol specification (RFC 5891, Sections 4 and 5 [RFC5891]).
+   Although most steps in the two processes are similar, the separation
+   reflects current practice in which per-registry (DNS zone)
+   restrictions and special processing are applied at registration time
+   but not during lookup.  Another significant benefit is that
+   separation facilitates incremental addition of permitted character
+   groups to avoid freezing on one particular version of Unicode.
+
+   The actual registration and lookup protocols for IDNA2008 are
+   specified in the Protocol document.
+
+3.  Permitted Characters: An Inclusion List
+
+   IDNA2008 adopts the inclusion model.  A code point is assumed to be
+   invalid for IDN use unless it is included as part of a Unicode
+   property-based rule or, in rare cases, included individually by an
+   exception.  When an implementation moves to a new version of Unicode,
+   the rules may indicate new valid code points.
+
+
+
+
+Klensin                       Informational                     [Page 9]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   This section provides an overview of the model used to establish the
+   algorithm and character lists of the Tables document [RFC5892] and
+   describes the names and applicability of the categories used there.
+   Note that the inclusion of a character in the PROTOCOL-VALID category
+   group (Section 3.1.1) does not imply that it can be used
+   indiscriminately; some characters are associated with contextual
+   rules that must be applied as well.
+
+   The information given in this section is provided to make the rules,
+   tables, and protocol easier to understand.  The normative generating
+   rules that correspond to this informal discussion appear in the
+   Tables document, and the rules that actually determine what labels
+   can be registered or looked up are in the Protocol document.
+
+3.1.  A Tiered Model of Permitted Characters and Labels
+
+   Moving to an inclusion model involves a new specification for the
+   list of characters that are permitted in IDNs.  In IDNA2003,
+   character validity is independent of context and fixed forever (or
+   until the standard is replaced).  However, globally context-
+   independent rules have proved to be impractical because some
+   characters, especially those that are called "Join_Controls" in
+   Unicode, are needed to make reasonable use of some scripts but have
+   no visible effect in others.  IDNA2003 prohibited those types of
+   characters entirely by discarding them.  We now have a consensus that
+   under some conditions, these "joiner" characters are legitimately
+   needed to allow useful mnemonics for some languages and scripts.  In
+   general, context-dependent rules help deal with characters (generally
+   characters that would otherwise be prohibited entirely) that are used
+   differently or perceived differently across different scripts, and
+   allow the standard to be applied more appropriately in cases where a
+   string is not universally handled the same way.
+
+   IDNA2008 divides all possible Unicode code points into four
+   categories: PROTOCOL-VALID, CONTEXTUAL RULE REQUIRED, DISALLOWED, and
+   UNASSIGNED.
+
+3.1.1.  PROTOCOL-VALID
+
+   Characters identified as PROTOCOL-VALID (often abbreviated PVALID)
+   are permitted in IDNs.  Their use may be restricted by rules about
+   the context in which they appear or by other rules that apply to the
+   entire label in which they are to be embedded.  For example, any
+   label that contains a character in this category that has a
+   "right-to-left" property must be used in context with the Bidi rules
+   [RFC5893].  The term PROTOCOL-VALID is used to stress the fact that
+   the presence of a character in this category does not imply that a
+   given registry need accept registrations containing any of the
+
+
+
+Klensin                       Informational                    [Page 10]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   characters in the category.  Registries are still expected to apply
+   judgment about labels they will accept and to maintain rules
+   consistent with those judgments (see the Protocol document [RFC5891]
+   and Section 3.3).
+
+   Characters that are placed in the PROTOCOL-VALID category are
+   expected to never be removed from it or reclassified.  While
+   theoretically characters could be removed from Unicode, such removal
+   would be inconsistent with the Unicode stability principles (see
+   UTR 39: Unicode Security Mechanisms [Unicode52], Appendix F) and
+   hence should never occur.
+
+3.1.2.  CONTEXTUAL RULE REQUIRED
+
+   Some characters may be unsuitable for general use in IDNs but
+   necessary for the plausible support of some scripts.  The two most
+   commonly cited examples are the ZERO WIDTH JOINER and ZERO WIDTH
+   NON-JOINER characters (ZWJ, U+200D and ZWNJ, U+200C), but other
+   characters may require special treatment because they would otherwise
+   be DISALLOWED (typically because Unicode considers them punctuation
+   or special symbols) but need to be permitted in limited contexts.
+   Other characters are given this special treatment because they pose
+   exceptional danger of being used to produce misleading labels or to
+   cause unacceptable ambiguity in label matching and interpretation.
+
+3.1.2.1.  Contextual Restrictions
+
+   Characters with contextual restrictions are identified as CONTEXTUAL
+   RULE REQUIRED and are associated with a rule.  The rule defines
+   whether the character is valid in a particular string, and also
+   whether the rule itself is to be applied on lookup as well as
+   registration.
+
+   A distinction is made between characters that indicate or prohibit
+   joining and ones similar to them (known as CONTEXT-JOINER or
+   CONTEXTJ) and other characters requiring contextual treatment
+   (CONTEXT-OTHER or CONTEXTO).  Only the former require full testing at
+   lookup time.
+
+   It is important to note that these contextual rules cannot prevent
+   all uses of the relevant characters that might be confusing or
+   problematic.  What they are expected to do is to confine
+   applicability of the characters to scripts (and narrower contexts)
+   where zone administrators are knowledgeable enough about the use of
+   those characters to be prepared to deal with them appropriately.
+
+
+
+
+
+
+Klensin                       Informational                    [Page 11]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   For example, a registry dealing with an Indic script that requires
+   ZWJ and/or ZWNJ as part of the writing system is expected to
+   understand where the characters have visible effect and where they do
+   not and to make registration rules accordingly.  By contrast, a
+   registry dealing primarily with Latin or Cyrillic script might not be
+   actively aware that the characters exist, much less about the
+   consequences of embedding them in labels drawn from those scripts and
+   therefore should avoid accepting registrations containing those
+   characters, at least in labels using characters from the Latin or
+   Cyrillic scripts.
+
+3.1.2.2.  Rules and Their Application
+
+   Rules have descriptions such as "Must follow a character from Script
+   XYZ", "Must occur only if the entire label is in Script ABC", or
+   "Must occur only if the previous and subsequent characters have the
+   DFG property".  The actual rules may be DEFINED or NULL.  If present,
+   they may have values of "True" (character may be used in any position
+   in any label), "False" (character may not be used in any label), or
+   may be a set of procedural rules that specify the context in which
+   the character is permitted.
+
+   Because it is easier to identify these characters than to know that
+   they are actually needed in IDNs or how to establish exactly the
+   right rules for each one, a rule may have a null value in a given
+   version of the tables.  Characters associated with null rules are not
+   permitted to appear in putative labels for either registration or
+   lookup.  Of course, a later version of the tables might contain a
+   non-null rule.
+
+   The actual rules and their descriptions are in Sections 2 and 3 of
+   the Tables document [RFC5892].  That document also specifies the
+   creation of a registry for future rules.
+
+3.1.3.  DISALLOWED
+
+   Some characters are inappropriate for use in IDNs and are thus
+   excluded for both registration and lookup (i.e., IDNA-conforming
+   applications performing name lookup should verify that these
+   characters are absent; if they are present, the label strings should
+   be rejected rather than converted to A-labels and looked up.  Some of
+   these characters are problematic for use in IDNs (such as the
+   FRACTION SLASH character, U+2044), while some of them (such as the
+   various HEART symbols, e.g., U+2665, U+2661, and U+2765, see
+   Section 7.6) simply fall outside the conventions for typical
+   identifiers (basically letters and numbers).
+
+
+
+
+
+Klensin                       Informational                    [Page 12]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   Of course, this category would include code points that had been
+   removed entirely from Unicode should such removals ever occur.
+
+   Characters that are placed in the DISALLOWED category are expected to
+   never be removed from it or reclassified.  If a character is
+   classified as DISALLOWED in error and the error is sufficiently
+   problematic, the only recourse would be either to introduce a new
+   code point into Unicode and classify it as PROTOCOL-VALID or for the
+   IETF to accept the considerable costs of an incompatible change and
+   replace the relevant RFC with one containing appropriate exceptions.
+
+   There is provision for exception cases but, in general, characters
+   are placed into DISALLOWED if they fall into one or more of the
+   following groups:
+
+   o  The character is a compatibility equivalent for another character.
+      In slightly more precise Unicode terms, application of
+      Normalization Form KC (NFKC) to the character yields some other
+      character.
+
+   o  The character is an uppercase form or some other form that is
+      mapped to another character by Unicode case folding.
+
+   o  The character is a symbol or punctuation form or, more generally,
+      something that is not a letter, digit, or a mark that is used to
+      form a letter or digit.
+
+3.1.4.  UNASSIGNED
+
+   For convenience in processing and table-building, code points that do
+   not have assigned values in a given version of Unicode are treated as
+   belonging to a special UNASSIGNED category.  Such code points are
+   prohibited in labels to be registered or looked up.  The category
+   differs from DISALLOWED in that code points are moved out of it by
+   the simple expedient of being assigned in a later version of Unicode
+   (at which point, they are classified into one of the other categories
+   as appropriate).
+
+   The rationale for restricting the processing of UNASSIGNED characters
+   is simply that the properties of such code points cannot be
+   completely known until actual characters are assigned to them.  For
+   example, assume that an UNASSIGNED code point were included in a
+   label to be looked up.  Assume that the code point was later assigned
+   to a character that required some set of contextual rules.  With that
+   combination, un-updated instances of IDNA-aware software might permit
+   lookup of labels containing the previously unassigned characters
+   while updated versions of the software might restrict use of the same
+
+
+
+
+Klensin                       Informational                    [Page 13]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   label in lookup, depending on the contextual rules.  It should be
+   clear that under no circumstance should an UNASSIGNED character be
+   permitted in a label to be registered as part of a domain name.
+
+3.2.  Registration Policy
+
+   While these recommendations cannot and should not define registry
+   policies, registries should develop and apply additional restrictions
+   as needed to reduce confusion and other problems.  For example, it is
+   generally believed that labels containing characters from more than
+   one script are a bad practice although there may be some important
+   exceptions to that principle.  Some registries may choose to restrict
+   registrations to characters drawn from a very small number of
+   scripts.  For many scripts, the use of variant techniques such as
+   those as described in the JET specification for the CJK script
+   [RFC3743] and its generalization [RFC4290], and illustrated for
+   Chinese by the tables provided by the Chinese Domain Name Consortium
+   [RFC4713] may be helpful in reducing problems that might be perceived
+   by users.
+
+   In general, users will benefit if registries only permit characters
+   from scripts that are well-understood by the registry or its
+   advisers.  If a registry decides to reduce opportunities for
+   confusion by constructing policies that disallow characters used in
+   historic writing systems or characters whose use is restricted to
+   specialized, highly technical contexts, some relevant information may
+   be found in Section 2.4 (Specific Character Adjustments) of Unicode
+   Identifier and Pattern Syntax [Unicode-UAX31], especially Table 4
+   (Candidate Characters for Exclusion from Identifiers), and Section
+   3.1 (General Security Profile for Identifiers) in Unicode Security
+   Mechanisms [Unicode-UTS39].
+
+   The requirement (in Section 4.1 of the Protocol document [RFC5891])
+   that registration procedures use only U-labels and/or A-labels is
+   intended to ensure that registrants are fully aware of exactly what
+   is being registered as well as encouraging use of those canonical
+   forms.  That provision should not be interpreted as requiring that
+   registrants need to provide characters in a particular code sequence.
+   Registrant input conventions and management are part of registrant-
+   registrar interactions and relationships between registries and
+   registrars and are outside the scope of these standards.
+
+   It is worth stressing that these principles of policy development and
+   application apply at all levels of the DNS, not only, e.g., top level
+   domain (TLD) or second level domain (SLD) registrations.  Even a
+   trivial, "anything is permitted that is valid under the protocol"
+   policy is helpful in that it helps users and application developers
+   know what to expect.
+
+
+
+Klensin                       Informational                    [Page 14]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+3.3.  Layered Restrictions: Tables, Context, Registration, and
+      Applications
+
+   The character rules in IDNA2008 are based on the realization that
+   there is no single magic bullet for any of the security,
+   confusability, or other issues associated with IDNs.  Instead, the
+   specifications define a variety of approaches.  The character tables
+   are the first mechanism, protocol rules about how those characters
+   are applied or restricted in context are the second, and those two in
+   combination constitute the limits of what can be done in the
+   protocol.  As discussed in the previous section (Section 3.2),
+   registries are expected to restrict what they permit to be
+   registered, devising and using rules that are designed to optimize
+   the balance between confusion and risk on the one hand and maximum
+   expressiveness in mnemonics on the other.
+
+   In addition, there is an important role for user interface programs
+   in warning against label forms that appear problematic given their
+   knowledge of local contexts and conventions.  Of course, no approach
+   based on naming or identifiers alone can protect against all threats.
+
+4.  Application-Related Issues
+
+4.1.  Display and Network Order
+
+   Domain names are always transmitted in network order (the order in
+   which the code points are sent in protocols), but they may have a
+   different display order (the order in which the code points are
+   displayed on a screen or paper).  When a domain name contains
+   characters that are normally written right to left, display order may
+   be affected although network order is not.  It gets even more
+   complicated if left-to-right and right-to-left labels are adjacent to
+   each other within a domain name.  The decision about the display
+   order is ultimately under the control of user agents -- including Web
+   browsers, mail clients, hosted Web applications and many more --
+   which may be highly localized.  Should a domain name abc.def, in
+   which both labels are represented in scripts that are written right
+   to left, be displayed as fed.cba or cba.fed?  Applications that are
+   in deployment today are already diverse, and one can find examples of
+   either choice.
+
+   The picture changes once again when an IDN appears in an
+   Internationalized Resource Identifier (IRI) [RFC3987].  An IRI or
+   internationalized email address contains elements other than the
+   domain name.  For example, IRIs contain protocol identifiers and
+   field delimiter syntax such as "http://" or "mailto:" while email
+   addresses contain the "@" to separate local parts from domain names.
+
+
+
+
+Klensin                       Informational                    [Page 15]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   An IRI in network order begins with "http://" followed by domain
+   labels in network order, thus "http://abc.def".
+
+   User interface programs are not required to display and allow input
+   of IRIs directly but often do so.  Implementers have to choose
+   whether the overall direction of these strings will always be left to
+   right (or right to left) for an IRI or email address.  The natural
+   order for a user typing a domain name on a right-to-left system is
+   fed.cba.  Should the right-to-left (RTL) user interface reverse the
+   entire domain name each time a domain name is typed?  Does this
+   change if the user types "http://" right before typing a domain name,
+   thus implying that the user is beginning at the beginning of the
+   network-order IRI?  Experience in the 1980s and 1990s with mixing
+   systems in which domain name labels were read in network order (left
+   to right) and those in which those labels were read right to left
+   would predict a great deal of confusion.
+
+   If each implementation of each application makes its own decisions on
+   these issues, users will develop heuristics that will sometimes fail
+   when switching applications.  However, while some display order
+   conventions, voluntarily adopted, would be desirable to reduce
+   confusion, such suggestions are beyond the scope of these
+   specifications.
+
+4.2.  Entry and Display in Applications
+
+   Applications can accept and display domain names using any character
+   set or character coding system.  The IDNA protocol does not
+   necessarily affect the interface between users and applications.  An
+   IDNA-aware application can accept and display internationalized
+   domain names in two formats: as the internationalized character
+   set(s) supported by the application (i.e., an appropriate local
+   representation of a U-label) and as an A-label.  Applications may
+   allow the display of A-labels, but are encouraged not to do so except
+   as an interface for special purposes, possibly for debugging, or to
+   cope with display limitations.  In general, they should allow, but
+   not encourage, user input of A-labels.  A-labels are opaque and ugly,
+   and malicious variations on them are not easily detected by users.
+   Where possible, they should thus only be exposed when they are
+   absolutely needed.  Because IDN labels can be rendered either as
+   A-labels or U-labels, the application may reasonably have an option
+   for the user to select the preferred method of display.  Rendering
+   the U-label should normally be the default.
+
+   Domain names are often stored and transported in many places.  For
+   example, they are part of documents such as mail messages and web
+   pages.  They are transported in many parts of many protocols, such as
+   both the control commands of SMTP and associated message body parts,
+
+
+
+Klensin                       Informational                    [Page 16]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   and in the headers and the body content in HTTP.  It is important to
+   remember that domain names appear both in domain name slots and in
+   the content that is passed over protocols, and it would be helpful if
+   protocols explicitly define what their domain name slots are.
+
+   In protocols and document formats that define how to handle
+   specification or negotiation of charsets, labels can be encoded in
+   any charset allowed by the protocol or document format.  If a
+   protocol or document format only allows one charset, the labels must
+   be given in that charset.  Of course, not all charsets can properly
+   represent all labels.  If a U-label cannot be displayed in its
+   entirety, the only choice (without loss of information) may be to
+   display the A-label.
+
+   Where a protocol or document format allows IDNs, labels should be in
+   whatever character encoding and escape mechanism the protocol or
+   document format uses in the local environment.  This provision is
+   intended to prevent situations in which, e.g., UTF-8 domain names
+   appear embedded in text that is otherwise in some other character
+   coding.
+
+   All protocols that use domain name slots (see Section 2.3.2.6 in the
+   Definitions document [RFC5890]) already have the capacity for
+   handling domain names in the ASCII charset.  Thus, A-labels can
+   inherently be handled by those protocols.
+
+   IDNA2008 does not specify required mappings between one character or
+   code point and others.  An extended discussion of mapping issues
+   appears in Section 6 and specific recommendations appear in the
+   Mapping document [IDNA2008-Mapping].  In general, IDNA2008 prohibits
+   characters that would be mapped to others by normalization or other
+   rules.  As examples, while mathematical characters based on Latin
+   ones are accepted as input to IDNA2003, they are prohibited in
+   IDNA2008.  Similarly, uppercase characters, double-width characters,
+   and other variations are prohibited as IDNA input although mapping
+   them as needed in user interfaces is strongly encouraged.
+
+   Since the rules in the Tables document [RFC5892] have the effect that
+   only strings that are not transformed by NFKC are valid, if an
+   application chooses to perform NFKC normalization before lookup, that
+   operation is safe since this will never make the application unable
+   to look up any valid string.  However, as discussed above, the
+   application cannot guarantee that any other application will perform
+   that mapping, so it should be used only with caution and for informed
+   users.
+
+
+
+
+
+
+Klensin                       Informational                    [Page 17]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   In many cases, these prohibitions should have no effect on what the
+   user can type as input to the lookup process.  It is perfectly
+   reasonable for systems that support user interfaces to perform some
+   character mapping that is appropriate to the local environment.  This
+   would normally be done prior to actual invocation of IDNA.  At least
+   conceptually, the mapping would be part of the Unicode conversions
+   discussed above and in the Protocol document [RFC5891].  However,
+   those changes will be local ones only -- local to environments in
+   which users will clearly understand that the character forms are
+   equivalent.  For use in interchanges among systems, it appears to be
+   much more important that U-labels and A-labels can be mapped back and
+   forth without loss of information.
+
+   One specific, and very important, instance of this strategy arises
+   with case folding.  In the ASCII-only DNS, names are looked up and
+   matched in a case-independent way, but no actual case folding occurs.
+   Names can be placed in the DNS in either uppercase or lowercase form
+   (or any mixture of them) and that form is preserved, returned in
+   queries, and so on.  IDNA2003 approximated that behavior for
+   non-ASCII strings by performing case folding at registration time
+   (resulting in only lowercase IDNs in the DNS) and when names were
+   looked up.
+
+   As suggested earlier in this section, it appears to be desirable to
+   do as little character mapping as possible as long as Unicode works
+   correctly (e.g., Normalization Form C (NFC) mapping to resolve
+   different codings for the same character is still necessary although
+   the specifications require that it be performed prior to invoking the
+   protocol) in order to make the mapping between A-labels and U-labels
+   idempotent.  Case mapping is not an exception to this principle.  If
+   only lowercase characters can be registered in the DNS (i.e., be
+   present in a U-label), then IDNA2008 should prohibit uppercase
+   characters as input even though user interfaces to applications
+   should probably map those characters.  Some other considerations
+   reinforce this conclusion.  For example, in ASCII case mapping for
+   individual characters, uppercase(character) is always equal to
+   uppercase(lowercase(character)).  That may not be true with IDNs.  In
+   some scripts that use case distinctions, there are a few characters
+   that do not have counterparts in one case or the other.  The
+   relationship between uppercase and lowercase may even be language-
+   dependent, with different languages (or even the same language in
+   different areas) expecting different mappings.  User interface
+   programs can meet the expectations of users who are accustomed to the
+   case-insensitive DNS environment by performing case folding prior to
+   IDNA processing, but the IDNA procedures themselves should neither
+   require such mapping nor expect them when they are not natural to the
+   localized environment.
+
+
+
+
+Klensin                       Informational                    [Page 18]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+4.3.  Linguistic Expectations: Ligatures, Digraphs, and Alternate
+      Character Forms
+
+   Users have expectations about character matching or equivalence that
+   are based on their own languages and the orthography of those
+   languages.  These expectations may not always be met in a global
+   system, especially if multiple languages are written using the same
+   script but using different conventions.  Some examples:
+
+   o  A Norwegian user might expect a label with the ae-ligature to be
+      treated as the same label as one using the Swedish spelling with
+      a-diaeresis even though applying that mapping to English would be
+      astonishing to users.
+
+   o  A German user might expect a label with an o-umlaut and a label
+      that had "oe" substituted, but was otherwise the same, to be
+      treated as equivalent even though that substitution would be a
+      clear error in Swedish.
+
+   o  A Chinese user might expect automatic matching of Simplified and
+      Traditional Chinese characters, but applying that matching for
+      Korean or Japanese text would create considerable confusion.
+
+   o  An English user might expect "theater" and "theatre" to match.
+
+   A number of languages use alphabetic scripts in which single phonemes
+   are written using two characters, termed a "digraph", for example,
+   the "ph" in "pharmacy" and "telephone".  (Such characters can also
+   appear consecutively without forming a digraph, as in "tophat".)
+   Certain digraphs may be indicated typographically by setting the two
+   characters closer together than they would be if used consecutively
+   to represent different phonemes.  Some digraphs are fully joined as
+   ligatures.  For example, the word "encyclopaedia" is sometimes set
+   with a U+00E6 LATIN SMALL LIGATURE AE.  When ligature and digraph
+   forms have the same interpretation across all languages that use a
+   given script, application of Unicode normalization generally resolves
+   the differences and causes them to match.  When they have different
+   interpretations, matching must utilize other methods, presumably
+   chosen at the registry level, or users must be educated to understand
+   that matching will not occur.
+
+   The nature of the problem can be illustrated by many words in the
+   Norwegian language, where the "ae" ligature is the 27th letter of a
+   29-letter extended Latin alphabet.  It is equivalent to the 28th
+   letter of the Swedish alphabet (also containing 29 letters),
+   U+00E4 LATIN SMALL LETTER A WITH DIAERESIS, for which an "ae" cannot
+   be substituted according to current orthographic standards.  That
+   character (U+00E4) is also part of the German alphabet where, unlike
+
+
+
+Klensin                       Informational                    [Page 19]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   in the Nordic languages, the two-character sequence "ae" is usually
+   treated as a fully acceptable alternate orthography for the "umlauted
+   a" character.  The inverse is however not true, and those two
+   characters cannot necessarily be combined into an "umlauted a".  This
+   also applies to another German character, the "umlauted o"
+   (U+00F6 LATIN SMALL LETTER O WITH DIAERESIS) which, for example,
+   cannot be used for writing the name of the author "Goethe".  It is
+   also a letter in the Swedish alphabet where, like the "a with
+   diaeresis", it cannot be correctly represented as "oe" and in the
+   Norwegian alphabet, where it is represented, not as "o with
+   diaeresis", but as "slashed o", U+00F8.
+
+   Some of the ligatures that have explicit code points in Unicode were
+   given special handling in IDNA2003 and now pose additional problems
+   in transition.  See Section 7.2.
+
+   Additional cases with alphabets written right to left are described
+   in Section 4.5.
+
+   Matching and comparison algorithm selection often requires
+   information about the language being used, context, or both --
+   information that is not available to IDNA or the DNS.  Consequently,
+   IDNA2008 makes no attempt to treat combined characters in any special
+   way.  A registry that is aware of the language context in which
+   labels are to be registered, and where that language sometimes (or
+   always) treats the two-character sequences as equivalent to the
+   combined form, should give serious consideration to applying a
+   "variant" model [RFC3743][RFC4290] or to prohibiting registration of
+   one of the forms entirely, to reduce the opportunities for user
+   confusion and fraud that would result from the related strings being
+   registered to different parties.
+
+4.4.  Case Mapping and Related Issues
+
+   In the DNS, ASCII letters are stored with their case preserved.
+   Matching during the query process is case-independent, but none of
+   the information that might be represented by choices of case has been
+   lost.  That model has been accidentally helpful because, as people
+   have created DNS labels by catenating words (or parts of words) to
+   form labels, case has often been used to distinguish among components
+   and make the labels more memorable.
+
+   Since DNS servers do not get involved in parsing IDNs, they cannot do
+   case-independent matching.  Thus, keeping the cases separate in
+   lookup or registration, and doing matching at the server, is not
+   feasible with IDNA or any similar approach.  Matching of characters
+   that are considered to differ only by case must be done, if desired,
+   by programs invoking IDNA lookup even though it wasn't done by ASCII-
+
+
+
+Klensin                       Informational                    [Page 20]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   only DNS clients.  That situation was recognized in IDNA2003 and
+   nothing in IDNA2008 fundamentally changes it or could do so.  In
+   IDNA2003, all characters are case folded and mapped by clients in a
+   standardized step.
+
+   Even in scripts that generally support case distinctions, some
+   characters do not have uppercase forms.  For example, the Unicode
+   case-folding operation maps Greek Final Form Sigma (U+03C2) to the
+   medial form (U+03C3) and maps Eszett (German Sharp S, U+00DF) to
+   "ss".  Neither of these mappings is reversible because the uppercase
+   of U+03C3 is the uppercase Sigma (U+03A3) and "ss" is an ASCII
+   string.  IDNA2008 permits, at the risk of some incompatibility,
+   slightly more flexibility in this area by avoiding case folding and
+   treating these characters as themselves.  Approaches to handling one-
+   way mappings are discussed in Section 7.2.
+
+   Because IDNA2003 maps Final Sigma and Eszett to other characters, and
+   the reverse mapping is never possible, neither Final Sigma nor Eszett
+   can be represented in the ACE form of IDNA2003 IDN nor in the native
+   character (U-label) form derived from it.  With IDNA2008, both
+   characters can be used in an IDN and so the A-label used for lookup
+   for any U-label containing those characters is now different.  See
+   Section 7.1 for a discussion of what kinds of changes might require
+   the IDNA prefix to change; after extended discussions, the IDNABIS
+   Working Group came to consensus that the change for these characters
+   did not justify a prefix change.
+
+4.5.  Right-to-Left Text
+
+   In order to be sure that the directionality of right-to-left text is
+   unambiguous, IDNA2003 required that any label in which right-to-left
+   characters appear both starts and ends with them and that it does not
+   include any characters with strong left-to-right properties (that
+   excludes other alphabetic characters but permits European digits).
+   Any other string that contains a right-to-left character and does not
+   meet those requirements is rejected.  This is one of the few places
+   where the IDNA algorithms (both in IDNA2003 and in IDNA2008) examine
+   an entire label, not just individual characters.  The algorithmic
+   model used in IDNA2003 rejects the label when the final character in
+   a right-to-left string requires a combining mark in order to be
+   correctly represented.
+
+   That prohibition is not acceptable for writing systems for languages
+   written with consonantal alphabets to which diacritical vocalic
+   systems are applied, and for languages with orthographies derived
+   from them where the combining marks may have different functionality.
+   In both cases, the combining marks can be essential components of the
+   orthography.  Examples of this are Yiddish, written with an extended
+
+
+
+Klensin                       Informational                    [Page 21]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   Hebrew script, and Dhivehi (the official language of Maldives), which
+   is written in the Thaana script (which is, in turn, derived from the
+   Arabic script).  IDNA2008 removes the restriction on final combining
+   characters with a new set of rules for right-to-left scripts and
+   their characters.  Those new rules are specified in the Bidi document
+   [RFC5893].
+
+5.  IDNs and the Robustness Principle
+
+   The "Robustness Principle" is often stated as "Be conservative about
+   what you send and liberal in what you accept" (see, e.g., Section
+   1.2.2 of the applications-layer Host Requirements specification
+   [RFC1123]).  This principle applies to IDNA.  In applying the
+   principle to registries as the source ("sender") of all registered
+   and useful IDNs, registries are responsible for being conservative
+   about what they register and put out in the Internet.  For IDNs to
+   work well, zone administrators (registries) must have and require
+   sensible policies about what is registered -- conservative policies
+   -- and implement and enforce them.
+
+   Conversely, lookup applications are expected to reject labels that
+   clearly violate global (protocol) rules (no one has ever seriously
+   claimed that being liberal in what is accepted requires being
+   stupid).  However, once one gets past such global rules and deals
+   with anything sensitive to script or locale, it is necessary to
+   assume that garbage has not been placed into the DNS, i.e., one must
+   be liberal about what one is willing to look up in the DNS rather
+   than guessing about whether it should have been permitted to be
+   registered.
+
+   If a string cannot be successfully found in the DNS after the lookup
+   processing described here, it makes no difference whether it simply
+   wasn't registered or was prohibited by some rule at the registry.
+   Application implementers should be aware that where DNS wildcards are
+   used, the ability to successfully resolve a name does not guarantee
+   that it was actually registered.
+
+6.  Front-end and User Interface Processing for Lookup
+
+   Domain names may be identified and processed in many contexts.  They
+   may be typed in by users themselves or embedded in an identifier such
+   as an email address, URI, or IRI.  They may occur in running text or
+   be processed by one system after being provided in another.  Systems
+   may try to normalize URLs to determine (or guess) whether a reference
+   is valid or if two references point to the same object without
+   actually looking the objects up (comparison without lookup is
+   necessary for URI types that are not intended to be resolved).  Some
+   of these goals may be more easily and reliably satisfied than others.
+
+
+
+Klensin                       Informational                    [Page 22]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   While there are strong arguments for any domain name that is placed
+   "on the wire" -- transmitted between systems -- to be in the zero-
+   ambiguity forms of A-labels, it is inevitable that programs that
+   process domain names will encounter U-labels or variant forms.
+
+   An application that implements the IDNA protocol [RFC5891] will
+   always take any user input and convert it to a set of Unicode code
+   points.  That user input may be acquired by any of several different
+   input methods, all with differing conversion processes to be taken
+   into consideration (e.g., typed on a keyboard, written by hand onto
+   some sort of digitizer, spoken into a microphone and interpreted by a
+   speech-to-text engine, etc.).  The process of taking any particular
+   user input and mapping it into a Unicode code point may be a simple
+   one: if a user strikes the "A" key on a US English keyboard, without
+   any modifiers such as the "Shift" key held down, in order to draw a
+   Latin small letter A ("a"), many (perhaps most) modern operating
+   system input methods will produce to the calling application the code
+   point U+0061, encoded in a single octet.
+
+   Sometimes the process is somewhat more complicated: a user might
+   strike a particular set of keys to represent a combining macron
+   followed by striking the "A" key in order to draw a Latin small
+   letter A with a macron above it.  Depending on the operating system,
+   the input method chosen by the user, and even the parameters with
+   which the application communicates with the input method, the result
+   might be the code point U+0101 (encoded as two octets in UTF-8 or
+   UTF-16, four octets in UTF-32, etc.), the code point U+0061 followed
+   by the code point U+0304 (again, encoded in three or more octets,
+   depending upon the encoding used) or even the code point U+FF41
+   followed by the code point U+0304 (and encoded in some form).  These
+   examples leave aside the issue of operating systems and input methods
+   that do not use Unicode code points for their character set.
+
+   In every case, applications (with the help of the operating systems
+   on which they run and the input methods used) need to perform a
+   mapping from user input into Unicode code points.
+
+   IDNA2003 used a model whereby input was taken from the user, mapped
+   (via whatever input method mechanisms were used) to a set of Unicode
+   code points, and then further mapped to a set of Unicode code points
+   using the Nameprep profile [RFC3491].  In this procedure, there are
+   two separate mapping steps: first, a mapping done by the input method
+   (which might be controlled by the operating system, the application,
+   or some combination) and then a second mapping performed by the
+   Nameprep portion of the IDNA protocol.  The mapping done in Nameprep
+   includes a particular mapping table to re-map some characters to
+   other characters, a particular normalization, and a set of prohibited
+   characters.
+
+
+
+Klensin                       Informational                    [Page 23]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   Note that the result of the two-step mapping process means that the
+   mapping chosen by the operating system or application in the first
+   step might differ significantly from the mapping supplied by the
+   Nameprep profile in the second step.  This has advantages and
+   disadvantages.  Of course, the second mapping regularizes what gets
+   looked up in the DNS, making for better interoperability between
+   implementations that use the Nameprep mapping.  However, the
+   application or operating system may choose mappings in their input
+   methods, which when passed through the second (Nameprep) mapping
+   result in characters that are "surprising" to the end user.
+
+   The other important feature of IDNA2003 is that, with very few
+   exceptions, it assumes that any set of Unicode code points provided
+   to the Nameprep mapping can be mapped into a string of Unicode code
+   points that are "sensible", even if that means mapping some code
+   points to nothing (that is, removing the code points from the
+   string).  This allowed maximum flexibility in input strings.
+
+   The present version of IDNA (IDNA2008) differs significantly in
+   approach from the original version.  First and foremost, it does not
+   provide explicit mapping instructions.  Instead, it assumes that the
+   application (perhaps via an operating system input method) will do
+   whatever mapping it requires to convert input into Unicode code
+   points.  This has the advantage of giving flexibility to the
+   application to choose a mapping that is suitable for its user given
+   specific user requirements, and avoids the two-step mapping of the
+   original protocol.  Instead of a mapping, IDNA2008 provides a set of
+   categories that can be used to specify the valid code points allowed
+   in a domain name.
+
+   In principle, an application ought to take user input of a domain
+   name and convert it to the set of Unicode code points that represent
+   the domain name the user intends.  As a practical matter, of course,
+   determining user intent is a tricky business, so an application needs
+   to choose a reasonable mapping from user input.  That may differ
+   based on the particular circumstances of a user, depending on locale,
+   language, type of input method, etc.  It is up to the application to
+   make a reasonable choice.
+
+
+
+
+
+
+
+
+
+
+
+
+
+Klensin                       Informational                    [Page 24]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+7.  Migration from IDNA2003 and Unicode Version Synchronization
+
+7.1.  Design Criteria
+
+   As mentioned above and in the IAB review and recommendations for IDNs
+   [RFC4690], two key goals of the IDNA2008 design are:
+
+   o  to enable applications to be agnostic about whether they are being
+      run in environments supporting any Unicode version from 3.2
+      onward.
+
+   o  to permit incrementally adding new characters, character groups,
+      scripts, and other character collections as they are incorporated
+      into Unicode, doing so without disruption and, in the long term,
+      without "heavy" processes (an IETF consensus process is required
+      by the IDNA2008 specifications and is expected to be required and
+      used until significant experience accumulates with IDNA operations
+      and new versions of Unicode).
+
+7.1.1.  Summary and Discussion of IDNA Validity Criteria
+
+   The general criteria for a label to be considered valid under IDNA
+   are (the actual rules are rigorously defined in the Protocol
+   [RFC5891] and Tables [RFC5892] documents):
+
+   o  The characters are "letters", marks needed to form letters,
+      numerals, or other code points used to write words in some
+      language.  Symbols, drawing characters, and various notational
+      characters are intended to be permanently excluded.  There is no
+      evidence that they are important enough to Internet operations or
+      internationalization to justify expansion of domain names beyond
+      the general principle of "letters, digits, and hyphen".
+      (Additional discussion and rationale for the symbol decision
+      appears in Section 7.6.)
+
+   o  Other than in very exceptional cases, e.g., where they are needed
+      to write substantially any word of a given language, punctuation
+      characters are excluded.  The fact that a word exists is not proof
+      that it should be usable in a DNS label, and DNS labels are not
+      expected to be usable for multiple-word phrases (although they are
+      certainly not prohibited if the conventions and orthography of a
+      particular language cause that to be possible).
+
+   o  Characters that are unassigned (have no character assignment at
+      all) in the version of Unicode being used by the registry or
+      application are not permitted, even on lookup.  The issues
+      involved in this decision are discussed in Section 7.7.
+
+
+
+
+Klensin                       Informational                    [Page 25]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   o  Any character that is mapped to another character by a current
+      version of NFKC is prohibited as input to IDNA (for either
+      registration or lookup).  With a few exceptions, this principle
+      excludes any character mapped to another by Nameprep [RFC3491].
+
+   The principles above drive the design of rules that are specified
+   exactly in the Tables document.  Those rules identify the characters
+   that are valid under IDNA.  The rules themselves are normative, and
+   the tables are derived from them, rather than vice versa.
+
+7.1.2.  Labels in Registration
+
+   Any label registered in a DNS zone must be validated -- i.e., the
+   criteria for that label must be met -- in order for applications to
+   work as intended.  This principle is not new.  For example, since the
+   DNS was first deployed, zone administrators have been expected to
+   verify that names meet "hostname" requirements [RFC0952] where those
+   requirements are imposed by the expected applications.  Other
+   applications contexts, such as the later addition of special service
+   location formats [RFC2782] imposed new requirements on zone
+   administrators.  For zones that will contain IDNs, support for
+   Unicode version-independence requires restrictions on all strings
+   placed in the zone.  In particular, for such zones (the exact rules
+   appear in Section 4 of the Protocol document [RFC5891]):
+
+   o  Any label that appears to be an A-label, i.e., any label that
+      starts in "xn--", must be valid under IDNA, i.e., they must be
+      valid A-labels, as discussed in Section 2 above.
+
+   o  The Unicode tables (i.e., tables of code points, character
+      classes, and properties) and IDNA tables (i.e., tables of
+      contextual rules such as those that appear in the Tables
+      document), must be consistent on the systems performing or
+      validating labels to be registered.  Note that this does not
+      require that tables reflect the latest version of Unicode, only
+      that all tables used on a given system are consistent with each
+      other.
+
+   Under this model, registry tables will need to be updated (both the
+   Unicode-associated tables and the tables of permitted IDN characters)
+   to enable a new script or other set of new characters.  The registry
+   will not be affected by newer versions of Unicode, or newly
+   authorized characters, until and unless it wishes to support them.
+   The zone administrator is responsible for verifying validity for IDNA
+   as well as its local policies -- a more extensive set of checks than
+   are required for looking up the labels.  Systems looking up or
+
+
+
+
+
+Klensin                       Informational                    [Page 26]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   resolving DNS labels, especially IDN DNS labels, must be able to
+   assume that applicable registration rules were followed for names
+   entered into the DNS.
+
+7.1.3.  Labels in Lookup
+
+   Any application processing a label through IDNA so it can be looked
+   up in a DNS zone is required to (the exact rules appear in Section 5
+   of the Protocol document [RFC5891]):
+
+   o  Maintain IDNA and Unicode tables that are consistent with regard
+      to versions, i.e., unless the application actually executes the
+      classification rules in the Tables document [RFC5892], its IDNA
+      tables must be derived from the version of Unicode that is
+      supported more generally on the system.  As with registration, the
+      tables need not reflect the latest version of Unicode, but they
+      must be consistent.
+
+   o  Validate the characters in labels to be looked up only to the
+      extent of determining that the U-label does not contain
+      "DISALLOWED" code points or code points that are unassigned in its
+      version of Unicode.
+
+   o  Validate the label itself for conformance with a small number of
+      whole-label rules.  In particular, it must verify that:
+
+      *  there are no leading combining marks,
+
+      *  the Bidi conditions are met if right-to-left characters appear,
+
+      *  any required contextual rules are available, and
+
+      *  any contextual rules that are associated with joiner characters
+         (and CONTEXTJ characters more generally) are tested.
+
+   o  Do not reject labels based on other contextual rules about
+      characters, including mixed-script label prohibitions.  Such rules
+      may be used to influence presentation decisions in the user
+      interface, but not to avoid looking up domain names.
+
+   To further clarify the rules about handling characters that require
+   contextual rules, note that one can have a context-required character
+   (i.e., one that requires a rule), but no rule.  In that case, the
+   character is treated the same way DISALLOWED characters are treated,
+   until and unless a rule is supplied.  That state is more or less
+   equivalent to "the idea of permitting this character is accepted in
+   principle, but it won't be permitted in practice until consensus is
+   reached on a safe way to use it".
+
+
+
+Klensin                       Informational                    [Page 27]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   The ability to add a rule more or less exempts these characters from
+   the prohibition against reclassifying characters from DISALLOWED to
+   PVALID.
+
+   And, obviously, "no rule" is different from "have a rule, but the
+   test either succeeds or fails".
+
+   Lookup applications that follow these rules, rather than having their
+   own criteria for rejecting lookup attempts, are not sensitive to
+   version incompatibilities with the particular zone registry
+   associated with the domain name except for labels containing
+   characters recently added to Unicode.
+
+   An application or client that processes names according to this
+   protocol and then resolves them in the DNS will be able to locate any
+   name that is registered, as long as those registrations are valid
+   under IDNA and its version of the IDNA tables is sufficiently up to
+   date to interpret all of the characters in the label.  Messages to
+   users should distinguish between "label contains an unallocated code
+   point" and other types of lookup failures.  A failure on the basis of
+   an old version of Unicode may lead the user to a desire to upgrade to
+   a newer version, but will have no other ill effects (this is
+   consistent with behavior in the transition to the DNS when some hosts
+   could not yet handle some forms of names or record types).
+
+7.2.  Changes in Character Interpretations
+
+   As a consequence of the elimination of mapping, the current version
+   of IDNA changes the interpretation of a few characters relative to
+   its predecessors.  This subsection outlines the issues and discusses
+   possible transition strategies.
+
+7.2.1.  Character Changes: Eszett and Final Sigma
+
+   In those scripts that make case distinctions, there are a few
+   characters for which an obvious and unique uppercase character has
+   not historically been available to match a lowercase one, or vice
+   versa.  For those characters, the mappings used in constructing the
+   Stringprep tables for IDNA2003, performed using the Unicode
+   toCaseFold operation (see Section 5.18 of the Unicode Standard
+   [Unicode52]), generate different characters or sets of characters.
+   Those operations are not reversible and lose even more information
+   than traditional uppercase or lowercase transformations, but are more
+   useful than those transformations for comparison purposes.  Two
+   notable characters of this type are the German character Eszett
+   (Sharp S, U+00DF) and the Greek Final Form Sigma (U+03C2).  The
+   former is case folded to the ASCII string "ss", the latter to a
+   medial (lowercase) Sigma (U+03C3).
+
+
+
+Klensin                       Informational                    [Page 28]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+7.2.2.  Character Changes: Zero Width Joiner and Zero Width Non-Joiner
+
+   IDNA2003 mapped both ZERO WIDTH JOINER (ZWJ, U+200D) and ZERO WIDTH
+   NON-JOINER (ZWNJ, U+200C) to nothing, effectively dropping these
+   characters from any label in which they appeared and treating strings
+   containing them as identical to strings that did not.  As discussed
+   in Section 3.1.2 above, those characters are essential for writing
+   many reasonable mnemonics for certain scripts.  However, treating
+   them as valid in IDNA2008, even with contextual restrictions, raises
+   approximately the same problem as exists with Eszett and Final Sigma:
+   strings that were valid under IDNA2003 have different interpretations
+   as labels, and different A-labels, than the same strings under this
+   newer version.
+
+7.2.3.  Character Changes and the Need for Transition
+
+   The decision to eliminate mandatory and standardized mappings,
+   including case folding, from the IDNA2008 protocol in order to make
+   A-labels and U-labels idempotent made these characters problematic.
+   If they were to be disallowed, important words and mnemonics could
+   not be written in orthographically reasonable ways.  If they were to
+   be permitted as distinct characters, there would be no information
+   loss and registries would have more flexibility, but IDNA2003 and
+   IDNA2008 lookups might result in different A-labels.
+
+   With the understanding that there would be incompatibility either way
+   but a judgment that the incompatibility was not significant enough to
+   justify a prefix change, the Working Group concluded that Eszett and
+   Final Form Sigma should be treated as distinct and Protocol-Valid
+   characters.
+
+   Since these characters are interpreted in different ways under the
+   older and newer versions of IDNA, transition strategies and policies
+   will be necessary.  Some actions can reasonably be taken by
+   applications' client programs (those that perform lookup operations
+   or cause them to be performed), but because of the diversity of
+   situations and uses of the DNS, much of the responsibility will need
+   to fall on registries.
+
+   Registries, especially those maintaining zones for third parties,
+   must decide how to introduce a new service in a way that does not
+   create confusion or significantly weaken or invalidate existing
+   identifiers.  This is not a new problem; registries were faced with
+   similar issues when IDNs were introduced (potentially, and especially
+   for Latin-based scripts, in conflict with existing labels that had
+   been rendered in ASCII characters by applying more or less
+   standardized conventions) and when other new forms of strings have
+   been permitted as labels.
+
+
+
+Klensin                       Informational                    [Page 29]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+7.2.4.  Transition Strategies
+
+   There are several approaches to the introduction of new characters or
+   changes in interpretation of existing characters from their mapped
+   forms in the earlier version of IDNA.  The transition issue is
+   complicated because the forms of these labels after the
+   ToUnicode(ToASCII()) translation in IDNA2003 not only remain valid
+   but do not provide strong indications of what the registrant
+   intended: a string containing "ss" could have simply been intended to
+   be that string or could have been intended to contain an Eszett; a
+   string containing lowercase Sigma could have been intended to contain
+   Final Sigma (one might make heuristic guesses based on position in a
+   string, but the long tradition of forming labels by concatenating
+   words makes such heuristics unreliable), and strings that do not
+   contain ZWJ or ZWNJ might have been intended to contain them.
+   Without any preference or claim to completeness, some of these, all
+   of which have been used by registries in the past for similar
+   transitions, are:
+
+   1.  Do not permit use of the newly available character at the
+       registry level.  This might cause lookup failures if a domain
+       name were to be written with the expectation of the IDNA2003
+       mapping behavior, but would eliminate any possibility of false
+       matches.
+
+   2.  Hold a "sunrise"-like arrangement in which holders of labels
+       containing "ss" in the Eszett case, lowercase Sigma in that case,
+       or that might have contained ZWJ or ZWNJ in context, are given
+       priority (and perhaps other benefits) for registering the
+       corresponding string containing Eszett, Final Sigma, or the
+       appropriate zero-width character respectively.
+
+   3.  Adopt some sort of "variant" approach in which registrants obtain
+       labels with both character forms.
+
+   4.  Adopt a different form of "variant" approach in which
+       registration of additional strings that would produce the same
+       A-label if interpreted according to IDNA2003 is either not
+       permitted at all or permitted only by the registrant who already
+       has one of the names.
+
+   5.  Ignore the issue and assume that the marketplace or other
+       mechanisms will sort things out.
+
+   In any event, a registry (at any level of the DNS tree) that chooses
+   to permit labels to be registered that contains these characters, or
+   considers doing so, will have to address the relationship with
+   existing, possibly conflicting, labels in some way, just as
+
+
+
+Klensin                       Informational                    [Page 30]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   registries that already had a considerable number of labels did when
+   IDNs were first introduced.
+
+7.3.  Elimination of Character Mapping
+
+   As discussed at length in Section 6, IDNA2003, via Nameprep (see
+   Section 7.5), mapped many characters into related ones.  Those
+   mappings no longer exist as requirements in IDNA2008.  These
+   specifications strongly prefer that only A-labels or U-labels be used
+   in protocol contexts and as much as practical more generally.
+   IDNA2008 does anticipate situations in which some mapping at the time
+   of user input into lookup applications is appropriate and desirable.
+   The issues are discussed in Section 6 and specific recommendations
+   are made in the Mapping document [IDNA2008-Mapping].
+
+7.4.  The Question of Prefix Changes
+
+   The conditions that would have required a change in the IDNA ACE
+   prefix ("xn--", used in IDNA2003) were of great concern to the
+   community.  A prefix change would have clearly been necessary if the
+   algorithms were modified in a manner that would have created serious
+   ambiguities during subsequent transition in registrations.  This
+   section summarizes the working group's conclusions about the
+   conditions under which a change in the prefix would have been
+   necessary and the implications of such a change.
+
+7.4.1.  Conditions Requiring a Prefix Change
+
+   An IDN prefix change would have been needed if a given string would
+   be looked up or otherwise interpreted differently depending on the
+   version of the protocol or tables being used.  This IDNA upgrade
+   would have required a prefix change if, and only if, one of the
+   following four conditions were met:
+
+   1.  The conversion of an A-label to Unicode (i.e., a U-label) would
+       have yielded one string under IDNA2003 and a different string
+       under IDNA2008.
+
+   2.  In a significant number of cases, an input string that was valid
+       under IDNA2003 and also valid under IDNA2008 would have yielded
+       two different A-labels with the different versions.  This
+       condition is believed to be essentially equivalent to the one
+       above except for a very small number of edge cases that were not
+       found to justify a prefix change (see Section 7.2).
+
+       Note that if the input string was valid under one version and not
+       valid under the other, this condition would not apply.  See the
+       first item in Section 7.4.2, below.
+
+
+
+Klensin                       Informational                    [Page 31]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   3.  A fundamental change was made to the semantics of the string that
+       would be inserted in the DNS, e.g., if a decision were made to
+       try to include language or script information in the encoding in
+       addition to the string itself.
+
+   4.  A sufficiently large number of characters were added to Unicode
+       so that the Punycode mechanism for block offsets would no longer
+       reference the higher-numbered planes and blocks.  This condition
+       is unlikely even in the long term and certain not to arise in the
+       next several years.
+
+7.4.2.  Conditions Not Requiring a Prefix Change
+
+   As a result of the principles described above, none of the following
+   changes required a new prefix:
+
+   1.  Prohibition of some characters as input to IDNA.  Such a
+       prohibition might make names that were previously registered
+       inaccessible, but did not change those names.
+
+   2.  Adjustments in IDNA tables or actions, including normalization
+       definitions, that affected characters that were already invalid
+       under IDNA2003.
+
+   3.  Changes in the style of the IDNA definition that did not alter
+       the actions performed by IDNA.
+
+7.4.3.  Implications of Prefix Changes
+
+   While it might have been possible to make a prefix change, the costs
+   of such a change are considerable.  Registries could not have
+   converted all IDNA2003 ("xn--") registrations to a new form at the
+   same time and synchronize that change with applications supporting
+   lookup.  Unless all existing registrations were simply to be declared
+   invalid (and perhaps even then), systems that needed to support both
+   labels with old prefixes and labels with new ones would be required
+   to first process a putative label under the IDNA2008 rules and try to
+   look it up and then, if it were not found, would be required to
+   process the label under IDNA2003 rules and look it up again.  That
+   process would probably have significantly slowed down all processing
+   that involved IDNs in the DNS, especially since a fully-qualified
+   name might contain a mixture of labels that were registered with the
+   old and new prefixes.  That would have made DNS caching very
+   difficult.  In addition, looking up the same input string as two
+   separate A-labels would have created some potential for confusion and
+   attacks, since the labels could map to different targets and then
+   resolve to different entries in the DNS.
+
+
+
+
+Klensin                       Informational                    [Page 32]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   Consequently, a prefix change should have been, and was, avoided if
+   at all possible, even if it means accepting some IDNA2003 decisions
+   about character distinctions as irreversible and/or giving special
+   treatment to edge cases.
+
+7.5.  Stringprep Changes and Compatibility
+
+   The Nameprep specification [RFC3491], a key part of IDNA2003, is a
+   profile of Stringprep [RFC3454].  While Nameprep is a Stringprep
+   profile specific to IDNA, Stringprep is used by a number of other
+   protocols.  Were Stringprep to have been modified by IDNA2008, those
+   changes to improve the handling of IDNs could cause problems for
+   non-DNS uses, most notably if they affected identification and
+   authentication protocols.  Several elements of IDNA2008 give
+   interpretations to strings prohibited under IDNA2003 or prohibit
+   strings that IDNA2003 permitted.  Those elements include the new
+   inclusion information in the Tables document [RFC5892], the reduction
+   in the number of characters permitted as input for registration or
+   lookup (Section 3), and even the changes in handling of right-to-left
+   strings as described in the Bidi document [RFC5893].  IDNA2008 does
+   not use Nameprep or Stringprep at all, so there are no side-effect
+   changes to other protocols.
+
+   It is particularly important to keep IDNA processing separate from
+   processing for various security protocols because some of the
+   constraints that are necessary for smooth and comprehensible use of
+   IDNs may be unwanted or undesirable in other contexts.  For example,
+   the criteria for good passwords or passphrases are very different
+   from those for desirable IDNs: passwords should be hard to guess,
+   while domain names should normally be easily memorable.  Similarly,
+   internationalized Small Computer System Interface (SCSI) identifiers
+   and other protocol components are likely to have different
+   requirements than IDNs.
+
+7.6.  The Symbol Question
+
+   One of the major differences between this specification and the
+   original version of IDNA is that IDNA2003 permitted non-letter
+   symbols of various sorts, including punctuation and line-drawing
+   symbols, in the protocol.  They were always discouraged in practice.
+   In particular, both the "IESG Statement" about IDNA and all versions
+   of the ICANN Guidelines specify that only language characters be used
+   in labels.  This specification disallows symbols entirely.  There are
+   several reasons for this, which include:
+
+   1.  As discussed elsewhere, the original IDNA specification assumed
+       that as many Unicode characters as possible should be permitted,
+       directly or via mapping to other characters, in IDNs.  This
+
+
+
+Klensin                       Informational                    [Page 33]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+       specification operates on an inclusion model, extrapolating from
+       the original "hostname" rules (LDH, see the Definitions document
+       [RFC5890]) -- which have served the Internet very well -- to a
+       Unicode base rather than an ASCII base.
+
+   2.  Symbol names are more problematic than letters because there may
+       be no general agreement on whether a particular glyph matches a
+       symbol; there are no uniform conventions for naming; variations
+       such as outline, solid, and shaded forms may or may not exist;
+       and so on.  As just one example, consider a "heart" symbol as it
+       might appear in a logo that might be read as "I love...".  While
+       the user might read such a logo as "I love..." or "I heart...",
+       considerable knowledge of the coding distinctions made in Unicode
+       is needed to know that there is more than one "heart" character
+       (e.g., U+2665, U+2661, and U+2765) and how to describe it.  These
+       issues are of particular importance if strings are expected to be
+       understood or transcribed by the listener after being read out
+       loud.
+
+   3.  Design of a screen reader used by blind Internet users who must
+       listen to renderings of IDN domain names and possibly reproduce
+       them on the keyboard becomes considerably more complicated when
+       the names of characters are not obvious and intuitive to anyone
+       familiar with the language in question.
+
+   4.  As a simplified example of this, assume one wanted to use a
+       "heart" or "star" symbol in a label.  This is problematic because
+       those names are ambiguous in the Unicode system of naming (the
+       actual Unicode names require far more qualification).  A user or
+       would-be registrant has no way to know -- absent careful study of
+       the code tables -- whether it is ambiguous (e.g., where there are
+       multiple "heart" characters) or not.  Conversely, the user seeing
+       the hypothetical label doesn't know whether to read it -- try to
+       transmit it to a colleague by voice -- as "heart", as "love", as
+       "black heart", or as any of the other examples below.
+
+   5.  The actual situation is even worse than this.  There is no
+       possible way for a normal, casual, user to tell the difference
+       between the hearts of U+2665 and U+2765 and the stars of U+2606
+       and U+2729 without somehow knowing to look for a distinction.  We
+       have a white heart (U+2661) and few black hearts.  Consequently,
+       describing a label as containing a heart is hopelessly ambiguous:
+       we can only know that it contains one of several characters that
+       look like hearts or have "heart" in their names.  In cities where
+       "Square" is a popular part of a location name, one might well
+       want to use a square symbol in a label as well and there are far
+       more squares of various flavors in Unicode than there are hearts
+       or stars.
+
+
+
+Klensin                       Informational                    [Page 34]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   The consequence of these ambiguities is that symbols are a very poor
+   basis for reliable communication.  Consistent with this conclusion,
+   the Unicode standard recommends that strings used in identifiers not
+   contain symbols or punctuation [Unicode-UAX31].  Of course, these
+   difficulties with symbols do not arise with actual pictographic
+   languages and scripts which would be treated like any other language
+   characters; the two should not be confused.
+
+7.7.  Migration between Unicode Versions: Unassigned Code Points
+
+   In IDNA2003, labels containing unassigned code points are looked up
+   on the assumption that, if they appear in labels and can be mapped
+   and then resolved, the relevant standards must have changed and the
+   registry has properly allocated only assigned values.
+
+   In the IDNA2008 protocol, strings containing unassigned code points
+   must not be either looked up or registered.  In summary, the status
+   of an unassigned character with regard to the DISALLOWED,
+   PROTOCOL-VALID, and CONTEXTUAL RULE REQUIRED categories cannot be
+   evaluated until a character is actually assigned and known.  There
+   are several reasons for this, with the most important ones being:
+
+   o  Tests involving the context of characters (e.g., some characters
+      being permitted only adjacent to others of specific types) and
+      integrity tests on complete labels are needed.  Unassigned code
+      points cannot be permitted because one cannot determine whether
+      particular code points will require contextual rules (and what
+      those rules should be) before characters are assigned to them and
+      the properties of those characters fully understood.
+
+   o  It cannot be known in advance, and with sufficient reliability,
+      whether a newly assigned code point will be associated with a
+      character that would be disallowed by the rules in the Tables
+      document [RFC5892] (such as a compatibility character).  In
+      IDNA2003, since there is no direct dependency on NFKC (many of the
+      entries in Stringprep's tables are based on NFKC, but IDNA2003
+      depends only on Stringprep), allocation of a compatibility
+      character might produce some odd situations, but it would not be a
+      problem.  In IDNA2008, where compatibility characters are
+      DISALLOWED unless character-specific exceptions are made,
+      permitting strings containing unassigned characters to be looked
+      up would violate the principle that characters in DISALLOWED are
+      not looked up.
+
+   o  The Unicode Standard specifies that an unassigned code point
+      normalizes (and, where relevant, case folds) to itself.  If the
+      code point is later assigned to a character, and particularly if
+      the newly assigned code point has a combining class that
+
+
+
+Klensin                       Informational                    [Page 35]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+      determines its placement relative to other combining characters,
+      it could normalize to some other code point or sequence.
+
+   It is possible to argue that the issues above are not important and
+   that, as a consequence, it is better to retain the principle of
+   looking up labels even if they contain unassigned characters because
+   all of the important scripts and characters have been coded as of
+   Unicode 5.2 (or even earlier), and hence unassigned code points will
+   be assigned only to obscure characters or archaic scripts.
+   Unfortunately, that does not appear to be a safe assumption for at
+   least two reasons.  First, much the same claim of completeness has
+   been made for earlier versions of Unicode.  The reality is that a
+   script that is obscure to much of the world may still be very
+   important to those who use it.  Cultural and linguistic preservation
+   principles make it inappropriate to declare the script of no
+   importance in IDNs.  Second, we already have counterexamples, e.g.,
+   in the relationships associated with new Han characters being added
+   (whether in the BMP or in Unicode Plane 2).
+
+   Independent of the technical transition issues identified above, it
+   can be observed that any addition of characters to an existing script
+   to make it easier to use or to better accommodate particular
+   languages may lead to transition issues.  Such additions may change
+   the preferred form for writing a particular string, changes that may
+   be reflected, e.g., in keyboard transition modules that would
+   necessarily be different from those for earlier versions of Unicode
+   where the newer characters may not exist.  This creates an inherent
+   transition problem because attempts to access labels may use either
+   the old or the new conventions, requiring registry action whether or
+   not the older conventions were used in labels.  The need to consider
+   transition mechanisms is inherent to evolution of Unicode to better
+   accommodate writing systems and is independent of how IDNs are
+   represented in the DNS or how transitions among versions of those
+   mechanisms occur.  The requirement for transitions of this type is
+   illustrated by the addition of Malayalam Chillu in Unicode 5.1.0.
+
+7.8.  Other Compatibility Issues
+
+   The 2003 IDNA model includes several odd artifacts of the context in
+   which it was developed.  Many, if not all, of these are potential
+   avenues for exploits, especially if the registration process permits
+   "source" names (names that have not been processed through IDNA and
+   Nameprep) to be registered.  As one example, since the character
+   Eszett, used in German, is mapped by IDNA2003 into the sequence "ss"
+   rather than being retained as itself or prohibited, a string
+   containing that character, but that is otherwise in ASCII, is not
+   really an IDN (in the U-label sense defined above).  After Nameprep
+   maps out the Eszett, the result is an ASCII string and so it does not
+
+
+
+Klensin                       Informational                    [Page 36]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   get an xn-- prefix, but the string that can be displayed to a user
+   appears to be an IDN.  IDNA2008 eliminates this artifact.  A
+   character is either permitted as itself or it is prohibited; special
+   cases that make sense only in a particular linguistic or cultural
+   context can be dealt with as localization matters where appropriate.
+
+8.  Name Server Considerations
+
+8.1.  Processing Non-ASCII Strings
+
+   Existing DNS servers do not know the IDNA rules for handling
+   non-ASCII forms of IDNs, and therefore need to be shielded from them.
+   All existing channels through which names can enter a DNS server
+   database (for example, master files (as described in RFC 1034) and
+   DNS update messages [RFC2136]) could not be IDNA-aware because they
+   predate IDNA.  Other sections of this document provide the needed
+   shielding by ensuring that internationalized domain names entering
+   DNS server databases through such channels have already been
+   converted to their equivalent ASCII A-label forms.
+
+   Because of the distinction made between the algorithms for
+   Registration and Lookup in Sections 4 and 5 (respectively) of the
+   Protocol document [RFC5891] (a domain name containing only ASCII code
+   points cannot be converted to an A-label), there cannot be more than
+   one A-label form for any given U-label.
+
+   As specified in clarifications to the DNS specification [RFC2181],
+   the DNS protocol explicitly allows domain labels to contain octets
+   beyond the ASCII range (0000..007F), and this document does not
+   change that.  However, although the interpretation of octets
+   0080..00FF is well-defined in the DNS, many application protocols
+   support only ASCII labels and there is no defined interpretation of
+   these non-ASCII octets as characters and, in particular, no
+   interpretation of case-independent matching for them (e.g., see the
+   clarification on DNS case insensitivity [RFC4343]).  If labels
+   containing these octets are returned to applications, unpredictable
+   behavior could result.  The A-label form, which cannot contain those
+   characters, is the only standard representation for internationalized
+   labels in the DNS protocol.
+
+8.2.  Root and Other DNS Server Considerations
+
+   IDNs in A-label form will generally be somewhat longer than current
+   domain names, so the bandwidth needed by the root servers is likely
+   to go up by a small amount.  Also, queries and responses for IDNs
+   will probably be somewhat longer than typical queries historically,
+
+
+
+
+
+Klensin                       Informational                    [Page 37]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   so Extension Mechanisms for DNS (EDNS0) [RFC2671] support may be more
+   important (otherwise, queries and responses may be forced to go to
+   TCP instead of UDP).
+
+9.  Internationalization Considerations
+
+   DNS labels and fully-qualified domain names provide mnemonics that
+   assist in identifying and referring to resources on the Internet.
+   IDNs expand the range of those mnemonics to include those based on
+   languages and character sets other than Western European and Roman-
+   derived ones.  But domain "names" are not, in general, words in any
+   language.  The recommendations of the IETF policy on character sets
+   and languages (BCP 18 [RFC2277]) are applicable to situations in
+   which language identification is used to provide language-specific
+   contexts.  The DNS is, by contrast, global and international and
+   ultimately has nothing to do with languages.  Adding languages (or
+   similar context) to IDNs generally, or to DNS matching in particular,
+   would imply context-dependent matching in DNS, which would be a very
+   significant change to the DNS protocol itself.  It would also imply
+   that users would need to identify the language associated with a
+   particular label in order to look that label up.  That knowledge is
+   generally not available because many labels are not words in any
+   language and some may be words in more than one.
+
+10.  IANA Considerations
+
+   This section gives an overview of IANA registries required for IDNA.
+   The actual definitions of, and specifications for, the first two,
+   which have been newly created for IDNA2008, appear in the Tables
+   document [RFC5892].  This document describes the registries, but it
+   does not specify any IANA actions.
+
+10.1.  IDNA Character Registry
+
+   The distinction among the major categories "UNASSIGNED",
+   "DISALLOWED", "PROTOCOL-VALID", and "CONTEXTUAL RULE REQUIRED" is
+   made by special categories and rules that are integral elements of
+   the Tables document.  While not normative, an IANA registry of
+   characters and scripts and their categories, updated for each new
+   version of Unicode and the characters it contains, are convenient for
+   programming and validation purposes.  The details of this registry
+   are specified in the Tables document.
+
+
+
+
+
+
+
+
+
+Klensin                       Informational                    [Page 38]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+10.2.  IDNA Context Registry
+
+   IANA has created and now maintains a list of approved contextual
+   rules for characters that are defined in the IDNA Character Registry
+   list as requiring a Contextual Rule (i.e., the types of rules
+   described in Section 3.1.2).  The details for those rules appear in
+   the Tables document.
+
+10.3.  IANA Repository of IDN Practices of TLDs
+
+   This registry, historically described as the "IANA Language Character
+   Set Registry" or "IANA Script Registry" (both somewhat misleading
+   terms), is maintained by IANA at the request of ICANN.  It is used to
+   provide a central documentation repository of the IDN policies used
+   by top level domain (TLD) registries who volunteer to contribute to
+   it and is used in conjunction with ICANN Guidelines for IDN use.
+
+   It is not an IETF-managed registry and, while the protocol changes
+   specified here may call for some revisions to the tables, IDNA2008
+   has no direct effect on that registry and no IANA action is required
+   as a result.
+
+11.  Security Considerations
+
+11.1.  General Security Issues with IDNA
+
+   This document is purely explanatory and informational and
+   consequently introduces no new security issues.  It would, of course,
+   be a poor idea for someone to try to implement from it; such an
+   attempt would almost certainly lead to interoperability problems and
+   might lead to security ones.  A discussion of security issues with
+   IDNA, including some relevant history, appears in the Definitions
+   document [RFC5890].
+
+12.  Acknowledgments
+
+   The editor and contributors would like to express their thanks to
+   those who contributed significant early (pre-working group) review
+   comments, sometimes accompanied by text, Paul Hoffman, Simon
+   Josefsson, and Sam Weiler.  In addition, some specific ideas were
+   incorporated from suggestions, text, or comments about sections that
+   were unclear supplied by Vint Cerf, Frank Ellerman, Michael Everson,
+   Asmus Freytag, Erik van der Poel, Michel Suignard, and Ken Whistler.
+   Thanks are also due to Vint Cerf, Lisa Dusseault, Debbie Garside, and
+   Jefsey Morfin for conversations that led to considerable improvements
+   in the content of this document and to several others, including Ben
+
+
+
+
+
+Klensin                       Informational                    [Page 39]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   Campbell, Martin Duerst, Subramanian Moonesamy, Peter Saint-Andre,
+   and Dan Winship, for catching specific errors and recommending
+   corrections.
+
+   A meeting was held on 30 January 2008 to attempt to reconcile
+   differences in perspective and terminology about this set of
+   specifications between the design team and members of the Unicode
+   Technical Consortium.  The discussions at and subsequent to that
+   meeting were very helpful in focusing the issues and in refining the
+   specifications.  The active participants at that meeting were (in
+   alphabetic order, as usual) Harald Alvestrand, Vint Cerf, Tina Dam,
+   Mark Davis, Lisa Dusseault, Patrik Faltstrom (by telephone), Cary
+   Karp, John Klensin, Warren Kumari, Lisa Moore, Erik van der Poel,
+   Michel Suignard, and Ken Whistler.  We express our thanks to Google
+   for support of that meeting and to the participants for their
+   contributions.
+
+   Useful comments and text on the working group versions of the working
+   draft were received from many participants in the IETF "IDNABIS"
+   working group and a number of document changes resulted from mailing
+   list discussions made by that group.  Marcos Sanz provided specific
+   analysis and suggestions that were exceptionally helpful in refining
+   the text, as did Vint Cerf, Martin Duerst, Andrew Sullivan, and Ken
+   Whistler.  Lisa Dusseault provided extensive editorial suggestions
+   during the spring of 2009, most of which were incorporated.
+
+13.  Contributors
+
+   While the listed editor held the pen, the core of this document and
+   the initial working group version represents the joint work and
+   conclusions of an ad hoc design team consisting of the editor and, in
+   alphabetic order, Harald Alvestrand, Tina Dam, Patrik Faltstrom, and
+   Cary Karp.  Considerable material describing mapping principles has
+   been incorporated from a draft of the Mapping document
+   [IDNA2008-Mapping] by Pete Resnick and Paul Hoffman.  In addition,
+   there were many specific contributions and helpful comments from
+   those listed in the Acknowledgments section and others who have
+   contributed to the development and use of the IDNA protocols.
+
+14.  References
+
+14.1.  Normative References
+
+   [RFC3490]    Faltstrom, P., Hoffman, P., and A. Costello,
+                "Internationalizing Domain Names in Applications
+                (IDNA)", RFC 3490, March 2003.
+
+
+
+
+
+Klensin                       Informational                    [Page 40]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   [RFC3492]    Costello, A., "Punycode: A Bootstring encoding of
+                Unicode for Internationalized Domain Names in
+                Applications (IDNA)", RFC 3492, March 2003.
+
+   [RFC5890]    Klensin, J., "Internationalized Domain Names for
+                Applications (IDNA): Definitions and Document
+                Framework", RFC 5890, August 2010.
+
+   [RFC5891]    Klensin, J., "Internationalized Domain Names in
+                Applications (IDNA): Protocol", RFC 5891, August 2010.
+
+   [RFC5892]    Faltstrom, P., "The Unicode Code Points and
+                Internationalized Domain Names for Applications (IDNA)",
+                RFC 5892, August 2010.
+
+   [RFC5893]    Alvestrand, H. and C. Karp, "Right-to-Left Scripts for
+                Internationalized Domain Names for Applications (IDNA)",
+                RFC 5893, August 2010.
+
+   [Unicode52]  The Unicode Consortium.  The Unicode Standard, Version
+                5.2.0, defined by: "The Unicode Standard, Version
+                5.2.0", (Mountain View, CA: The Unicode Consortium,
+                2009. ISBN 978-1-936213-00-9).
+                <http://www.unicode.org/versions/Unicode5.2.0/>.
+
+14.2.  Informative References
+
+   [IDNA2008-Mapping]
+                Resnick, P. and P. Hoffman, "Mapping Characters in
+                Internationalized Domain Names for Applications (IDNA)",
+                Work in Progress, April 2010.
+
+   [RFC0952]    Harrenstien, K., Stahl, M., and E. Feinler, "DoD
+                Internet host table specification", RFC 952,
+                October 1985.
+
+   [RFC1034]    Mockapetris, P., "Domain names - concepts and
+                facilities", STD 13, RFC 1034, November 1987.
+
+   [RFC1035]    Mockapetris, P., "Domain names - implementation and
+                specification", STD 13, RFC 1035, November 1987.
+
+   [RFC1123]    Braden, R., "Requirements for Internet Hosts -
+                Application and Support", STD 3, RFC 1123, October 1989.
+
+   [RFC2136]    Vixie, P., Thomson, S., Rekhter, Y., and J.  Bound,
+                "Dynamic Updates in the Domain Name System (DNS
+                UPDATE)", RFC 2136, April 1997.
+
+
+
+Klensin                       Informational                    [Page 41]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   [RFC2181]    Elz, R. and R. Bush, "Clarifications to the DNS
+                Specification", RFC 2181, July 1997.
+
+   [RFC2277]    Alvestrand, H., "IETF Policy on Character Sets and
+                Languages", BCP 18, RFC 2277, January 1998.
+
+   [RFC2671]    Vixie, P., "Extension Mechanisms for DNS (EDNS0)",
+                RFC 2671, August 1999.
+
+   [RFC2782]    Gulbrandsen, A., Vixie, P., and L. Esibov, "A DNS RR for
+                specifying the location of services (DNS SRV)",
+                RFC 2782, February 2000.
+
+   [RFC3454]    Hoffman, P. and M. Blanchet, "Preparation of
+                Internationalized Strings ("stringprep")", RFC 3454,
+                December 2002.
+
+   [RFC3491]    Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep
+                Profile for Internationalized Domain Names (IDN)",
+                RFC 3491, March 2003.
+
+   [RFC3743]    Konishi, K., Huang, K., Qian, H., and Y. Ko, "Joint
+                Engineering Team (JET) Guidelines for Internationalized
+                Domain Names (IDN) Registration and Administration for
+                Chinese, Japanese, and Korean", RFC 3743, April 2004.
+
+   [RFC3987]    Duerst, M. and M. Suignard, "Internationalized Resource
+                Identifiers (IRIs)", RFC 3987, January 2005.
+
+   [RFC4290]    Klensin, J., "Suggested Practices for Registration of
+                Internationalized Domain Names (IDN)", RFC 4290,
+                December 2005.
+
+   [RFC4343]    Eastlake, D., "Domain Name System (DNS) Case
+                Insensitivity Clarification", RFC 4343, January 2006.
+
+   [RFC4690]    Klensin, J., Faltstrom, P., Karp, C., and IAB, "Review
+                and Recommendations for Internationalized Domain Names
+                (IDNs)", RFC 4690, September 2006.
+
+   [RFC4713]    Lee, X., Mao, W., Chen, E., Hsu, N., and J.  Klensin,
+                "Registration and Administration Recommendations for
+                Chinese Domain Names", RFC 4713, October 2006.
+
+
+
+
+
+
+
+
+Klensin                       Informational                    [Page 42]
+
+RFC 5894                     IDNA Rationale                  August 2010
+
+
+   [Unicode-UAX31]
+                The Unicode Consortium, "Unicode Standard Annex #31:
+                Unicode Identifier and Pattern Syntax, Revision 11",
+                September 2009,
+                <http://www.unicode.org/reports/tr31/tr31-11.html>.
+
+   [Unicode-UTS39]
+                The Unicode Consortium, "Unicode Technical Standard #39:
+                Unicode Security Mechanisms, Revision 2", August 2006,
+                <http://www.unicode.org/reports/tr39/tr39-2.html>.
+
+Author's Address
+
+   John C Klensin
+   1770 Massachusetts Ave, Ste 322
+   Cambridge, MA  02140
+   USA
+
+   Phone: +1 617 245 1457
+   EMail: john+ietf@jck.com
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Klensin                       Informational                    [Page 43]
+