summaryrefslogtreecommitdiff
path: root/doc/rfc/rfc4042.txt
diff options
context:
space:
mode:
Diffstat (limited to 'doc/rfc/rfc4042.txt')
-rw-r--r--doc/rfc/rfc4042.txt507
1 files changed, 507 insertions, 0 deletions
diff --git a/doc/rfc/rfc4042.txt b/doc/rfc/rfc4042.txt
new file mode 100644
index 0000000..d0a5d0f
--- /dev/null
+++ b/doc/rfc/rfc4042.txt
@@ -0,0 +1,507 @@
+
+
+
+
+
+
+Network Working Group M. Crispin
+Request for Comments: 4042 Panda Programming
+Category: Informational 1 April 2005
+
+
+ UTF-9 and UTF-18
+ Efficient Transformation Formats of Unicode
+
+Status of This Memo
+
+ This memo provides information for the Internet community. It does
+ not specify an Internet standard of any kind. Distribution of this
+ memo is unlimited.
+
+Copyright Notice
+
+ Copyright (C) The Internet Society (2005).
+
+Abstract
+
+ ISO-10646 defines a large character set called the Universal
+ Character Set (UCS), which encompasses most of the world's writing
+ systems. The same set of codepoints is defined by Unicode, which
+ further defines additional character properties and other
+ implementation details. By policy of the relevant standardization
+ committees, changes to Unicode and amendments and additions to
+ ISO/IEC 646 track each other, so that the character repertoires and
+ code point assignments remain in synchronization.
+
+ The current representation formats for Unicode (UTF-7, UTF-8, UTF-16)
+ are not storage and computation efficient on platforms that utilize
+ the 9 bit nonet as a natural storage unit instead of the 8 bit octet.
+
+ This document describes a transformation format of Unicode that takes
+ advantage of the nonet so that the format will be storage and
+ computation efficient.
+
+1. Introduction
+
+ A number of Internet sites utilize platforms that are not based upon
+ the traditional 8-bit byte or octet. One such platform is the PDP-
+ 10, which is based upon a 36-bit word. On these platforms, it is
+ wasteful to represent data in octets, since 4 bits are left unused in
+ each word. The 9-bit nonet is a much more sensible representation.
+
+ Although these platforms support IETF standards, many of these
+ platforms still utilize a text representation based upon the septet,
+
+
+
+
+Crispin Informational [Page 1]
+
+RFC 4042 UTF-9 and UTF-18 1 April 2005
+
+
+ which is only suitable for [US-ASCII] (although it has been used for
+ various ISO 10646 national variants).
+
+ To maximize international and multi-lingual interoperability, the IAB
+ has recommended ([IAB-CHARACTER]) that [ISO-10646] be the default
+ coded character set.
+
+ Although other transformation formats of [UNICODE] exist, and
+ conceivably can be used on nonet-oriented machines (most notably
+ [UTF-8]), they suffer significant disadvantages:
+
+ [UTF-8]
+ requires one to three octets to represent codepoints in the
+ Basic Multilingual Plane (BMP), four octets to represent
+ [UNICODE] codepoints outside the BMP, and six octets to
+ represent non-[UNICODE] codepoints. When stored in nonets,
+ this results in as many as four wasted bits per [UNICODE]
+ character.
+
+ [UTF-16]
+ requires a hexadecet to represent codepoints in the BMP, and
+ two hexadecets to represent [UNICODE] codepoints outside the
+ BMP. When stored in nonet pairs, this results in as many as
+ four wasted bits per [UNICODE] character. This transformation
+ format requires complex surrogates to represent codepoints
+ outside the BMP, and can not represent non-[UNICODE] codepoints
+ at all.
+
+ [UTF-7]
+ requires one to five septets to represent codepoints in the
+ BMP, and as many as eight septets to represent codepoints
+ outside the BMP. When stored in nonets, this results in as
+ many as sixteen wasted bits per character. This transformation
+ format requires very complex and computationally expensive
+ shifting and "modified BASE64" processing, and can not
+ represent non-[UNICODE] codepoints at all.
+
+ By comparison, UTF-9 uses one to two nonets to represent codepoints
+ in the BMP, three nonets to represent [UNICODE] codepoints outside
+ the BMP, and three or four nonets to represent non-[UNICODE]
+ codepoints. There are no wasted bits, and as the examples in this
+ document demonstrate, the computational processing is minimal.
+
+ Transformation between [UTF-8] and UTF-9 is straightforward, with
+ most of the complexity in the handling of [UTF-8]. It is hoped that
+ future extensions to protocols such as SMTP will permit the use of
+ UTF-9 in these protocols between nonet platforms without the use of
+ [UTF-8] as an "on the wire" format.
+
+
+
+Crispin Informational [Page 2]
+
+RFC 4042 UTF-9 and UTF-18 1 April 2005
+
+
+ Similarly, transformation between [UNICODE] codepoints and UTF-18 is
+ also quite simple. Although (like UCS-2) UTF-18 only represents a
+ subset of the available [UNICODE] codepoints, it encompasses the
+ non-private codepoints that are currently assigned in [UNICODE].
+
+1.1. Conventions Used in This Document
+
+ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+ "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
+ document are to be interpreted as described in BCP 14, RFC 2119
+ [KEYWORDS].
+
+2. Overview
+
+ UTF-9 encodes [UNICODE] codepoints in the low order 8 bits of a
+ nonet, using the high order bit to indicate continuation. Surrogates
+ are not used.
+
+ [UNICODE] codepoints in the range U+0000 - U+00FF ([US-ASCII] and
+ Latin 1) are represented by a single nonet; codepoints in the range
+ U+0100 - U+FFFF (the remainder of the BMP) are represented by two
+ nonets; and codepoints in the range U+1000 - U+10FFFF (remainder of
+ [UNICODE]) are represented by three nonets.
+
+ Non-[UNICODE] codepoints in [ISO-10646] (that is, codepoints in the
+ range 0x110000 - 0x7fffffff) can also be represented in UTF-9 by
+ obvious extension, but this is not discussed further as these
+ codepoints have been removed from [ISO-10646] by ISO.
+
+ UTF-18 encodes [UNICODE] codepoints in the Basic Multilingual Plane
+ (BMP, plane 0), Supplementary Multilingual Plane (SMP, plane 1),
+ Supplementary Ideographic Plane (SIP, plane 2), and Supplementary
+ Special-purpose Plane (SSP, plane 14) in a single 18-bit value. It
+ does not encode planes 3 though 13, which are currently unused; nor
+ planes 15 or 16, which are private spaces.
+
+ Normally, UTF-9 and UTF-18 should only be used in the context of 9
+ bit storage and transport. Although some protocols, e.g., [FTP],
+ support transport of nonets, the current IETF protocol suite is quite
+ deficient in this area. The IETF is urged to take action to improve
+ IETF protocol support for nonets.
+
+3. UTF-9 Definition
+
+ A UTF-9 stream represents [ISO-10646] codepoints using 9 bit nonets.
+ The low order 8-bits of a nonet is an octet, and the high order bit
+ indicates continuation.
+
+
+
+
+Crispin Informational [Page 3]
+
+RFC 4042 UTF-9 and UTF-18 1 April 2005
+
+
+ UTF-9 does not use surrogates; consequently a UTF-16 value must be
+ transformed into the UCS-4 equivalent, and U+D800 - U+DBFF are never
+ transmitted in UTF-9.
+
+ Octets of the [UNICODE] codepoint value are then copied into
+ successive UTF-9 nonets, starting with the most-significant non-zero
+ octet. All but the least significant octet have the continuation bit
+ set in the associated nonet.
+
+ Examples:
+
+ Character Name UTF-9 (in octal)
+ --------- ---- ----------------
+ U+0041 LATIN CAPITAL LETTER A 101
+ U+00C0 LATIN CAPITAL LETTER A WITH GRAVE 300
+ U+0391 GREEK CAPITAL LETTER ALPHA 403 221
+ U+611B <CJK ideograph meaning "love"> 541 33
+ U+10330 GOTHIC LETTER AHSA 401 403 60
+ U+E0041 TAG LATIN CAPITAL LETTER A 416 400 101
+ U+10FFFD <Plane 16 Private Use, Last> 420 777 375
+ 0x345ecf1b (UCS-4 value not in [UNICODE]) 464 536 717 33
+
+4. UTF-18 Definition
+
+ A UTF-18 stream represents [ISO-10646] codepoints using a pair of 9
+ bit nonets to form an 18-bit value.
+
+ UTF-18 does not use surrogates; consequently a UTF-16 value must be
+ transformed into the UCS-4 equivalent, and U+D800 - U+DBFF are never
+ transmitted in UTF-18.
+
+ [UNICODE] codepoint values in the range U+0000 - U+2FFFF are copied
+ as the same value into a UTF-18 value. [UNICODE] codepoint values in
+ the range U+E0000 - U+EFFFF are copied as values 0x30000 - 0x3ffff;
+ that is, these values are shifted by 0x70000. Other codepoint values
+ can not be represented in UTF-18.
+
+ Examples:
+
+ Character Name UTF-18 (in octal)
+ --------- ---- ----------------
+ U+0041 LATIN CAPITAL LETTER A 000101
+ U+00C0 LATIN CAPITAL LETTER A WITH GRAVE 000300
+ U+0391 GREEK CAPITAL LETTER ALPHA 001621
+ U+611B <CJK ideograph meaning "love"> 060433
+ U+10330 GOTHIC LETTER AHSA 201460
+ U+E0041 TAG LATIN CAPITAL LETTER A 600101
+
+
+
+
+Crispin Informational [Page 4]
+
+RFC 4042 UTF-9 and UTF-18 1 April 2005
+
+
+5. Sample Routines
+
+5.1. [UNICODE] Codepoint to UTF-9 Conversion
+
+ The following routines demonstrate conversion from UCS-4 to UTF-9.
+ For simplicity, these routines do not do any validity checking.
+ Routines used in applications SHOULD reject invalid UTF-9 sequences;
+ that is, the first nonet with a value of 400 octal (0x100), or
+ sequences that result in an overflow (exceeding 0x10ffff for
+ [UNICODE]), or codepoints used for UTF-16 surrogates.
+
+ ; Return UCS-4 value from UTF-9 string (PDP-10 assembly version)
+ ; Accepts: P1/ 9-bit byte pointer to UTF-9 string
+ ; Returns +1: Always, T1/ UCS-4 value, P1/ updated byte pointer
+ ; Clobbers T2
+
+ UT92U4: TDZA T1,T1 ; start with zero
+ U92U41: XOR T1,T2 ; insert octet into UCS-4 value
+ LSH T1,^D8 ; shift UCS-4 value
+ ILDB T2,P1 ; get next nonet
+ TRZE T2,400 ; extract octet, any continuation?
+ JRST U92U41 ; yes, continue
+ XOR T1,T2 ; insert final octet
+ POPJ P,
+
+ /* Return UCS-4 value from UTF-9 string (C version)
+ * Accepts: pointer to pointer to UTF-9 string
+ * Returns: UCS-4 character, nonet pointer updated
+ */
+
+ UINT31 UTF9_to_UCS4 (UINT9 **utf9PP)
+ {
+ UINT9 nonet;
+ UINT31 ucs4;
+ for (ucs4 = (nonet = *(*utf9PP)++) & 0xff;
+ nonet & 0x100;
+ ucs4 |= (nonet = *(*utf9PP)++) & 0xff)
+ ucs4 <<= 8;
+ return ucs4;
+ }
+
+5.2. UTF-9 to UCS-4 Conversion
+
+ The following routines demonstrate conversion from UTF-9 to UCS-4.
+ For simplicity, these routines do not do any validity checking.
+ Routines used in applications SHOULD reject invalid UCS-4 codepoints;
+ that is, codepoints used for UTF-16 surrogates or codepoints with
+ values exceeding 0x10ffff for [UNICODE].
+
+
+
+Crispin Informational [Page 5]
+
+RFC 4042 UTF-9 and UTF-18 1 April 2005
+
+
+ ; Write UCS-4 character to UTF-9 string (PDP-10 assembly version)
+ ; Accepts: P1/ 9-bit byte pointer to UTF-9 string
+ ; T1/ UCS-4 character to write
+ ; Returns +1: Always, P1/ updated byte pointer
+ ; Clobbers T1, T2; (T1, T2) must be an accumulator pair
+
+ U42UT9: SETO T2, ; we'll need some of these 1-bits later
+ ASHC T1,-^D8 ; low octet becomes nonet with high 0-bit
+ U32U91: JUMPE T1,U42U9X ; done if no more octets
+ LSHC T1,-^D8 ; shift next octet into T2
+ ROT T2,-1 ; turn it into nonet with high 1 bit
+ PUSHJ P,U42U91 ; recurse for remainder
+ U42U9X: LSHC T1,^D9 ; get next nonet back from T2
+ IDPB T1,P1 ; write nonet
+ POPJ P,
+
+ /* Write UCS-4 character to UTF-9 string (C version)
+ * Accepts: pointer to nonet string
+ * UCS-4 character to write
+ * Returns: updated pointer
+ */
+
+ UINT9 *UCS4_to_UTF9 (UINT9 *utf9P,UINT31 ucs4)
+ {
+ if (ucs4 > 0x100) {
+ if (ucs4 > 0x10000) {
+ if (ucs4 > 0x1000000)
+ *utf9P++ = 0x100 | ((ucs4 >> 24) & 0xff);
+ *utf9P++ = 0x100 | ((ucs4 >> 16) & 0xff);
+ }
+ *utf9P++ = 0x100 | ((ucs4 >> 8) & 0xff);
+ }
+ *utf9P++ = ucs4 & 0xff;
+ return utf9P;
+ }
+
+6. Implementation Experience
+
+ As the sample routines demonstrate, it is quite simple to implement
+ UTF-9 and UTF-18 on a nonet-based architecture. More sophisticated
+ routines can be found in ftp://panda.com/tops-20/utools.mac.txt or
+ from lingling.panda.com via the file <UTF9>UTOOLS.MAC via ANONYMOUS
+ [FTP].
+
+
+
+
+
+
+
+
+Crispin Informational [Page 6]
+
+RFC 4042 UTF-9 and UTF-18 1 April 2005
+
+
+ We are now in the process of implementing support for nonet-based
+ text files and automated transformation between septet, octet, and
+ nonet textual data.
+
+7. References
+
+7.1. Normative References
+
+ [FTP] Postel, J. and J. Reynolds, "File Transfer Protocol",
+ STD 9, RFC 959, October 1985.
+
+ [IAB-CHARACTER] Weider, C., Preston, C., Simonsen, K., Alvestrand,
+ H., Atkinson, R., Crispin, M., and P. Svanberg, "The
+ Report of the IAB Character Set Workshop held 29
+ February - 1 March, 1996", RFC 2130, April 1997.
+
+ [ISO-10646] International Organization for Standardization,
+ "Information Technology - Universal Multiple-octet
+ coded Character Set (UCS)", ISO/IEC Standard 10646,
+ comprised of ISO/IEC 10646-1:2000, "Information
+ technology - Universal Multiple-Octet Coded Character
+ Set (UCS) - Part 1: Architecture and Basic
+ Multilingual Plane", ISO/IEC 10646-2:2001,
+ "Information technology - Universal Multiple-Octet
+ Coded Character Set (UCS) - Part 2: Supplementary
+ Planes" and ISO/IEC 10646-1:2000/Amd 1:2002,
+ "Mathematical symbols and other characters".
+
+ [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate
+ Requirement Levels", BCP 14, RFC 2119, March 1997.
+
+ [UNICODE] The Unicode Consortium, "The Unicode Standard -
+ Version 3.2", defined by The Unicode Standard,
+ Version 3.0 (Reading, MA, Addison-Wesley, 2000. ISBN
+ 0-201-61633-5), as amended by the Unicode Standard
+ Annex #27: Unicode 3.1 and by the Unicode Standard
+ Annex #28: Unicode 3.2, March 2002.
+
+7.2. Informative References
+
+ [US-ASCII] American National Standards Institute, "Coded
+ Character Set - 7-bit American Standard Code for
+ Information Interchange", ANSI X3.4, 1986.
+
+ [UTF-16] Hoffman, P. and F. Yergeau, "UTF-16, an encoding of
+ ISO 10646", RFC 2781, February 2000.
+
+
+
+
+
+Crispin Informational [Page 7]
+
+RFC 4042 UTF-9 and UTF-18 1 April 2005
+
+
+ [UTF-7] Goldsmith, D. and M. Davis, "UTF-7 A Mail-Safe
+ Transformation Format of Unicode", RFC 2152, May
+ 1997.
+
+ [UTF-8] Sollins, K., "Architectural Principles of Uniform
+ Resource Name Resolution", RFC 2276, January 1998.
+
+8. Security Considerations
+
+ As with UTF-8, UTF-9 can represent codepoints that are not in
+ [UNICODE]. Applications should validate UTF-9 strings to ensure that
+ all codepoints do not exceed the [UNICODE] maximum of U+10FFFF.
+
+ The sample routines in this document are for example purposes, and
+ make no attempt to validate their arguments, e.g., test for overflow
+ ([UNICODE] values great than 0x10ffff) or codepoints used for
+ surrogates. Besides resulting in invalid data, this can also create
+ covert channels.
+
+9. IANA Considerations
+
+ The IANA shall reserve the charset names "UTF-9" and "UTF-18" for
+ future assignment.
+
+Author's Address
+
+ Mark R. Crispin
+ Panda Programming
+ 6158 NE Lariat Loop
+ Bainbridge Island, WA 98110-2098
+
+ Phone: (206) 842-2385
+ EMail: UTF9@Lingling.Panda.COM
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Crispin Informational [Page 8]
+
+RFC 4042 UTF-9 and UTF-18 1 April 2005
+
+
+Full Copyright Statement
+
+ Copyright (C) The Internet Society (2005).
+
+ This document is subject to the rights, licenses and restrictions
+ contained in BCP 78 and at www.rfc-editor.org/copyright.html, and
+ except as set forth therein, the authors retain all their rights.
+
+ This document and the information contained herein are provided on an
+ "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
+ OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
+ ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
+ INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
+ INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
+ WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
+
+Intellectual Property
+
+ The IETF takes no position regarding the validity or scope of any
+ Intellectual Property Rights or other rights that might be claimed to
+ pertain to the implementation or use of the technology described in
+ this document or the extent to which any license under such rights
+ might or might not be available; nor does it represent that it has
+ made any independent effort to identify any such rights. Information
+ on the procedures with respect to rights in RFC documents can be
+ found in BCP 78 and BCP 79.
+
+ Copies of IPR disclosures made to the IETF Secretariat and any
+ assurances of licenses to be made available, or the result of an
+ attempt made to obtain a general license or permission for the use of
+ such proprietary rights by implementers or users of this
+ specification can be obtained from the IETF on-line IPR repository at
+ http://www.ietf.org/ipr.
+
+ The IETF invites any interested party to bring to its attention any
+ copyrights, patents or patent applications, or other proprietary
+ rights that may cover technology that may be required to implement
+ this standard. Please address the information to the IETF at ietf-
+ ipr@ietf.org.
+
+Acknowledgement
+
+ Funding for the RFC Editor function is currently provided by the
+ Internet Society.
+
+
+
+
+
+
+
+Crispin Informational [Page 9]
+