1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
|
Network Working Group F. Yergeau
Request for Comments: 3629 Alis Technologies
STD: 63 November 2003
Obsoletes: 2279
Category: Standards Track
UTF-8, a transformation format of ISO 10646
Status of this Memo
This document specifies an Internet standards track protocol for the
Internet community, and requests discussion and suggestions for
improvements. Please refer to the current edition of the "Internet
Official Protocol Standards" (STD 1) for the standardization state
and status of this protocol. Distribution of this memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2003). All Rights Reserved.
Abstract
ISO/IEC 10646-1 defines a large character set called the Universal
Character Set (UCS) which encompasses most of the world's writing
systems. The originally proposed encodings of the UCS, however, were
not compatible with many current applications and protocols, and this
has led to the development of UTF-8, the object of this memo. UTF-8
has the characteristic of preserving the full US-ASCII range,
providing compatibility with file systems, parsers and other software
that rely on US-ASCII values but are transparent to other values.
This memo obsoletes and replaces RFC 2279.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Notational conventions . . . . . . . . . . . . . . . . . . . . 3
3. UTF-8 definition . . . . . . . . . . . . . . . . . . . . . . . 4
4. Syntax of UTF-8 Byte Sequences . . . . . . . . . . . . . . . . 5
5. Versions of the standards . . . . . . . . . . . . . . . . . . 6
6. Byte order mark (BOM) . . . . . . . . . . . . . . . . . . . . 6
7. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
8. MIME registration . . . . . . . . . . . . . . . . . . . . . . 9
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10
10. Security Considerations . . . . . . . . . . . . . . . . . . . 10
11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 11
12. Changes from RFC 2279 . . . . . . . . . . . . . . . . . . . . 11
13. Normative References . . . . . . . . . . . . . . . . . . . . . 12
Yergeau Standards Track [Page 1]
^L
RFC 3629 UTF-8 November 2003
14. Informative References . . . . . . . . . . . . . . . . . . . . 12
15. URI's . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
16. Intellectual Property Statement . . . . . . . . . . . . . . . 13
17. Author's Address . . . . . . . . . . . . . . . . . . . . . . . 13
18. Full Copyright Statement . . . . . . . . . . . . . . . . . . . 14
1. Introduction
ISO/IEC 10646 [ISO.10646] defines a large character set called the
Universal Character Set (UCS), which encompasses most of the world's
writing systems. The same set of characters is defined by the
Unicode standard [UNICODE], which further defines additional
character properties and other application details of great interest
to implementers. Up to the present time, changes in Unicode and
amendments and additions to ISO/IEC 10646 have tracked each other, so
that the character repertoires and code point assignments have
remained in sync. The relevant standardization committees have
committed to maintain this very useful synchronism.
ISO/IEC 10646 and Unicode define several encoding forms of their
common repertoire: UTF-8, UCS-2, UTF-16, UCS-4 and UTF-32. In an
encoding form, each character is represented as one or more encoding
units. All standard UCS encoding forms except UTF-8 have an encoding
unit larger than one octet, making them hard to use in many current
applications and protocols that assume 8 or even 7 bit characters.
UTF-8, the object of this memo, has a one-octet encoding unit. It
uses all bits of an octet, but has the quality of preserving the full
US-ASCII [US-ASCII] range: US-ASCII characters are encoded in one
octet having the normal US-ASCII value, and any octet with such a
value can only stand for a US-ASCII character, and nothing else.
UTF-8 encodes UCS characters as a varying number of octets, where the
number of octets, and the value of each, depend on the integer value
assigned to the character in ISO/IEC 10646 (the character number,
a.k.a. code position, code point or Unicode scalar value). This
encoding form has the following characteristics (all values are in
hexadecimal):
o Character numbers from U+0000 to U+007F (US-ASCII repertoire)
correspond to octets 00 to 7F (7 bit US-ASCII values). A direct
consequence is that a plain ASCII string is also a valid UTF-8
string.
Yergeau Standards Track [Page 2]
^L
RFC 3629 UTF-8 November 2003
o US-ASCII octet values do not appear otherwise in a UTF-8 encoded
character stream. This provides compatibility with file systems
or other software (e.g., the printf() function in C libraries)
that parse based on US-ASCII values but are transparent to other
values.
o Round-trip conversion is easy between UTF-8 and other encoding
forms.
o The first octet of a multi-octet sequence indicates the number of
octets in the sequence.
o The octet values C0, C1, F5 to FF never appear.
o Character boundaries are easily found from anywhere in an octet
stream.
o The byte-value lexicographic sorting order of UTF-8 strings is the
same as if ordered by character numbers. Of course this is of
limited interest since a sort order based on character numbers is
almost never culturally valid.
o The Boyer-Moore fast search algorithm can be used with UTF-8 data.
o UTF-8 strings can be fairly reliably recognized as such by a
simple algorithm, i.e., the probability that a string of
characters in any other encoding appears as valid UTF-8 is low,
diminishing with increasing string length.
UTF-8 was devised in September 1992 by Ken Thompson, guided by design
criteria specified by Rob Pike, with the objective of defining a UCS
transformation format usable in the Plan9 operating system in a non-
disruptive manner. Thompson's design was stewarded through
standardization by the X/Open Joint Internationalization Group XOJIG
(see [FSS_UTF]), bearing the names FSS-UTF (variant FSS/UTF), UTF-2
and finally UTF-8 along the way.
2. Notational conventions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
UCS characters are designated by the U+HHHH notation, where HHHH is a
string of from 4 to 6 hexadecimal digits representing the character
number in ISO/IEC 10646.
Yergeau Standards Track [Page 3]
^L
RFC 3629 UTF-8 November 2003
3. UTF-8 definition
UTF-8 is defined by the Unicode Standard [UNICODE]. Descriptions and
formulae can also be found in Annex D of ISO/IEC 10646-1 [ISO.10646]
In UTF-8, characters from the U+0000..U+10FFFF range (the UTF-16
accessible range) are encoded using sequences of 1 to 4 octets. The
only octet of a "sequence" of one has the higher-order bit set to 0,
the remaining 7 bits being used to encode the character number. In a
sequence of n octets, n>1, the initial octet has the n higher-order
bits set to 1, followed by a bit set to 0. The remaining bit(s) of
that octet contain bits from the number of the character to be
encoded. The following octet(s) all have the higher-order bit set to
1 and the following bit set to 0, leaving 6 bits in each to contain
bits from the character to be encoded.
The table below summarizes the format of these different octet types.
The letter x indicates bits available for encoding bits of the
character number.
Char. number range | UTF-8 octet sequence
(hexadecimal) | (binary)
--------------------+---------------------------------------------
0000 0000-0000 007F | 0xxxxxxx
0000 0080-0000 07FF | 110xxxxx 10xxxxxx
0000 0800-0000 FFFF | 1110xxxx 10xxxxxx 10xxxxxx
0001 0000-0010 FFFF | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
Encoding a character to UTF-8 proceeds as follows:
1. Determine the number of octets required from the character number
and the first column of the table above. It is important to note
that the rows of the table are mutually exclusive, i.e., there is
only one valid way to encode a given character.
2. Prepare the high-order bits of the octets as per the second
column of the table.
3. Fill in the bits marked x from the bits of the character number,
expressed in binary. Start by putting the lowest-order bit of
the character number in the lowest-order position of the last
octet of the sequence, then put the next higher-order bit of the
character number in the next higher-order position of that octet,
etc. When the x bits of the last octet are filled in, move on to
the next to last octet, then to the preceding one, etc. until all
x bits are filled in.
Yergeau Standards Track [Page 4]
^L
RFC 3629 UTF-8 November 2003
The definition of UTF-8 prohibits encoding character numbers between
U+D800 and U+DFFF, which are reserved for use with the UTF-16
encoding form (as surrogate pairs) and do not directly represent
characters. When encoding in UTF-8 from UTF-16 data, it is necessary
to first decode the UTF-16 data to obtain character numbers, which
are then encoded in UTF-8 as described above. This contrasts with
CESU-8 [CESU-8], which is a UTF-8-like encoding that is not meant for
use on the Internet. CESU-8 operates similarly to UTF-8 but encodes
the UTF-16 code values (16-bit quantities) instead of the character
number (code point). This leads to different results for character
numbers above 0xFFFF; the CESU-8 encoding of those characters is NOT
valid UTF-8.
Decoding a UTF-8 character proceeds as follows:
1. Initialize a binary number with all bits set to 0. Up to 21 bits
may be needed.
2. Determine which bits encode the character number from the number
of octets in the sequence and the second column of the table
above (the bits marked x).
3. Distribute the bits from the sequence to the binary number, first
the lower-order bits from the last octet of the sequence and
proceeding to the left until no x bits are left. The binary
number is now equal to the character number.
Implementations of the decoding algorithm above MUST protect against
decoding invalid sequences. For instance, a naive implementation may
decode the overlong UTF-8 sequence C0 80 into the character U+0000,
or the surrogate pair ED A1 8C ED BE B4 into U+233B4. Decoding
invalid sequences may have security consequences or cause other
problems. See Security Considerations (Section 10) below.
4. Syntax of UTF-8 Byte Sequences
For the convenience of implementors using ABNF, a definition of UTF-8
in ABNF syntax is given here.
A UTF-8 string is a sequence of octets representing a sequence of UCS
characters. An octet sequence is valid UTF-8 only if it matches the
following syntax, which is derived from the rules for encoding UTF-8
and is expressed in the ABNF of [RFC2234].
UTF8-octets = *( UTF8-char )
UTF8-char = UTF8-1 / UTF8-2 / UTF8-3 / UTF8-4
UTF8-1 = %x00-7F
UTF8-2 = %xC2-DF UTF8-tail
Yergeau Standards Track [Page 5]
^L
RFC 3629 UTF-8 November 2003
UTF8-3 = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2( UTF8-tail ) /
%xED %x80-9F UTF8-tail / %xEE-EF 2( UTF8-tail )
UTF8-4 = %xF0 %x90-BF 2( UTF8-tail ) / %xF1-F3 3( UTF8-tail ) /
%xF4 %x80-8F 2( UTF8-tail )
UTF8-tail = %x80-BF
NOTE -- The authoritative definition of UTF-8 is in [UNICODE]. This
grammar is believed to describe the same thing Unicode describes, but
does not claim to be authoritative. Implementors are urged to rely
on the authoritative source, rather than on this ABNF.
5. Versions of the standards
ISO/IEC 10646 is updated from time to time by publication of
amendments and additional parts; similarly, new versions of the
Unicode standard are published over time. Each new version obsoletes
and replaces the previous one, but implementations, and more
significantly data, are not updated instantly.
In general, the changes amount to adding new characters, which does
not pose particular problems with old data. In 1996, Amendment 5 to
the 1993 edition of ISO/IEC 10646 and Unicode 2.0 moved and expanded
the Korean Hangul block, thereby making any previous data containing
Hangul characters invalid under the new version. Unicode 2.0 has the
same difference from Unicode 1.1. The justification for allowing
such an incompatible change was that there were no major
implementations and no significant amounts of data containing Hangul.
The incident has been dubbed the "Korean mess", and the relevant
committees have pledged to never, ever again make such an
incompatible change (see Unicode Consortium Policies [1]).
New versions, and in particular any incompatible changes, have
consequences regarding MIME charset labels, to be discussed in MIME
registration (Section 8).
6. Byte order mark (BOM)
The UCS character U+FEFF "ZERO WIDTH NO-BREAK SPACE" is also known
informally as "BYTE ORDER MARK" (abbreviated "BOM"). This character
can be used as a genuine "ZERO WIDTH NO-BREAK SPACE" within text, but
the BOM name hints at a second possible usage of the character: to
prepend a U+FEFF character to a stream of UCS characters as a
"signature". A receiver of such a serialized stream may then use the
initial character as a hint that the stream consists of UCS
characters and also to recognize which UCS encoding is involved and,
with encodings having a multi-octet encoding unit, as a way to
Yergeau Standards Track [Page 6]
^L
RFC 3629 UTF-8 November 2003
recognize the serialization order of the octets. UTF-8 having a
single-octet encoding unit, this last function is useless and the BOM
will always appear as the octet sequence EF BB BF.
It is important to understand that the character U+FEFF appearing at
any position other than the beginning of a stream MUST be interpreted
with the semantics for the zero-width non-breaking space, and MUST
NOT be interpreted as a signature. When interpreted as a signature,
the Unicode standard suggests than an initial U+FEFF character may be
stripped before processing the text. Such stripping is necessary in
some cases (e.g., when concatenating two strings, because otherwise
the resulting string may contain an unintended "ZERO WIDTH NO-BREAK
SPACE" at the connection point), but might affect an external process
at a different layer (such as a digital signature or a count of the
characters) that is relying on the presence of all characters in the
stream. It is therefore RECOMMENDED to avoid stripping an initial
U+FEFF interpreted as a signature without a good reason, to ignore it
instead of stripping it when appropriate (such as for display) and to
strip it only when really necessary.
U+FEFF in the first position of a stream MAY be interpreted as a
zero-width non-breaking space, and is not always a signature. In an
attempt at diminishing this uncertainty, Unicode 3.2 adds a new
character, U+2060 "WORD JOINER", with exactly the same semantics and
usage as U+FEFF except for the signature function, and strongly
recommends its exclusive use for expressing word-joining semantics.
Eventually, following this recommendation will make it all but
certain that any initial U+FEFF is a signature, not an intended "ZERO
WIDTH NO-BREAK SPACE".
In the meantime, the uncertainty unfortunately remains and may affect
Internet protocols. Protocol specifications MAY restrict usage of
U+FEFF as a signature in order to reduce or eliminate the potential
ill effects of this uncertainty. In the interest of striking a
balance between the advantages (reduction of uncertainty) and
drawbacks (loss of the signature function) of such restrictions, it
is useful to distinguish a few cases:
o A protocol SHOULD forbid use of U+FEFF as a signature for those
textual protocol elements that the protocol mandates to be always
UTF-8, the signature function being totally useless in those
cases.
o A protocol SHOULD also forbid use of U+FEFF as a signature for
those textual protocol elements for which the protocol provides
character encoding identification mechanisms, when it is expected
that implementations of the protocol will be in a position to
always use the mechanisms properly. This will be the case when
Yergeau Standards Track [Page 7]
^L
RFC 3629 UTF-8 November 2003
the protocol elements are maintained tightly under the control of
the implementation from the time of their creation to the time of
their (properly labeled) transmission.
o A protocol SHOULD NOT forbid use of U+FEFF as a signature for
those textual protocol elements for which the protocol does not
provide character encoding identification mechanisms, when a ban
would be unenforceable, or when it is expected that
implementations of the protocol will not be in a position to
always use the mechanisms properly. The latter two cases are
likely to occur with larger protocol elements such as MIME
entities, especially when implementations of the protocol will
obtain such entities from file systems, from protocols that do not
have encoding identification mechanisms for payloads (such as FTP)
or from other protocols that do not guarantee proper
identification of character encoding (such as HTTP).
When a protocol forbids use of U+FEFF as a signature for a certain
protocol element, then any initial U+FEFF in that protocol element
MUST be interpreted as a "ZERO WIDTH NO-BREAK SPACE". When a
protocol does NOT forbid use of U+FEFF as a signature for a certain
protocol element, then implementations SHOULD be prepared to handle a
signature in that element and react appropriately: using the
signature to identify the character encoding as necessary and
stripping or ignoring the signature as appropriate.
7. Examples
The character sequence U+0041 U+2262 U+0391 U+002E "A<NOT IDENTICAL
TO><ALPHA>." is encoded in UTF-8 as follows:
--+--------+-----+--
41 E2 89 A2 CE 91 2E
--+--------+-----+--
The character sequence U+D55C U+AD6D U+C5B4 (Korean "hangugeo",
meaning "the Korean language") is encoded in UTF-8 as follows:
--------+--------+--------
ED 95 9C EA B5 AD EC 96 B4
--------+--------+--------
The character sequence U+65E5 U+672C U+8A9E (Japanese "nihongo",
meaning "the Japanese language") is encoded in UTF-8 as follows:
--------+--------+--------
E6 97 A5 E6 9C AC E8 AA 9E
--------+--------+--------
Yergeau Standards Track [Page 8]
^L
RFC 3629 UTF-8 November 2003
The character U+233B4 (a Chinese character meaning 'stump of tree'),
prepended with a UTF-8 BOM, is encoded in UTF-8 as follows:
--------+-----------
EF BB BF F0 A3 8E B4
--------+-----------
8. MIME registration
This memo serves as the basis for registration of the MIME charset
parameter for UTF-8, according to [RFC2978]. The charset parameter
value is "UTF-8". This string labels media types containing text
consisting of characters from the repertoire of ISO/IEC 10646
including all amendments at least up to amendment 5 of the 1993
edition (Korean block), encoded to a sequence of octets using the
encoding scheme outlined above. UTF-8 is suitable for use in MIME
content types under the "text" top-level type.
It is noteworthy that the label "UTF-8" does not contain a version
identification, referring generically to ISO/IEC 10646. This is
intentional, the rationale being as follows:
A MIME charset label is designed to give just the information needed
to interpret a sequence of bytes received on the wire into a sequence
of characters, nothing more (see [RFC2045], section 2.2). As long as
a character set standard does not change incompatibly, version
numbers serve no purpose, because one gains nothing by learning from
the tag that newly assigned characters may be received that one
doesn't know about. The tag itself doesn't teach anything about the
new characters, which are going to be received anyway.
Hence, as long as the standards evolve compatibly, the apparent
advantage of having labels that identify the versions is only that,
apparent. But there is a disadvantage to such version-dependent
labels: when an older application receives data accompanied by a
newer, unknown label, it may fail to recognize the label and be
completely unable to deal with the data, whereas a generic, known
label would have triggered mostly correct processing of the data,
which may well not contain any new characters.
Now the "Korean mess" (ISO/IEC 10646 amendment 5) is an incompatible
change, in principle contradicting the appropriateness of a version
independent MIME charset label as described above. But the
compatibility problem can only appear with data containing Korean
Hangul characters encoded according to Unicode 1.1 (or equivalently
ISO/IEC 10646 before amendment 5), and there is arguably no such data
to worry about, this being the very reason the incompatible change
was deemed acceptable.
Yergeau Standards Track [Page 9]
^L
RFC 3629 UTF-8 November 2003
In practice, then, a version-independent label is warranted, provided
the label is understood to refer to all versions after Amendment 5,
and provided no incompatible change actually occurs. Should
incompatible changes occur in a later version of ISO/IEC 10646, the
MIME charset label defined here will stay aligned with the previous
version until and unless the IETF specifically decides otherwise.
9. IANA Considerations
The entry for UTF-8 in the IANA charset registry has been updated to
point to this memo.
10. Security Considerations
Implementers of UTF-8 need to consider the security aspects of how
they handle illegal UTF-8 sequences. It is conceivable that in some
circumstances an attacker would be able to exploit an incautious
UTF-8 parser by sending it an octet sequence that is not permitted by
the UTF-8 syntax.
A particularly subtle form of this attack can be carried out against
a parser which performs security-critical validity checks against the
UTF-8 encoded form of its input, but interprets certain illegal octet
sequences as characters. For example, a parser might prohibit the
NUL character when encoded as the single-octet sequence 00, but
erroneously allow the illegal two-octet sequence C0 80 and interpret
it as a NUL character. Another example might be a parser which
prohibits the octet sequence 2F 2E 2E 2F ("/../"), yet permits the
illegal octet sequence 2F C0 AE 2E 2F. This last exploit has
actually been used in a widespread virus attacking Web servers in
2001; thus, the security threat is very real.
Another security issue occurs when encoding to UTF-8: the ISO/IEC
10646 description of UTF-8 allows encoding character numbers up to
U+7FFFFFFF, yielding sequences of up to 6 bytes. There is therefore
a risk of buffer overflow if the range of character numbers is not
explicitly limited to U+10FFFF or if buffer sizing doesn't take into
account the possibility of 5- and 6-byte sequences.
Security may also be impacted by a characteristic of several
character encodings, including UTF-8: the "same thing" (as far as a
user can tell) can be represented by several distinct character
sequences. For instance, an e with acute accent can be represented
by the precomposed U+00E9 E ACUTE character or by the canonically
equivalent sequence U+0065 U+0301 (E + COMBINING ACUTE). Even though
UTF-8 provides a single byte sequence for each character sequence,
the existence of multiple character sequences for "the same thing"
may have security consequences whenever string matching, indexing,
Yergeau Standards Track [Page 10]
^L
RFC 3629 UTF-8 November 2003
searching, sorting, regular expression matching and selection are
involved. An example would be string matching of an identifier
appearing in a credential and in access control list entries. This
issue is amenable to solutions based on Unicode Normalization Forms,
see [UAX15].
11. Acknowledgements
The following have participated in the drafting and discussion of
this memo: James E. Agenbroad, Harald Alvestrand, Andries Brouwer,
Mark Davis, Martin J. Duerst, Patrick Faltstrom, Ned Freed, David
Goldsmith, Tony Hansen, Edwin F. Hart, Paul Hoffman, David Hopwood,
Simon Josefsson, Kent Karlsson, Dan Kohn, Markus Kuhn, Michael Kung,
Alain LaBonte, Ira McDonald, Alexey Melnikov, MURATA Makoto, John
Gardiner Myers, Chris Newman, Dan Oscarsson, Roozbeh Pournader,
Murray Sargent, Markus Scherer, Keld Simonsen, Arnold Winkler,
Kenneth Whistler and Misha Wolf.
12. Changes from RFC 2279
o Restricted the range of characters to 0000-10FFFF (the UTF-16
accessible range).
o Made Unicode the source of the normative definition of UTF-8,
keeping ISO/IEC 10646 as the reference for characters.
o Straightened out terminology. UTF-8 now described in terms of an
encoding form of the character number. UCS-2 and UCS-4 almost
disappeared.
o Turned the note warning against decoding of invalid sequences into
a normative MUST NOT.
o Added a new section about the UTF-8 BOM, with advice for
protocols.
o Removed suggested UNICODE-1-1-UTF-8 MIME charset registration.
o Added an ABNF syntax for valid UTF-8 octet sequences
o Expanded Security Considerations section, in particular impact of
Unicode normalization
Yergeau Standards Track [Page 11]
^L
RFC 3629 UTF-8 November 2003
13. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[ISO.10646] International Organization for Standardization,
"Information Technology - Universal Multiple-octet coded
Character Set (UCS)", ISO/IEC Standard 10646, comprised
of ISO/IEC 10646-1:2000, "Information technology --
Universal Multiple-Octet Coded Character Set (UCS) --
Part 1: Architecture and Basic Multilingual Plane",
ISO/IEC 10646-2:2001, "Information technology --
Universal Multiple-Octet Coded Character Set (UCS) --
Part 2: Supplementary Planes" and ISO/IEC 10646-
1:2000/Amd 1:2002, "Mathematical symbols and other
characters".
[UNICODE] The Unicode Consortium, "The Unicode Standard -- Version
4.0", defined by The Unicode Standard, Version 4.0
(Boston, MA, Addison-Wesley, 2003. ISBN 0-321-18578-1),
April 2003, <http://www.unicode.org/unicode/standard/
versions/enumeratedversions.html#Unicode_4_0_0>.
14. Informative References
[CESU-8] Phipps, T., "Unicode Technical Report #26: Compatibility
Encoding Scheme for UTF-16: 8-Bit (CESU-8)", UTR 26,
April 2002,
<http://www.unicode.org/unicode/reports/tr26/>.
[FSS_UTF] X/Open Company Ltd., "X/Open Preliminary Specification --
File System Safe UCS Transformation Format (FSS-UTF)",
May 1993, <http://wwwold.dkuug.dk/jtc1/sc22/wg20/docs/
N193-FSS-UTF.pdf>.
[RFC2045] Freed, N. and N. Borenstein, "Multipurpose Internet Mail
Extensions (MIME) Part One: Format of Internet Message
Bodies", RFC 2045, November 1996.
[RFC2234] Crocker, D. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", RFC 2234, November 1997.
[RFC2978] Freed, N. and J. Postel, "IANA Charset Registration
Procedures", BCP 19, RFC 2978, October 2000.
Yergeau Standards Track [Page 12]
^L
RFC 3629 UTF-8 November 2003
[UAX15] Davis, M. and M. Duerst, "Unicode Standard Annex #15:
Unicode Normalization Forms", An integral part of The
Unicode Standard, Version 4.0.0, April 2003, <http://
www.unicode.org/unicode/reports/tr15>.
[US-ASCII] American National Standards Institute, "Coded Character
Set - 7-bit American Standard Code for Information
Interchange", ANSI X3.4, 1986.
15. URIs
[1] <http://www.unicode.org/unicode/standard/policies.html>
16. Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any
intellectual property or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; neither does it represent that it
has made any effort to identify any such rights. Information on the
IETF's procedures with respect to rights in standards-track and
standards-related documentation can be found in BCP-11. Copies of
claims of rights made available for publication and any assurances of
licenses to be made available, or the result of an attempt made to
obtain a general license or permission for the use of such
proprietary rights by implementors or users of this specification can
be obtained from the IETF Secretariat.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights which may cover technology that may be required to practice
this standard. Please address the information to the IETF Executive
Director.
17. Author's Address
Francois Yergeau
Alis Technologies
100, boul. Alexis-Nihon, bureau 600
Montreal, QC H4M 2P2
Canada
Phone: +1 514 747 2547
Fax: +1 514 747 2561
EMail: fyergeau@alis.com
Yergeau Standards Track [Page 13]
^L
RFC 3629 UTF-8 November 2003
18. Full Copyright Statement
Copyright (C) The Internet Society (2003). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assignees.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement
Funding for the RFC Editor function is currently provided by the
Internet Society.
Yergeau Standards Track [Page 14]
^L
|