0001
0002
0003
0004
0005
0006
0007
0008
0009
0010
0011
0012
0013
0014
0015
0016
0017
0018
0019
0020
0021
0022
0023
0024
0025
0026
0027
0028
0029
0030
0031
0032
0033
0034
0035
0036
0037
0038
0039
0040
0041
0042
0043
0044
0045
0046
0047
0048
0049
0050
0051
0052
0053
0054
0055
0056
0057
0058
0059
0060
0061
0062
0063
0064
0065
0066
0067
0068
0069
0070
0071
0072
0073
0074
0075
0076
0077
0078
0079
0080
0081
0082
0083
0084
0085
0086
0087
0088
0089
0090
0091
0092
0093
0094
0095
0096
0097
0098
0099
0100
0101
0102
0103
0104
0105
0106
0107
0108
0109
0110
0111
0112
0113
0114
0115
0116
0117
0118
0119
0120
0121
0122
0123
0124
0125
0126
0127
0128
0129
0130
0131
0132
0133
0134
0135
0136
0137
0138
0139
0140
0141
0142
0143
0144
0145
0146
0147
0148
0149
0150
0151
0152
0153
0154
0155
0156
0157
0158
0159
0160
0161
0162
0163
0164
0165
0166
0167
0168
0169
0170
0171
0172
0173
0174
0175
0176
0177
0178
0179
0180
0181
0182
0183
0184
0185
0186
0187
0188
0189
0190
0191
0192
0193
0194
0195
0196
0197
0198
0199
0200
0201
0202
0203
0204
0205
0206
0207
0208
0209
0210
0211
0212
0213
0214
0215
0216
0217
0218
0219
0220
0221
0222
0223
0224
0225
0226
0227
0228
0229
0230
0231
0232
0233
0234
0235
0236
0237
0238
0239
0240
0241
0242
0243
0244
0245
0246
0247
0248
0249
0250
0251
0252
0253
0254
0255
0256
0257
0258
0259
0260
0261
0262
0263
0264
0265
0266
0267
0268
0269
0270
0271
0272
0273
0274
0275
0276
0277
0278
0279
0280
0281
0282
0283
0284
0285
0286
0287
0288
0289
0290
0291
0292
0293
0294
0295
0296
0297
0298
0299
0300
0301
0302
0303
0304
0305
0306
0307
0308
0309
0310
0311
0312
0313
0314
0315
0316
0317
0318
0319
0320
0321
0322
0323
0324
0325
0326
0327
0328
0329
0330
0331
0332
0333
0334
0335
0336
0337
0338
0339
0340
0341
0342
0343
0344
0345
0346
0347
0348
0349
0350
0351
0352
0353
0354
0355
0356
0357
0358
0359
0360
0361
0362
0363
0364
0365
0366
0367
0368
0369
0370
0371
0372
0373
0374
0375
0376
0377
0378
0379
0380
0381
0382
0383
0384
0385
0386
0387
0388
0389
0390
0391
0392
0393
0394
0395
0396
0397
0398
0399
0400
0401
0402
0403
0404
0405
0406
0407
0408
0409
0410
0411
0412
0413
0414
0415
0416
0417
0418
0419
0420
0421
0422
0423
0424
0425
0426
0427
0428
0429
0430
0431
0432
0433
0434
0435
0436
0437
0438
0439
0440
0441
0442
0443
0444
0445
0446
0447
0448
0449
0450
0451
0452
0453
0454
0455
0456
Network Working Group                                       F. Yergeau
Request for Comments: 2279                           Alis Technologies
Obsoletes: 2044                                           January 1998
Category: Standards Track

              UTF-8, a transformation format of ISO 10646

Status of this Memo

   This document specifies an Internet standards track protocol for the
   Internet community, and requests discussion and suggestions for
   improvements.  Please refer to the current edition of the "Internet
   Official Protocol Standards" (STD 1) for the standardization state
   and status of this protocol.  Distribution of this memo is unlimited.

Copyright Notice

   Copyright (C) The Internet Society (1998).  All Rights Reserved.

Abstract

   ISO/IEC 10646-1 defines a multi-octet character set called the
   Universal Character Set (UCS) which encompasses most of the world's
   writing systems. Multi-octet characters, however, are not compatible
   with many current applications and protocols, and this has led to the
   development of a few so-called UCS transformation formats (UTF), each
   with different characteristics.  UTF-8, the object of this memo, has
   the characteristic of preserving the full US-ASCII range, providing
   compatibility with file systems, parsers and other software that rely
   on US-ASCII values but are transparent to other values. This memo
   updates and replaces RFC 2044, in particular addressing the question
   of versions of the relevant standards.

1.  Introduction

   ISO/IEC 10646-1 [ISO-10646] defines a multi-octet character set
   called the Universal Character Set (UCS), which encompasses most of
   the world's writing systems.  Two multi-octet encodings are defined,
   a four-octet per character encoding called UCS-4 and a two-octet per
   character encoding called UCS-2, able to address only the first 64K
   characters of the UCS (the Basic Multilingual Plane, BMP), outside of
   which there are currently no assignments.

   It is noteworthy that the same set of characters is defined by the
   Unicode standard [UNICODE], which further defines additional
   character properties and other application details of great interest
   to implementors, but does not have the UCS-4 encoding.  Up to the

Yergeau                     Standards Track                     [Page 1]

RFC 2279                         UTF-8                      January 1998

   present time, changes in Unicode and amendments to ISO/IEC 10646 have
   tracked each other, so that the character repertoires and code point
   assignments have remained in sync.  The relevant standardization
   committees have committed to maintain this very useful synchronism.

   The UCS-2 and UCS-4 encodings, however, are hard to use in many
   current applications and protocols that assume 8 or even 7 bit
   characters.  Even newer systems able to deal with 16 bit characters
   cannot process UCS-4 data. This situation has led to the development
   of so-called UCS transformation formats (UTF), each with different
   characteristics.

   UTF-1 has only historical interest, having been removed from ISO/IEC
   10646.  UTF-7 has the quality of encoding the full BMP repertoire
   using only octets with the high-order bit clear (7 bit US-ASCII
   values, [US-ASCII]), and is thus deemed a mail-safe encoding
   ([RFC2152]).  UTF-8, the object of this memo, uses all bits of an
   octet, but has the quality of preserving the full US-ASCII range:
   US-ASCII characters are encoded in one octet having the normal US-
   ASCII value, and any octet with such a value can only stand for an
   US-ASCII character, and nothing else.

   UTF-16 is a scheme for transforming a subset of the UCS-4 repertoire
   into pairs of UCS-2 values from a reserved range.  UTF-16 impacts
   UTF-8 in that UCS-2 values from the reserved range must be treated
   specially in the UTF-8 transformation.

   UTF-8 encodes UCS-2 or UCS-4 characters as a varying number of
   octets, where the number of octets, and the value of each, depend on
   the integer value assigned to the character in ISO/IEC 10646.  This
   transformation format has the following characteristics (all values
   are in hexadecimal):

   -  Character values from 0000 0000 to 0000 007F (US-ASCII repertoire)
      correspond to octets 00 to 7F (7 bit US-ASCII values). A direct
      consequence is that a plain ASCII string is also a valid UTF-8
      string.

   -  US-ASCII values do not appear otherwise in a UTF-8 encoded
      character stream.  This provides compatibility with file systems
      or other software (e.g. the printf() function in C libraries) that
      parse based on US-ASCII values but are transparent to other
      values.

   -  Round-trip conversion is easy between UTF-8 and either of UCS-4,
      UCS-2.

Yergeau                     Standards Track                     [Page 2]

RFC 2279                         UTF-8                      January 1998

   -  The first octet of a multi-octet sequence indicates the number of
      octets in the sequence.

   -  The octet values FE and FF never appear.

   -  Character boundaries are easily found from anywhere in an octet
      stream.

   -  The lexicographic sorting order of UCS-4 strings is preserved.  Of
      course this is of limited interest since the sort order is not
      culturally valid in either case.

   -  The Boyer-Moore fast search algorithm can be used with UTF-8 data.

   -  UTF-8 strings can be fairly reliably recognized as such by a
      simple algorithm, i.e. the probability that a string of characters
      in any other encoding appears as valid UTF-8 is low, diminishing
      with increasing string length.

   UTF-8 was originally a project of the X/Open Joint
   Internationalization Group XOJIG with the objective to specify a File
   System Safe UCS Transformation Format [FSS-UTF] that is compatible
   with UNIX systems, supporting multilingual text in a single encoding.
   The original authors were Gary Miller, Greger Leijonhufvud and John
   Entenmann.  Later, Ken Thompson and Rob Pike did significant work for
   the formal UTF-8.

   A description can also be found in Unicode Technical Report #4 and in
   the Unicode Standard, version 2.0 [UNICODE].  The definitive
   reference, including provisions for UTF-16 data within UTF-8, is
   Annex R of ISO/IEC 10646-1 [ISO-10646].

2.  UTF-8 definition

   In UTF-8, characters are encoded using sequences of 1 to 6 octets.
   The only octet of a "sequence" of one has the higher-order bit set to
   0, the remaining 7 bits being used to encode the character value. In
   a sequence of n octets, n>1, the initial octet has the n higher-order
   bits set to 1, followed by a bit set to 0.  The remaining bit(s) of
   that octet contain bits from the value of the character to be
   encoded.  The following octet(s) all have the higher-order bit set to
   1 and the following bit set to 0, leaving 6 bits in each to contain
   bits from the character to be encoded.

   The table below summarizes the format of these different octet types.
   The letter x indicates bits available for encoding bits of the UCS-4
   character value.

Yergeau                     Standards Track                     [Page 3]

RFC 2279                         UTF-8                      January 1998

   UCS-4 range (hex.)           UTF-8 octet sequence (binary)
   0000 0000-0000 007F   0xxxxxxx
   0000 0080-0000 07FF   110xxxxx 10xxxxxx
   0000 0800-0000 FFFF   1110xxxx 10xxxxxx 10xxxxxx

   0001 0000-001F FFFF   11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
   0020 0000-03FF FFFF   111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
   0400 0000-7FFF FFFF   1111110x 10xxxxxx ... 10xxxxxx

   Encoding from UCS-4 to UTF-8 proceeds as follows:

   1) Determine the number of octets required from the character value
      and the first column of the table above.  It is important to note
      that the rows of the table are mutually exclusive, i.e. there is
      only one valid way to encode a given UCS-4 character.

   2) Prepare the high-order bits of the octets as per the second column
      of the table.

   3) Fill in the bits marked x from the bits of the character value,
      starting from the lower-order bits of the character value and
      putting them first in the last octet of the sequence, then the
      next to last, etc. until all x bits are filled in.

      The algorithm for encoding UCS-2 (or Unicode) to UTF-8 can be
      obtained from the above, in principle, by simply extending each
      UCS-2 character with two zero-valued octets.  However, pairs of
      UCS-2 values between D800 and DFFF (surrogate pairs in Unicode
      parlance), being actually UCS-4 characters transformed through
      UTF-16, need special treatment: the UTF-16 transformation must be
      undone, yielding a UCS-4 character that is then transformed as
      above.

      Decoding from UTF-8 to UCS-4 proceeds as follows:

   1) Initialize the 4 octets of the UCS-4 character with all bits set
      to 0.

   2) Determine which bits encode the character value from the number of
      octets in the sequence and the second column of the table above
      (the bits marked x).

   3) Distribute the bits from the sequence to the UCS-4 character,
      first the lower-order bits from the last octet of the sequence and
      proceeding to the left until no x bits are left.

      If the UTF-8 sequence is no more than three octets long, decoding
      can proceed directly to UCS-2.

Yergeau                     Standards Track                     [Page 4]

RFC 2279                         UTF-8                      January 1998

        NOTE -- actual implementations of the decoding algorithm above
        should protect against decoding invalid sequences.  For
        instance, a naive implementation may (wrongly) decode the
        invalid UTF-8 sequence C0 80 into the character U+0000, which
        may have security consequences and/or cause other problems.  See
        the Security Considerations section below.

   A more detailed algorithm and formulae can be found in [FSS_UTF],
   [UNICODE] or Annex R to [ISO-10646].

3.  Versions of the standards

   ISO/IEC 10646 is updated from time to time by published amendments;
   similarly, different versions of the Unicode standard exist: 1.0, 1.1
   and 2.0 as of this writing.  Each new version obsoletes and replaces
   the previous one, but implementations, and more significantly data,
   are not updated instantly.

   In general, the changes amount to adding new characters, which does
   not pose particular problems with old data.  Amendment 5 to ISO/IEC
   10646, however, has moved and expanded the Korean Hangul block,
   thereby making any previous data containing Hangul characters invalid
   under the new version.  Unicode 2.0 has the same difference from
   Unicode 1.1. The official justification for allowing such an
   incompatible change was that no implementations and no data
   containing Hangul existed, a statement that is likely to be true but
   remains unprovable.  The incident has been dubbed the "Korean mess",
   and the relevant committees have pledged to never, ever again make
   such an incompatible change.

   New versions, and in particular any incompatible changes, have q
   conseuences regarding MIME character encoding labels, to be discussed
   in section 5.

4.  Examples

   The UCS-2 sequence "A<NOT IDENTICAL TO><ALPHA>." (0041, 2262, 0391,
   002E) may be encoded in UTF-8 as follows:

   41 E2 89 A2 CE 91 2E

   The UCS-2 sequence representing the Hangul characters for the Korean
   word "hangugo" (D55C, AD6D, C5B4) may be encoded as follows:

   ED 95 9C EA B5 AD EC 96 B4

Yergeau                     Standards Track                     [Page 5]

RFC 2279                         UTF-8                      January 1998

   The UCS-2 sequence representing the Han characters for the Japanese
   word "nihongo" (65E5, 672C, 8A9E) may be encoded as follows:

   E6 97 A5 E6 9C AC E8 AA 9E

5.  MIME registration

   This memo is meant to serve as the basis for registration of a MIME
   character set parameter (charset) [CHARSET-REG].  The proposed
   charset parameter value is "UTF-8".  This string labels media types
   containing text consisting of characters from the repertoire of
   ISO/IEC 10646 including all amendments at least up to amendment 5
   (Korean block), encoded to a sequence of octets using the encoding
   scheme outlined above.  UTF-8 is suitable for use in MIME content
   types under the "text" top-level type.

   It is noteworthy that the label "UTF-8" does not contain a version
   identification, referring generically to ISO/IEC 10646.  This is
   intentional, the rationale being as follows:

   A MIME charset label is designed to give just the information needed
   to interpret a sequence of bytes received on the wire into a sequence
   of characters, nothing more (see RFC 2045, section 2.2, in [MIME]).
   As long as a character set standard does not change incompatibly,
   version numbers serve no purpose, because one gains nothing by
   learning from the tag that newly assigned characters may be received
   that one doesn't know about.  The tag itself doesn't teach anything
   about the new characters, which are going to be received anyway.

   Hence, as long as the standards evolve compatibly, the apparent
   advantage of having labels that identify the versions is only that,
   apparent.  But there is a disadvantage to such version-dependent
   labels: when an older application receives data accompanied by a
   newer, unknown label, it may fail to recognize the label and be
   completely unable to deal with the data, whereas a generic, known
   label would have triggered mostly correct processing of the data,
   which may well not contain any new characters.

   Now the "Korean mess" (ISO/IEC 10646 amendment 5) is an incompatible
   change, in principle contradicting the appropriateness of a version
   independent MIME charset label as described above.  But the
   compatibility problem can only appear with data containing Korean
   Hangul characters encoded according to Unicode 1.1 (or equivalently
   ISO/IEC 10646 before amendment 5), and there is arguably no such data
   to worry about, this being the very reason the incompatible change
   was deemed acceptable.

Yergeau                     Standards Track                     [Page 6]

RFC 2279                         UTF-8                      January 1998

   In practice, then, a version-independent label is warranted, provided
   the label is understood to refer to all versions after Amendment 5,
   and provided no incompatible change actually occurs.  Should
   incompatible changes occur in a later version of ISO/IEC 10646, the
   MIME charset label defined here will stay aligned with the previous
   version until and unless the IETF specifically decides otherwise.

   It is also proposed to register the charset parameter value
   "UNICODE-1-1-UTF-8", for the exclusive purpose of labelling text data
   containing Hangul syllables encoded to UTF-8 without taking into
   account Amendment 5 of ISO/IEC 10646 (i.e. using the pre-amendment 5
   code point assignments).  Any other UTF-8 data SHOULD NOT use this
   label, in particular data not containing any Hangul syllables, and it
   is felt important to strongly recommend against creating any new
   Hangul-containing data without taking Amendment 5 of ISO/IEC 10646
   into account.

6.  Security Considerations

   Implementors of UTF-8 need to consider the security aspects of how
   they handle illegal UTF-8 sequences.  It is conceivable that in some
   circumstances an attacker would be able to exploit an incautious
   UTF-8 parser by sending it an octet sequence that is not permitted by
   the UTF-8 syntax.

   A particularly subtle form of this attack could be carried out
   against a parser which performs security-critical validity checks
   against the UTF-8 encoded form of its input, but interprets certain
   illegal octet sequences as characters.  For example, a parser might
   prohibit the NUL character when encoded as the single-octet sequence
   00, but allow the illegal two-octet sequence C0 80 and interpret it
   as a NUL character.  Another example might be a parser which
   prohibits the octet sequence 2F 2E 2E 2F ("/../"), yet permits the
   illegal octet sequence 2F C0 AE 2E 2F.

Acknowledgments

   The following have participated in the drafting and discussion of
   this memo:

   James E. Agenbroad    Andries Brouwer
   Martin J. D|rst       Ned Freed
   David Goldsmith       Edwin F. Hart
   Kent Karlsson         Markus Kuhn
   Michael Kung          Alain LaBonte
   John Gardiner Myers   Murray Sargent
   Keld Simonsen         Arnold Winkler

Yergeau                     Standards Track                     [Page 7]

RFC 2279                         UTF-8                      January 1998

Bibliography

   [CHARSET-REG]  Freed, N., and J. Postel, "IANA Charset Registration
                  Procedures", BCP 19, RFC 2278, January 1998.

   [FSS_UTF]      X/Open CAE Specification C501 ISBN 1-85912-082-2 28cm.
                  22p. pbk. 172g.  4/95, X/Open Company Ltd., "File
                  System Safe UCS Transformation Format (FSS_UTF)",
                  X/Open Preleminary Specification, Document Number
                  P316.  Also published in Unicode Technical Report #4.

   [ISO-10646]    ISO/IEC 10646-1:1993. International Standard --
                  Information technology -- Universal Multiple-Octet
                  Coded Character Set (UCS) -- Part 1: Architecture and
                  Basic Multilingual Plane.  Five amendments and a
                  technical corrigendum have been published up to now.
                  UTF-8 is described in Annex R, published as Amendment
                  2.  UTF-16 is described in Annex Q, published as
                  Amendment 1. 17 other amendments are currently at
                  various stages of standardization.

   [MIME]         Freed, N., and N. Borenstein, "Multipurpose Internet
                  Mail Extensions (MIME) Part One:  Format of Internet
                  Message Bodies", RFC 2045.  N. Freed, N. Borenstein,
                  "Multipurpose Internet Mail Extensions (MIME) Part
                  Two:  Media Types", RFC 2046.  K. Moore, "MIME
                  (Multipurpose Internet Mail Extensions) Part Three:
                  Message Header Extensions for Non-ASCII Text", RFC
                  2047.  N.  Freed, J. Klensin, J. Postel, "Multipurpose
                  Internet Mail Extensions (MIME) Part Four:
                  Registration Procedures", RFC 2048.  N. Freed, N.
                  Borenstein, " Multipurpose Internet Mail Extensions
                  (MIME) Part Five: Conformance Criteria and Examples",
                  RFC 2049.  All November 1996.

   [RFC2152]      Goldsmith, D., and M. Davis, "UTF-7: A Mail-safe
                  Transformation Format of Unicode", RFC 1642, Taligent
                  inc., May 1997. (Obsoletes RFC1642)

   [UNICODE]      The Unicode Consortium, "The Unicode Standard --
                  Version 2.0", Addison-Wesley, 1996.

   [US-ASCII]     Coded Character Set--7-bit American Standard Code for
                  Information Interchange, ANSI X3.4-1986.

Yergeau                     Standards Track                     [Page 8]

RFC 2279                         UTF-8                      January 1998

Author's Address

   Francois Yergeau
   Alis Technologies
   100, boul. Alexis-Nihon
   Suite 600
   Montreal  QC  H4M 2P2
   Canada

   Phone: +1 (514) 747-2547
   Fax:   +1 (514) 747-2561
   EMail: fyergeau@alis.com

Yergeau                     Standards Track                     [Page 9]

RFC 2279                         UTF-8                      January 1998

Full Copyright Statement

   Copyright (C) The Internet Society (1998).  All Rights Reserved.

   This document and translations of it may be copied and furnished to
   others, and derivative works that comment on or otherwise explain it
   or assist in its implementation may be prepared, copied, published
   and distributed, in whole or in part, without restriction of any
   kind, provided that the above copyright notice and this paragraph are
   included on all such copies and derivative works.  However, this
   document itself may not be modified in any way, such as by removing
   the copyright notice or references to the Internet Society or other
   Internet organizations, except as needed for the purpose of
   developing Internet standards in which case the procedures for
   copyrights defined in the Internet Standards process must be
   followed, or as required to translate it into languages other than
   English.

   The limited permissions granted above are perpetual and will not be
   revoked by the Internet Society or its successors or assigns.

   This document and the information contained herein is provided on an
   "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
   TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
   BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
   HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
   MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Yergeau                     Standards Track                    [Page 10]

Search by Algolia