Contents of /trunk/mkinitrd-magellan/busybox/docs/unicode_UTF-8-test.txt
Parent Directory | Revision Log
Revision 1123 -
(show annotations)
(download)
Wed Aug 18 21:56:57 2010 UTC (14 years, 1 month ago) by niro
File MIME type: text/plain
File size: 20334 byte(s)
Wed Aug 18 21:56:57 2010 UTC (14 years, 1 month ago) by niro
File MIME type: text/plain
File size: 20334 byte(s)
-updated to busybox-1.17.1
1 | UTF-8 decoder capability and stress test |
2 | ---------------------------------------- |
3 | |
4 | Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2003-02-19 |
5 | |
6 | This test file can help you examine, how your UTF-8 decoder handles |
7 | various types of correct, malformed, or otherwise interesting UTF-8 |
8 | sequences. This file is not meant to be a conformance test. It does |
9 | not prescribes any particular outcome and therefore there is no way to |
10 | "pass" or "fail" this test file, even though the texts suggests a |
11 | preferable decoder behaviour at some places. The aim is instead to |
12 | help you think about and test the behaviour of your UTF-8 on a |
13 | systematic collection of unusual inputs. Experience so far suggests |
14 | that most first-time authors of UTF-8 decoders find at least one |
15 | serious problem in their decoder by using this file. |
16 | |
17 | The test lines below cover boundary conditions, malformed UTF-8 |
18 | sequences as well as correctly encoded UTF-8 sequences of Unicode code |
19 | points that should never occur in a correct UTF-8 file. |
20 | |
21 | According to ISO 10646-1:2000, sections D.7 and 2.3c, a device |
22 | receiving UTF-8 shall interpret a "malformed sequence in the same way |
23 | that it interprets a character that is outside the adopted subset" and |
24 | "characters that are not within the adopted subset shall be indicated |
25 | to the user" by a receiving device. A quite commonly used approach in |
26 | UTF-8 decoders is to replace any malformed UTF-8 sequence by a |
27 | replacement character (U+FFFD), which looks a bit like an inverted |
28 | question mark, or a similar symbol. It might be a good idea to |
29 | visually distinguish a malformed UTF-8 sequence from a correctly |
30 | encoded Unicode character that is just not available in the current |
31 | font but otherwise fully legal, even though ISO 10646-1 doesn't |
32 | mandate this. In any case, just ignoring malformed sequences or |
33 | unavailable characters does not conform to ISO 10646, will make |
34 | debugging more difficult, and can lead to user confusion. |
35 | |
36 | Please check, whether a malformed UTF-8 sequence is (1) represented at |
37 | all, (2) represented by exactly one single replacement character (or |
38 | equivalent signal), and (3) the following quotation mark after an |
39 | illegal UTF-8 sequence is correctly displayed, i.e. proper |
40 | resynchronization takes place immageately after any malformed |
41 | sequence. This file says "THE END" in the last line, so if you don't |
42 | see that, your decoder crashed somehow before, which should always be |
43 | cause for concern. |
44 | |
45 | All lines in this file are exactly 79 characters long (plus the line |
46 | feed). In addition, all lines end with "|", except for the two test |
47 | lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls |
48 | U+0000 and U+007F. If you display this file with a fixed-width font, |
49 | these "|" characters should all line up in column 79 (right margin). |
50 | This allows you to test quickly, whether your UTF-8 decoder finds the |
51 | correct number of characters in every line, that is whether each |
52 | malformed sequences is replaced by a single replacement character. |
53 | |
54 | Note that as an alternative to the notion of malformed sequence used |
55 | here, it is also a perfectly acceptable (and in some situations even |
56 | preferable) solution to represent each individual byte of a malformed |
57 | sequence by a replacement character. If you follow this strategy in |
58 | your decoder, then please ignore the "|" column. |
59 | |
60 | |
61 | Here come the tests: | |
62 | | |
63 | 1 Some correct UTF-8 text | |
64 | | |
65 | You should see the Greek word 'kosme': "κόσμε" | |
66 | | |
67 | 2 Boundary condition test cases | |
68 | | |
69 | 2.1 First possible sequence of a certain length | |
70 | | |
71 | 2.1.1 1 byte (U-00000000): " |