|
| 1 | +# Notes |
| 2 | + |
| 3 | +## Todos |
| 4 | + |
| 5 | + |
| 6 | +## Terminology |
| 7 | + |
| 8 | +Use Upcase, Downcase AND Titlecase (!) |
| 9 | + |
| 10 | +- Example: Ö -> Upcase: OE, Downcase: oe, Titlecase: Oe (!) |
| 11 | +- Example: Æ -> Upcase: AE, Downcase: ae, Titlecase: Ae (!) |
| 12 | + |
| 13 | + |
| 14 | + |
| 15 | +## Libraries |
| 16 | + |
| 17 | +**Ruby** |
| 18 | + |
| 19 | +- <https://github.com/SixArm/sixarm_ruby_unaccent> - Replace a string's accent characters with ASCII characters. Based on Perl Text::Unaccent from CPAN. |
| 20 | + |
| 21 | +- <https://github.com/fractalsoft/diacritics> - support downcase, upcase and permanent link with diacritical characters |
| 22 | + |
| 23 | +**Perl** |
| 24 | + |
| 25 | +- <https://metacpan.org/pod/Unicode::Diacritic::Strip> - strip diacritics from Unicode text |
| 26 | + |
| 27 | +**JavaScript** |
| 28 | + |
| 29 | +- <https://github.com/dundalek/latinize> - convert accents (diacritics) from strings to latin characters |
| 30 | + |
| 31 | +- <https://github.com/tyxla/remove-accents> - removes the accents from a string, converting them to their corresponding non-accented ascii characters |
| 32 | + |
| 33 | +**PostgreSQL** |
| 34 | + |
| 35 | +- <https://www.postgresql.org/docs/current/unaccent.html> - unaccent is a text search dictionary that removes accents (diacritic signs) from lexemes |
| 36 | + |
| 37 | + |
| 38 | +## Links |
| 39 | + |
| 40 | +**Unicode w/ Ruby - Ruby ♡ Unicode** |
| 41 | + |
| 42 | +- <https://idiosyncratic-ruby.com/66-ruby-has-character> |
| 43 | + |
| 44 | +Ruby has Character - Ruby comes with good support for Unicode-related features. Read on if you want to learn more about important Unicode fundamentals and how to use them in Ruby... |
| 45 | + |
| 46 | +- <https://idiosyncratic-ruby.com/41-proper-unicoding> |
| 47 | + |
| 48 | +Proper Unicoding - Ruby's Regexp engine has a powerful feature built in: It can match for Unicode character properties. But what exactly are properties you can match for? |
| 49 | + |
| 50 | +- <https://idiosyncratic-ruby.com/30-regex-with-class> |
| 51 | + |
| 52 | +Regex with Class - Ruby's regex engine defines a lot of shortcut character classes. Besides the common meta characters (\w, etc.), there is also the POSIX style expressions and the unicode property syntax. This is an overview of all character classes |
| 53 | + |
| 54 | + |
| 55 | +**Unicode** |
| 56 | + |
| 57 | +- <https://unicode.org/reports/tr15/> - Unicode Standard Annex #15 - UNICODE NORMALIZATION FORMS |
| 58 | + |
| 59 | +**W3C** |
| 60 | + |
| 61 | +- <https://www.w3.org/TR/charmod-norm/> |
| 62 | +- <https://www.w3.org/International/wiki/Case_folding> |
| 63 | + |
| 64 | +In Western European languages, the letter 'i' (U+0069) upper cases to a dotless 'I' (U+0049). In Turkish, this letter upper cases to a dotted upper case letter 'İ' (U+0130). Similarly, 'I' (U+0049) lower cases to 'ı' (U+0131), which is a dotless lowercase letter i. |
| 65 | + |
| 66 | +**Wikipedia** |
| 67 | + |
| 68 | +- <https://en.wikipedia.org/wiki/Diacritic> |
| 69 | + |
| 70 | +**More** |
| 71 | + |
| 72 | +- [The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)](https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/) |
| 73 | +by Joel Spolsky, 2003 |
| 74 | + |
| 75 | +- [Unicode Normalization in Ruby](https://www.honeybadger.io/blog/ruby-unicode-normalization/) by Starr Horne, 2017 |
| 76 | + |
| 77 | + |
| 78 | +## Mappings |
| 79 | + |
| 80 | +Open questions ... |
| 81 | + |
| 82 | +``` |
| 83 | + Þ => TH ??? |
| 84 | + þ => th ??? |
| 85 | +``` |
| 86 | + |
| 87 | + |
| 88 | +## Alphabets |
| 89 | + |
| 90 | +Add more alphabets... why? why not? |
| 91 | + |
| 92 | + |
| 93 | +- Portuguese [Â, "abcdefghijklmnopqrstuvwxyzáâãàçéêíóôõú", "ABCDEFGHIJKLMNOPQRSTUVWXYZÁÂÃÀÇÉÊÍÓÔÕÚ"] |
| 94 | +- Russian [Щ, Ъ, Э, "абвгдеёжзийклмнопрстуфхцчшщъыьэюя", "АБВГДЕЁЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯ"] |
| 95 | +- Greek [Β, Μ, Χ, Ω, Ή, Ύ, Ώ, ΐ, ΰ, Ϊ, Ϋ] |
| 96 | +- Slovak ["aáäeéiíoóôuúyýbcčdďfghjklĺľmnňpqrŕsštťvwxzž", "AÁÄEÉIÍOÓÔUÚYÝBCČDĎFGHJKLĹĽMNŇPQRŔSŠTŤVWXZŽ"] |
| 97 | +- Italian ["aàbcdeèéfghiìíîlmnoòópqrstuùúvz", "AÀBCDEÈÉFGHIÌÍÎLMNOÒÓPQRSTUÙÚVZ"] |
| 98 | +- Romanian ["aăâbcdefghiîjklmnopqrsștțuvwxyz", "AĂÂBCDEFGHIÎJKLMNOPQRSȘTȚUVWXYZ"] |
| 99 | +- Danish [å, â, ô, Å, Â, Ô] |
| 100 | + |
| 101 | +``` |
| 102 | + def de |
| 103 | + { # German |
| 104 | + downcase: %w(ä ö ü ß), |
| 105 | + upcase: %w(Ä Ö Ü ẞ), |
| 106 | + permanent: %w(ae oe ue ss) |
| 107 | + } |
| 108 | + end |
| 109 | +
|
| 110 | + def pl |
| 111 | + { # Polish |
| 112 | + downcase: %w(ą ć ę ł ń ó ś ż ź), |
| 113 | + upcase: %w(Ą Ć Ę Ł Ń Ó Ś Ż Ź), |
| 114 | + permanent: %w(a c e l n o s z z) |
| 115 | + } |
| 116 | + end |
| 117 | +
|
| 118 | + def cs |
| 119 | + { # Czech uses acute (á é í ó ú ý), caron (č ď ě ň ř š ť ž), ring (ů) |
| 120 | + # aábcčdďeéěfghiíjklmnňoópqrřsštťuúůvwxyýzž |
| 121 | + # AÁBCČDĎEÉĚFGHIÍJKLMNŇOÓPQRŘSŠTŤUÚŮVWXYÝZŽ |
| 122 | + downcase: %w(á é í ó ú ý č ď ě ň ř š ť ů ž), |
| 123 | + upcase: %w(Á É Í Ó Ú Ý Č Ď Ě Ň Ř Š Ť Ů Ž), |
| 124 | + permanent: %w(a e i o u y c d e n r s t u z) |
| 125 | + } |
| 126 | + end |
| 127 | +
|
| 128 | + def fr |
| 129 | + { # French |
| 130 | + # abcdefghijklmnopqrstuvwxyzàâæçéèêëîïôœùûüÿ |
| 131 | + # ABCDEFGHIJKLMNOPQRSTUVWXYZÀÂÆÇÉÈÊËÎÏÔŒÙÛÜŸ |
| 132 | + downcase: %w(à â é è ë ê ï î ô ù û ü ÿ ç œ æ), |
| 133 | + upcase: %w(À Â É È Ë Ê Ï Î Ô Ù Û Ü Ÿ Ç Œ Æ), |
| 134 | + permanent: %w(a a e e e e i i o u u ue y c oe ae) |
| 135 | + } |
| 136 | + end |
| 137 | +
|
| 138 | + def it |
| 139 | + { # Italian |
| 140 | + downcase: %w(à è é ì î ò ó ù), |
| 141 | + upcase: %w(À È É Ì Î Ò Ó Ù), |
| 142 | + permanent: %w(a e e i i o o u) |
| 143 | + } |
| 144 | + end |
| 145 | +
|
| 146 | + def eo |
| 147 | + { # Esperantohas the symbols ŭ, ĉ, ĝ, ĥ, ĵ and ŝ |
| 148 | + downcase: %w(ĉ ĝ ĥ ĵ ŝ ŭ), |
| 149 | + upcase: %w(Ĉ Ĝ Ĥ Ĵ Ŝ Ŭ), |
| 150 | + permanent: %w(c g h j s u) |
| 151 | + } |
| 152 | + end |
| 153 | +
|
| 154 | + def is |
| 155 | + { # Iceland |
| 156 | + downcase: %w(ð þ), |
| 157 | + upcase: %w(Ð Þ), |
| 158 | + permanent: %w(d p) |
| 159 | + } |
| 160 | + end |
| 161 | +
|
| 162 | + def pt |
| 163 | + { # Portugal uses á, â, ã, à, ç, é, ê, í, ó, ô, õ and ú |
| 164 | + downcase: %w(ã ç), |
| 165 | + upcase: %w(Ã Ç), |
| 166 | + permanent: %w(a c) |
| 167 | + } |
| 168 | + end |
| 169 | +
|
| 170 | + def sp |
| 171 | + { # Spanish |
| 172 | + downcase: ['ñ', 'õ', '¿', '¡'], |
| 173 | + upcase: ['Ñ', 'Õ', '¿', '¡'], |
| 174 | + permanent: ['n', 'o', '', ''] |
| 175 | + } |
| 176 | + end |
| 177 | +
|
| 178 | + def hu |
| 179 | + { # Hungarian |
| 180 | + downcase: %w(ő), |
| 181 | + upcase: %w(Ő), |
| 182 | + permanent: %w(oe) |
| 183 | + } |
| 184 | + end |
| 185 | +
|
| 186 | + def nn |
| 187 | + { # Norwegian |
| 188 | + downcase: %w(æ å), |
| 189 | + upcase: %w(Æ Å), |
| 190 | + permanent: %w(ae a) |
| 191 | + } |
| 192 | + end |
| 193 | +``` |
0 commit comments