Skip to content

Commit befef1c

Browse files
authored
Always generate fast-mode data for NFD and NFKD tries (#7222)
The intent is to allow the normalizer to assume that these tries are fast in the baked data mode. This changeset removes the ability to generate serde-loadable data with small-mode tries for NFD and NFKD. Per previous discussion, we can add the ability to generate small tries for these back if there are users of ICU4X who really need that mode for these tries. Fixes #6836
1 parent 1b29670 commit befef1c

File tree

16 files changed

+11543
-7674
lines changed

16 files changed

+11543
-7674
lines changed

provider/data/normalizer/data/normalizer_nfd_data_v1.rs.data

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

provider/data/normalizer/data/normalizer_nfkd_data_v1.rs.data

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
normalizer/nfc/v1, <singleton>, 5332B, 5310B, dd048e68b718a993
2-
normalizer/nfd/data/v1, <singleton>, 28208B, 28144B, a37d3c274c030323
2+
normalizer/nfd/data/v1, <singleton>, 34912B, 34848B, 933447e5c056e49c
33
normalizer/nfd/supplement/v1, <singleton>, 4486B, 4403B, a587960aa6faa95c
44
normalizer/nfd/tables/v1, <singleton>, 2177B, 2133B, 8818c6d23b22796a
5-
normalizer/nfkd/data/v1, <singleton>, 43400B, 43336B, 855553ea574c0178
5+
normalizer/nfkd/data/v1, <singleton>, 51488B, 51424B, a5a4848d32dd13f8
66
normalizer/nfkd/tables/v1, <singleton>, 5865B, 5821B, 7f50ce23df00365d
77
normalizer/uts46/data/v1, <singleton>, 56552B, 56493B, 2f3422c2c9cc708b

provider/data/normalizer/stubdata/normalizer_nfd_data_v1.rs.data

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

provider/data/normalizer/stubdata/normalizer_nfkd_data_v1.rs.data

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

provider/icu4x-datagen/src/main.rs

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -142,7 +142,8 @@ struct Cli {
142142
#[arg(
143143
help = "Whether to optimize CodePointTrie data structures for size (\"small\") or speed (\"fast\").\n\
144144
Using \"fast\" mode increases performance of CJK text processing and segmentation. For more\n\
145-
information, see the TrieType enum."
145+
information, see the TrieType enum. The tries for the core (UAX #15 but not UAX #46)\n\
146+
normalization forms use the fast trie type regardless of this setting."
146147
)]
147148
#[cfg(feature = "provider")]
148149
trie_type: TrieType,

0 commit comments

Comments
 (0)