Add benchmark for deserializing large added vocab + optimizations - #1782

Closed

ArthurZucker wants to merge 20 commits into

huggingface:mainfrom

ArthurZucker:codex/optimize-addedvocabulary-deserialization-and-add-token-metho

Commits on May 27, 2025

Add benchmark for deserializing large added vocab
ArthurZucker
committed
7cf18aa
revert dumb stuff, isolate changes
ArthurZucker
committed
47c2e9f
try to only normalize once
ArthurZucker
committed
a8f6a71
Merge branch 'main' into codex/optimize-addedvocabulary-deserialization-and-add-token-metho
ArthurZucker
authored
35a9427
small improvement?
ArthurZucker
committed
8ba1d20
Merge branch 'codex/optimize-addedvocabulary-deserialization-and-add-token-metho' of github.com:ArthurZucker/tokenizers into codex/optimize-addedvocabulary-deserialization-and-add-token-metho
ArthurZucker
committed
a71555c
some updates
ArthurZucker
committed
6714ceb
nit
ArthurZucker
committed
e07ecfc
fmt
ArthurZucker
committed
8849d71
normalized string are a fucking waste of time when you just want to add tokens to the vocab man....
ArthurZucker
committed
5da668a
more attempts
ArthurZucker
committed
8e7ce86
works
ArthurZucker
committed
948eead
let's fucking go, parity
ArthurZucker
committed
43cef92
update
ArthurZucker
committed
ae8a7b4
hahahhahaha
ArthurZucker
committed
44beeb7
revert changes that are not actually even needed
ArthurZucker
committed
e7f8954
add a python test!
ArthurZucker
committed
8d49849
use normalizer before come on
ArthurZucker
committed
d8f07fa
nit
ArthurZucker
committed
f6df603

Commits on Nov 27, 2025

Merge branch 'main' of github.com:huggingface/tokenizers into codex/optimize-addedvocabulary-deserialization-and-add-token-metho
ArthurZucker
committed
96a9563