Skip to content

Commit f9935c0

Browse files
CodemodService Botfacebook-github-bot
CodemodService Bot
authored andcommitted
Fix CQS signal. Id] 10393698 -- performance-faster-string-find in fbcode/pytorch/tokenizers/src (#44)
Summary: Pull Request resolved: #44 Reviewed By: dtolnay Differential Revision: D72950899
1 parent f52b18b commit f9935c0

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

src/tiktoken.cpp

+1-1
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ static Result<std::pair<std::string, uint64_t>> _parse(
6767
// Tiktoken format
6868
// https://github.com/openai/tiktoken/blob/main/tiktoken/load.py#L140 <base64
6969
// encoded token str> <rank>
70-
auto pos = line.find(" ");
70+
auto pos = line.find(' ');
7171
TK_CHECK_OR_RETURN_ERROR(
7272
pos != std::string::npos,
7373
ParseFailure,

0 commit comments

Comments
 (0)