[go: nahoru, domu]
Skip to content

Pull requests: huggingface/tokenizers

New pull request New
Clear current search query, filters, and sorts
11 Open 587 Closed
11 Open 587 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

C++ bindings Stale
#559 by alexeyr was closed May 7, 2024 • Draft
2 of 7 tasks
5 👀 3
Python - improve compatibility with sentencepiece in the conversion script
#229 by sdadas was merged Apr 13, 2020 Loading…
1 👀 1
Convert word counts to u64
#1433 by stephenroller was merged Feb 6, 2024 Loading…
1
10 👀 1
implement a simple max_sentencepiece_length into BPE
#1228 by chris-ha458 was merged May 16, 2023 Loading…
32 👀 1
fix fmt::Display for WordPiece error
#3 by epwalsh was merged Dec 12, 2019 Loading…
1
Add benchmark framework and benches for BPE (GPT2)
#4 by epwalsh was merged Jan 1, 2020 Loading…
11
Add RustFmt and Clippy to CI pipeline
#5 by epwalsh was merged Dec 13, 2019 Loading…
1
Add BPE tests and documentation
#6 by epwalsh was merged Dec 20, 2019 Loading…
1
remove Cargo.lock
#7 by epwalsh was merged Dec 24, 2019 Loading…
3
simplify initialization of BpeTrainer
#8 by epwalsh was merged Dec 24, 2019 Loading…
1
Allow importing from tokenizers modules.
#18 by mfuntowicz was closed Jan 14, 2020 Loading…
1
Clean up Rust docs
#20 by epwalsh was merged Dec 30, 2019 Loading…
2
Implement dropout for BPE
#21 by epwalsh was merged Dec 31, 2019 Loading…
1
3
replace print statements with logging
#22 by epwalsh was closed Jan 1, 2020 Loading…
1
make sure we don't warn on empty tokens
#1554 by ArthurZucker was merged Jun 20, 2024 Loading…
4
set capacity on BPE cache, change Mutex to RwLock, create BpeBuilder
#24 by epwalsh was merged Jan 2, 2020 Loading…
12
refactor benchmarks
#25 by epwalsh was merged Jan 2, 2020 Loading…
TokenizerBuilder
#27 by epwalsh was closed Oct 20, 2020 Loading…
5
[WIP] try parking_lot::RwLock
#28 by epwalsh was closed Jan 3, 2020 Loading…
3
[WIP] try chashmap in cache implementations
#29 by epwalsh was closed Jan 3, 2020 Loading…
3
[WIP] try evmap
#32 by epwalsh was closed Jan 3, 2020 Loading…
avoid unnecessary write locks in the BPE cache
#34 by epwalsh was merged Jan 4, 2020 Loading…
7
Update benchmarks
#36 by epwalsh was merged Jan 4, 2020 Loading…
1
2
make cache optional
#37 by epwalsh was merged Jan 3, 2020 Loading…
4
Avoid creating unnecessary vectors when accessing cache
#23 by epwalsh was merged Jan 1, 2020 Loading…
1
ProTip! Adding no:label will show everything without a label.