[go: nahoru, domu]
Skip to content

Pull requests: huggingface/tokenizers

New pull request New
Clear current search query, filters, and sorts
11 Open 590 Closed
11 Open 590 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Adding ByteFallback support for tokenizers.
#1183 by Narsil was merged Mar 23, 2023 Loading…
1
15 🎉 2
Add a visualization utility to render tokens and annotations in a notebook
#508 by talolard was merged Dec 4, 2020 Loading…
30 🎉 2
Ability to train from memory
#544 by n1t0 was merged Nov 28, 2020 Loading…
1 task done
2
🎉 2
Fix BPE trainer pair counts
#179 by n1t0 was merged Mar 2, 2020 Loading…
🎉 2
Fixing tokenizers with 1.53 (updated some dependencies + clippy)
#764 by Narsil was merged Jul 21, 2021 Loading…
4 🎉 1
Port of unigram algorithm.
#292 by Narsil was merged Sep 2, 2020 Loading…
30 🎉 1
Improve PreTokenizer and Model interfaces
#360 by n1t0 was merged Aug 3, 2020 Loading…
30 🎉 1
Fix broken links in docs
#1133 by hvaara was merged Dec 23, 2022 Loading…
1
2 🎉 1
fix fmt::Display for WordPiece error
#3 by epwalsh was merged Dec 12, 2019 Loading…
1
Add RustFmt and Clippy to CI pipeline
#5 by epwalsh was merged Dec 13, 2019 Loading…
1
Add BPE tests and documentation
#6 by epwalsh was merged Dec 20, 2019 Loading…
1
simplify initialization of BpeTrainer
#8 by epwalsh was merged Dec 24, 2019 Loading…
1
Allow importing from tokenizers modules.
#18 by mfuntowicz was closed Jan 14, 2020 Loading…
1
Clean up Rust docs
#20 by epwalsh was merged Dec 30, 2019 Loading…
2
Implement dropout for BPE
#21 by epwalsh was merged Dec 31, 2019 Loading…
1
3
replace print statements with logging
#22 by epwalsh was closed Jan 1, 2020 Loading…
1
make sure we don't warn on empty tokens
#1554 by ArthurZucker was merged Jun 20, 2024 Loading…
4
TokenizerBuilder
#27 by epwalsh was closed Oct 20, 2020 Loading…
5
[WIP] try parking_lot::RwLock
#28 by epwalsh was closed Jan 3, 2020 Loading…
3
[WIP] try chashmap in cache implementations
#29 by epwalsh was closed Jan 3, 2020 Loading…
3
[WIP] try evmap
#32 by epwalsh was closed Jan 3, 2020 Loading…
avoid unnecessary write locks in the BPE cache
#34 by epwalsh was merged Jan 4, 2020 Loading…
7
Update benchmarks
#36 by epwalsh was merged Jan 4, 2020 Loading…
1
2
make cache optional
#37 by epwalsh was merged Jan 3, 2020 Loading…
4
Avoid creating unnecessary vectors when accessing cache
#23 by epwalsh was merged Jan 1, 2020 Loading…
1
ProTip! Type g p on any issue or pull request to go back to the pull request listing page.