[go: nahoru, domu]
Skip to content

Pull requests: huggingface/tokenizers

New pull request New
Clear current search query, filters, and sorts
11 Open 587 Closed
11 Open 587 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

make sure we don't warn on empty tokens
#1554 by ArthurZucker was merged Jun 20, 2024 Loading…
4
feat(ci): add trufflehog secrets detection
#1551 by McPatate was merged Jun 10, 2024 Loading…
2
Enable dropout = 0.0 as an equivalent to none in BPE
#1550 by mcognetta was merged Jun 24, 2024 Loading…
10
Fixing for clippy 1.78
#1548 by Narsil was merged Jun 6, 2024 Loading…
1
Switch from cached_download to hf_hub_download in tests
#1547 by Wauplin was merged Jun 11, 2024 Loading…
3
Adding pretty print of tokenizer
#1540 by haixuanTao was closed Jun 5, 2024 Loading…
4
Make USED_PARALLELISM atomic
#1532 by nathaniel-daniel was merged Jun 6, 2024 Loading…
1
4
remove enforcement of non special when adding tokens
#1521 by ArthurZucker was merged Apr 30, 2024 Loading…
3
[BREAKING CHANGE] Ignore added_tokens (both special and not) in the decoder
#1513 by Narsil was merged May 6, 2024 Loading…
16
Fix "dictionnary" typo
#1511 by nprisbrey was merged Jun 11, 2024 Loading…
4
Add .editorconfig and rustfmt.toml for Consistent Code Formatting Stale
#1508 by tal7aouy was closed Jun 1, 2024 Loading…
1
add serialization for ignore_merges
#1504 by ArthurZucker was merged Apr 17, 2024 Loading…
5
corrected typo in the documentations for pre-tokenizers Stale
#1503 by GorkaUrbizu was closed May 23, 2024 Loading…
Fixing doc.
#1499 by Narsil was merged Apr 17, 2024 Loading…
1
Bumping all versions 3 times (ty transformers :) )
#1498 by Narsil was merged Apr 16, 2024 Loading…
2
Remove 3.13 (potential undefined behavior.)
#1497 by Narsil was merged Apr 16, 2024 Loading…
1
PyO3 0.21.
#1494 by Narsil was merged Apr 16, 2024 Loading…
1
Add more support for tiktoken based tokenizers
#1493 by ArthurZucker was merged Apr 15, 2024 Loading…
8
Fix unsoundness in tokenizers::utils::parallelism
#1492 by albertsgarde was closed Jun 6, 2024 Loading…
1
4
Fix data directory for test Stale
#1488 by atupone was closed May 13, 2024 Loading…
1
Fix Strip decoder doc comment Stale
#1486 by jacklee1792 was closed May 9, 2024 Loading…
Candidate release
#1484 by ArthurZucker was closed Apr 16, 2024 Loading…
1
fix: change var name from vocab to vocab_file Stale
#1483 by shenxiangzhuang was closed May 7, 2024 Loading…
fix: typo Stale
#1482 by shenxiangzhuang was closed May 7, 2024 Loading…
Bump express from 4.18.1 to 4.19.2 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code Stale
#1479 by dependabot bot was closed May 4, 2024 Loading…
2
ProTip! Mix and match filters to narrow down what you’re looking for.