[go: nahoru, domu]
Skip to content

Pull requests: huggingface/tokenizers

New pull request New
Clear current search query, filters, and sorts
11 Open 590 Closed
11 Open 590 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Remove Node Container
#365 by n1t0 was merged Aug 4, 2020 Loading…
Use byte offsets
#377 by n1t0 was merged Aug 19, 2020 Loading…
1
Accept numpy arrays as input
#379 by n1t0 was merged Aug 21, 2020 Loading…
Expose the trainer to Python bindings.
#387 by Narsil was merged Sep 2, 2020 Loading…
6
Adding a new pre_tokenizer: Digits.
#390 by Narsil was closed Sep 3, 2020 Loading…
Adding a new PreTokenizer for UnicodeScripts.
#392 by Narsil was closed Sep 3, 2020 Loading…
Adding a new PreTokenizer for UnicodeScripts.
#393 by Narsil was closed Sep 3, 2020 Loading…
Adding a new pre_tokenizer: Digits.
#394 by Narsil was merged Sep 4, 2020 Loading…
3
Template processor
#403 by n1t0 was merged Sep 10, 2020 Loading…
1
8
Chain tokenizer methods
#407 by n1t0 was merged Sep 10, 2020 Loading…
1
Adding a new normalizer that strips accents by removing combining
#416 by Narsil was merged Sep 17, 2020 Loading…
1
5
Adding a new convert script, that will convert all tokenizers from transformer to tokenizers
#420 by Narsil was merged Sep 22, 2020 Loading…
1
NormalizedString - Do not compute original alignments
#421 by n1t0 was merged Sep 18, 2020 Loading…
Update PyO3 to 0.12
#426 by n1t0 was merged Sep 22, 2020 Loading…
2
Replace Normalizer can use Regex
#433 by n1t0 was merged Sep 24, 2020 Loading…
1
Fixed Dead Link: Build your own #435
#436 by dagisky was merged Sep 25, 2020 Loading…
1
3
There was a bug with the new merge type vs continuing subword prefix.
#440 by Narsil was merged Sep 28, 2020 Loading…
1
Improve template processing
#441 by n1t0 was merged Sep 29, 2020 Loading…
1
Update utils.py
#445 by yaelahaiz was closed Oct 1, 2020 Loading…
Fix TemplateProcessing
#448 by n1t0 was merged Oct 5, 2020 Loading…
Fixing Trainer with u8 instead of chars.
#452 by Narsil was merged Oct 9, 2020 Loading…
6
Fixing 1.47 complaints from clippy.
#453 by Narsil was merged Oct 9, 2020 Loading…
Proper fixing of new clippy lints.
#454 by Narsil was merged Oct 13, 2020 Loading…
5
Fixing a bug where long tokenizer files would be incorrectly deserialized
#459 by Narsil was merged Oct 13, 2020 Loading…
6
Fix RobertaProcessing deserialization in PostProcessorWrapper
#464 by n1t0 was merged Oct 15, 2020 Loading…
ProTip! Type g p on any issue or pull request to go back to the pull request listing page.