-
Notifications
You must be signed in to change notification settings - Fork 747
Pull requests: huggingface/tokenizers
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
build(node): Include binaries in NPM packing
Stale
#1459
by aaronclong
was closed Apr 22, 2024
Loading…
Allow pre-tokenized inputs to encode/encode_batch
#249
by n1t0
was merged May 21, 2020
Loading…
2 tasks
Make LongestFirst truncation constant time and consistent
#389
by thomlake
was merged Sep 18, 2020
Loading…
Optimize GPT-2 regex as logic for improved performance
#973
by benwtrent
was closed Apr 7, 2022
Loading…
Added ability to inspect a 'Sequence' pre-tokenizer.
#1341
by eaplatanios
was merged Sep 21, 2023
Loading…
Add a visualization utility to render tokens and annotations in a notebook
#508
by talolard
was merged Dec 4, 2020
Loading…
implement a simple max_sentencepiece_length into BPE
#1228
by chris-ha458
was merged May 16, 2023
Loading…
Previous Next
ProTip!
Find all pull requests that aren't related to any open issues with -linked:issue.