Commits

Commits on Mar 2, 2023

Using clippy 1.67 (#1167 )
Narsil
committedMar 2, 2023

Commits on Jan 23, 2023

Improved version. (#1154 )
Narsil
committedJan 23, 2023
Prevent using from_pretrained on invalid ids (better error message). (#1153 )
Narsil
committedJan 23, 2023
Making Tokenizer clone. (#1152 )
Narsil
committedJan 23, 2023

Commits on Jan 16, 2023

Adding python 3.8 for M1 (#1147 )
Narsil
committedJan 16, 2023

Commits on Jan 15, 2023

Add missing build targets (#1145 )
Narsil
committedJan 15, 2023

Commits on Dec 19, 2022

Adding stale bot ? (#1123 )
Narsil
committedDec 19, 2022
Fixing conda ssl location (#1124 )
Narsil
committedDec 19, 2022

Commits on Nov 9, 2022

Adding rust audit. (#1099 )
Narsil
committedNov 9, 2022

Commits on Nov 7, 2022

Fixing extra wheels memory usage. (#1098 )
Narsil
committedNov 7, 2022

Commits on Oct 6, 2022

New version. (#1082 )
Narsil
committedOct 6, 2022

Commits on Oct 5, 2022

Update old gh actions, remove deprecated doc building. (#1069 )
Narsil
committedOct 5, 2022
Updating python formatting. (#1079 )
Narsil
committedOct 5, 2022

Commits on Sep 26, 2022

Fixing roberta type id (everything is zero). (#1072 )
Narsil
committedSep 26, 2022

Commits on Sep 22, 2022

Moving versions numbers to dev mode. (#1067 )
Narsil
committedSep 22, 2022

Commits on Sep 21, 2022

Enabling static interpreter embedding for manylinux. (#1064 )
Narsil
committedSep 21, 2022

Commits on Sep 19, 2022

Removing python3.6 from manylinux it's not supported anymore. (#1063 )
Narsil
committedSep 19, 2022

Commits on Sep 16, 2022

Turns out we introduced a regression because bad code. (#1060 )
Narsil
committedSep 16, 2022

Commits on Sep 12, 2022

Preparing rc1 release. (#1056 )
Narsil
committedSep 12, 2022

Commits on Aug 25, 2022

Adding Sequence for PostProcessor. (#1052 )
Narsil
committedAug 25, 2022
Making process_encodings not eat up the encodings any more. (#1051 )
Narsil
committedAug 25, 2022
Adding m1 build to the release process for Python. (#1055 )
Narsil
committedAug 25, 2022

Commits on Aug 24, 2022

Modify Processor trait to support chaining. (#1054 )
Narsil
committedAug 24, 2022
Updating code according to clippy. (#1048 )
Narsil
committedAug 24, 2022

Commits on Jun 10, 2022

Adding unstable_wasm feature + example to run tokenizers on wasm. (#1009 )

committedJun 10, 2022

Commits on Jun 2, 2022

Changing Decoder trait to be more composable. (#938 ) (#1008 )
Narsil
committedJun 2, 2022

Commits on Apr 14, 2022

Fixing the documentation for ByteLevel in Python (#982 )
Narsil
committedApr 14, 2022

Commits on Apr 12, 2022

Preparing for 0.12.1 (#978 )
Narsil
committedApr 12, 2022
Adding a new document that is the checklist to make (#975 )

committedApr 12, 2022

Commits on Apr 4, 2022

Revert "Changing Decoder trait to be more composable. (#938 )" (#971 )
Narsil
committedApr 4, 2022

Commits on Mar 31, 2022

Update the builder to use earlier windows version (2022) is not understood. (#969 )
Narsil
committedMar 31, 2022
Preparing 0.12 release. (#967 )
Narsil
committedMar 31, 2022

Commits on Mar 24, 2022

Adding a link to the Ruby port of tokenizers (#961 )
Narsil
committedMar 24, 2022
Preventing yelling on empty OrderedVocab (triggered by pickle.dumps). (#963 )
Narsil
committedMar 24, 2022

Commits on Mar 23, 2022

Fix the error test for Python 3.10 (error message is different). (#962 )
Narsil
committedMar 23, 2022