[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix decode #1555

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open

Fix decode #1555

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
0f72528
Make USED_PARALLELISM atomic (#1532)
nathaniel-daniel Jun 6, 2024
c910d73
Fixing for clippy 1.78 (#1548)
Narsil Jun 6, 2024
6d17c4b
feat(ci): add trufflehog secrets detection (#1551)
McPatate Jun 10, 2024
b23d8c3
Switch from cached_download to hf_hub_download in tests (#1547)
Wauplin Jun 11, 2024
b801eed
Fix 'dictionnary' typo (#1511)
nprisbrey Jun 11, 2024
d59973a
current state of affaiers
ArthurZucker Jun 17, 2024
34d7cf8
something that works
ArthurZucker Jun 18, 2024
3ff2016
feature dependent test
ArthurZucker Jun 18, 2024
948358f
nit about 嗎
ArthurZucker Jun 18, 2024
17a286a
Merge branch 'main' into fix-decode
ArthurZucker Jun 18, 2024
b107253
fix merge conflicts
ArthurZucker Jun 18, 2024
487810a
update
ArthurZucker Jun 18, 2024
ed3428a
actuallyfix it
ArthurZucker Jun 28, 2024
f53e514
revert and simplify
ArthurZucker Jun 28, 2024
0f72944
update the test
ArthurZucker Jun 28, 2024
eff3eee
add it
ArthurZucker Jun 28, 2024
de895de
fix
ArthurZucker Jun 28, 2024
be0ae2e
stub
ArthurZucker Jun 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixing for clippy 1.78 (#1548)
  • Loading branch information
@Narsil @ArthurZucker
Narsil authored and ArthurZucker committed Jun 17, 2024
commit c910d73fd40eb89ab11e806b1417a90d220064ca
2 changes: 1 addition & 1 deletion tokenizers/src/models/bpe/trainer.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -523,7 +523,7 @@ impl BpeTrainer {
.get(&new_token)
.copied()
.unwrap_or(id_to_word.len() as u32);
if word_to_id.get(&new_token).is_none() {
if !word_to_id.contains_key(&new_token) {
id_to_word.push(new_token.clone());
word_to_id.insert(new_token.clone(), new_token_id);
}
Expand Down
4 changes: 2 additions & 2 deletions tokenizers/src/models/wordpiece/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,10 +180,10 @@ impl WordPiece {
pub fn from_bpe(bpe: &BPE) -> Self {
let mut wp = Self::builder().vocab(bpe.get_vocab()).build().unwrap();
if let Some(unk) = bpe.get_unk_token() {
wp.unk_token = unk.to_owned();
unk.clone_into(&mut wp.unk_token);
}
if let Some(prefix) = bpe.get_continuing_subword_prefix() {
wp.continuing_subword_prefix = prefix.to_owned();
prefix.clone_into(&mut wp.continuing_subword_prefix);
}
wp
}
Expand Down