[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MarianMTModel/MBartForConditionalGeneration does not support device_map='auto' yet #22305

Closed
2 of 4 tasks
TranPhu1999 opened this issue Mar 22, 2023 · 5 comments · Fixed by #22309
Closed
2 of 4 tasks

MarianMTModel/MBartForConditionalGeneration does not support device_map='auto' yet #22305

TranPhu1999 opened this issue Mar 22, 2023 · 5 comments · Fixed by #22309

Comments

@TranPhu1999
Copy link
TranPhu1999 commented Mar 22, 2023 •
edited
Loading

System Info

Hi, I'm experiment some Transformer models for Translation task. These model are vinai-translate-en2vi, wmt19-ru-en

In the attempt to optimize the Transformer inference time on single GPU, I tried to follow the instruction on this document but stump on this error. I found a similar case here where the solution is to Add accelerate support for the correspond model. Is it the solution for my problem too? Can anyone share your experience to optimize Transformer inference time? Thanks a lot.

Who can help?

No response

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

I use this code like the example
` from transformers import AutoModelForSeq2SeqLM

model_name = "vinai-translate-en2vi"
model_8bit = AutoModelForSeq2SeqLM.from_pretrained(model_name, device_map="auto", load_in_8bit=True)`

and

` from transformers import AutoModelForSeq2SeqLM

model_name = "wmt19-ru-en"
model_8bit = AutoModelForSeq2SeqLM.from_pretrained(model_name, device_map="auto", load_in_8bit=True)`

Error
ValueError: MarianMTModel does not support device_map='auto' yet.

and

ValueError: MBartForConditionalGeneration does not support device_map='auto' yet.

Expected behavior

The code in the instruction should be working
@amyeroberts
Copy link
Collaborator

Hi @TranPhu1999, thanks for raising this issue.

Yes, it seems an equivalent update to the MBart and MarianMT models would need to be added, as the one added to XGLM. Would you like to open a PR to add these changes?

cc @younesbelkada

@younesbelkada younesbelkada mentioned this issue Mar 22, 2023
Merged
@younesbelkada younesbelkada reopened this Mar 23, 2023
@younesbelkada
Copy link
Contributor
younesbelkada commented Mar 23, 2023 •
edited
Loading

Hi @TranPhu1999
You should be now able to use 8bit models for MBart, you can just do:

pip install git+https://github.com/huggingface/transformers.git

I will work later on adding the same support for Marian as well

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@maryevan
Copy link

Any updates on MarianMT models?

@ArthurZucker
Copy link
Collaborator

cc @younesbelkada

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[MBart] Add accelerate support for MBart younesbelkada/transformers
5 participants
@amyeroberts @TranPhu1999 @ArthurZucker @younesbelkada @maryevan