-
Notifications
You must be signed in to change notification settings - Fork 26.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AttributeError: 'LlamaForCausalLM' object has no attribute '_setup_cache' #31157
Comments
Static Cache was moved to be a standalone object in #30476. You have to init StaticCache outside the model and pass in every forward call, similar to following: past_key_values = StaticCache(model.config, bs, max_cache_length, model.device, model.dtype)
for i in range(max_new_tokens):
out = model(next_token, past_key_values=past_key_values, return_dict=True, **model_kwargs)
past_key_values = out.past_key_values
next_token = sample_next_token(out)
model_kwargs = update_model_kwargs(model_kwargs) Also you can simply pass-in model.forward = torch.compile(model.forward, mode="reduce-overhead", fullgraph=True)
model.generation_config.max_new_tokens = 20
out = model.generate(input_ids, cache_implementation="static) |
Thanks, will try that!
That's not the same thing. This will compile the whole forward pass, which will force re-compilation every time the prompt length changes. I only want to compile the decoding part and leave the prefill phase uncompiled. |
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
System Info
transformers
version: 4.41.1Who can help?
@ArthurZucker @gante
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Seems like
_setup_cache()
was removed in the newer version of transformers. Is there an alternative to this?It's necessary to do that call in order to properly compile the model for faster generation.
Thanks!
Expected behavior
model._setup_cache()
should setup the cache.The text was updated successfully, but these errors were encountered: