One possibility that you could try is to not add the embeddings to modules_to_save but instead just LoRA-tune them by adding them to target_modules. This could be especially useful for Gemma models, since they have huge embedding layers, so fully fine-tuning them pushes the number of trainable parameters up by a lot.

Another possibility (untested) is to try to manually tie the weights after initializing the PEFT models. So something along the lines of:

config = LoraConfig(..., modules_to_save=["embed_tokens", "lm_head"])
model = get_peft_model(model, config)
# exact names depend on architecture:
model.base_model.model.model.decoder.embed_tokens.modules_to_save["default"].weight = model.base_model.model.lm_head.modules_to_save["default"].weight

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.