Related to #56.

@younesbelkada @pacman100

Consider the following LoraModel instance:

from transformers import AutoModelForImageClassification, TrainingArguments, Trainer
from peft import LoraConfig, LoraModel


model_checkpoint = "google/vit-base-patch16-224-in21k" 
model = AutoModelForImageClassification.from_pretrained(
    model_checkpoint,
    label2id=label2id,
    id2label=id2label,
    ignore_mismatched_sizes=True,  # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
)

config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=["query", "value"],
    lora_dropout=0.1,
    bias="none",
    modules_to_save=["classifier"],
)
lora_model = LoraModel(config, model)

If I call lora_model.save_pretrained("lora_vit"), I see that the state_dict is about the same size as that of model. Is this expected?

I would have expected to only see the LoRA trainable parameters alongside the modules_to_save ones. This would help reduce the size of the state dict and would also help with portability, especially for very large models. This is also how it's implemented in diffusers.

Also, PeftConfig is somehow unable to find the config.json here whereas it's clearly there as we can see. What am I missing out on?

This currently blocks the inference and sharing section of the notebook.