Do I need to report the model file in bin format during the training process? #61

ScottishFold007 · 2023-02-09T02:27:04Z

Do I need to keep the model file in bin format when training the model with peft at that time? I saved it and used it in combination with the 'lora.pt' file and found that the model generation was poor and did not make much sense.
This is my infering code:

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import get_peft_config, get_peft_model, LoraConfig, TaskType, peft_model_load_and_dispatch


model_name_or_path = "/root/gaochangkuan_AI/PromptCLUE_Finetuning/model_finetuning_1_epoch"
checkpoint_name="/root/gaochangkuan_AI/PromptCLUE_Finetuning/model_finetuning_1_epoch/promptclue_lora_fsdp_v1.pt"
max_memory={0: "1GIB", 1: "1GIB", 2: "2GIB", 3: "10GIB", "cpu":"30GB"}
peft_config = LoraConfig(
    task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=True, r=8, lora_alpha=32, lora_dropout=0.1
)
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name_or_path, 
                                #device_map="auto", 
                                max_memory=max_memory
                                )
#model = get_peft_model(model, peft_config)
device = torch.device('cuda:7') # cuda
model.to(device)
peft_model_load_and_dispatch(model, torch.load(checkpoint_name), peft_config, max_memory)

Note:The model file in "model_finetuning_1_epoch" is saved during training, not the initial model.

So, where might the problem lie?

The text was updated successfully, but these errors were encountered:

pacman100 · 2023-02-09T05:04:08Z

Hello, @ScottishFold007, could you provide the minimal training code and the training setup such as how many GPUs.
When saving a model trained using PEFT, you don't need to save the entire model, i.e., remove the below line:

- unwrapped_model.save_pretrained()

For saving and loading, please use the new cool HF hub utils from main branch:

Install from main branch:

pip install git+https://github.com/huggingface/peft.git

Saving PEFT model:

peft_model_id = f"/root/gaochangkuan_AI/PromptCLUE_Finetuning/model_finetuning_1_epoch/"
model.save_pretrained(peft_model_id)

Loading PEFT model for inference:

from peft import PeftModel, PeftConfig
peft_model_id = f"/root/gaochangkuan_AI/PromptCLUE_Finetuning/model_finetuning_1_epoch/"

config = PeftConfig.from_pretrained(peft_model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path)
model = PeftModel.from_pretrained(model, peft_model_id)

Please refer this notebook for an end-to-end example: https://github.com/huggingface/peft/blob/main/examples/conditional_generation/peft_lora_seq2seq.ipynb

ScottishFold007 · 2023-02-09T06:44:17Z

Thank you very much, I solved this problem with your guidance! ***@***.*** From: Sourab Mangrulkar Date: 2023-02-09 13:04 To: huggingface/peft CC: Scottish_Fold007; Mention Subject: Re: [huggingface/peft] Do I need to report the model file in bin format during the training process? (Issue #61) Hello, @ScottishFold007, could you provide the minimal training code and the training setup such as how many GPUs. When saving a model trained using PEFT, you don't need to save the entire model, i.e., remove the below line: - unwrapped_model.save_pretrained() For saving and loading, please use the new cool HF hub utils from main branch: Install from main branch: pip install git+https://github.com/huggingface/peft.git Saving PEFT model: peft_model_id = f"/root/gaochangkuan_AI/PromptCLUE_Finetuning/model_finetuning_1_epoch/" model.save_pretrained(peft_model_id) Loading PEFT model for inference: from peft import PeftModel, PeftConfig peft_model_id = f"/root/gaochangkuan_AI/PromptCLUE_Finetuning/model_finetuning_1_epoch/" config = PeftConfig.from_pretrained(peft_model_id) model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path) model = PeftModel.from_pretrained(model, peft_model_id) Please refer this notebook for an end-to-end example: https://github.com/huggingface/peft/blob/main/examples/conditional_generation/peft_lora_seq2seq.ipynb — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

ScottishFold007 closed this as completed Feb 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do I need to report the model file in bin format during the training process? #61

Do I need to report the model file in bin format during the training process? #61

Do I need to report the model file in bin format during the training process? #61

Do I need to report the model file in bin format during the training process? #61

Comments