Hello @YooSungHyun, as per triton-inference-server/server#5989 (comment), the issue seems to stem from the fact that torch.compile is incompatible with Peft models at present. We will put the deep dive for this in our backlog but this will take time.

@pacman100 thx, if you need something, i will help you. plz just tagging me🤗

if i use like this.

model = AutoModelForCausalLM.from_pretrained(
    optional_config["model_path"],
    low_cpu_mem_usage=bool(strtobool(optional_config["low_cpu_mem_usage"])),
    load_in_8bit=bool(strtobool(optional_config["load_in_8bit"])),
    torch_dtype=getattr(torch, optional_config["torch_dtype"], None),
    device_map=device_map,
)

self.model = PeftModel.from_pretrained(
    model,
    optional_config["aa_lora_weights"],
    adapter_name=optional_config["aa_lora_name"],
    device_map=device_map,
)
self.model.load_adapter(
    optional_config["bb_lora_weights"],
    adapter_name=optional_config["bb_lora_name"],
    device_map=device_map,
)
model = torch.compile(model)
self.model.eval()

it work fine

Great! So, it works!

@pacman100 but, this is weired... why important torch.compile's implement line number...?

Torch.compile should always be at the end of all the preprocessing of the model. For example, when using DDP too, torch.compile is applied after wrapping the model in DDP.

@pacman100 oh thx

@pacman100 ah.. how about eval()?

model.eval()
model = torch.compile(model)

model = torch.compile(model)
model.eval()

each correct?

@pacman100 hello?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

@pacman100

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.