[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Question]: 如何使用LoRA #8647

Open
carbonatedbeverages opened this issue Jun 21, 2024 · 1 comment
Open

[Question]: 如何使用LoRA #8647

carbonatedbeverages opened this issue Jun 21, 2024 · 1 comment
Assignees
@ZHUI
Labels
question Further information is requested

Comments

@carbonatedbeverages
Copy link

请提出你的问题

model=ppnlp.transformers.NeZhaForSequenceClassification.from_pretrained('nezha-large-wwm-chinese', num_classes=14)
tokenizer = ppnlp.transformers.NeZhaTokenizer.from_pretrained('nezha-large-wwm-chinese')
lora_config = LoRAConfig(
r=16,
target_modules=['query', 'value'],
lora_alpha=32,
lora_dropout=0.05,
)
lora_model = LoRAModel(model, lora_config)
model = lora_model
运行后显示[2024-06-21 21:58:38,253] [ INFO] - Frozen parameters: 0.00e+00 || Trainable parameters:3.25e+08 || Total parameters:3.25e+08|| Trainable:100.00%
LoRA没有起作用是为什么呢？以上写法的问题在哪
@carbonatedbeverages carbonatedbeverages added the question Further information is requested label Jun 21, 2024
@DrownFish19
Copy link
Collaborator
DrownFish19 commented Jun 24, 2024 •
edited
Loading

target_modules: List of module names or regex expression of the module names to replace with Lora.
target_modules=['query', 'value'],应该使用完整的参数名称或者正则表达式，如target_modules=['*query*', '*value*'],
可参考target_module写法

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ZHUI ZHUI

Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@DrownFish19 @ZHUI @carbonatedbeverages