I think that right now, it would not be possible to have different tasks/adapters within a single batch. Moreover, I don't think it can be easily supported (not impossible, but would require serious refactoring).

What does work already is to have a single model with multiple adapters and then switch the adapters by calling set_adapter(<adapter-name>) between batches (should also work with LoRA). That way, you would still have a rather resource efficient deployment.

In practice, this approach would probably mean that you have to buffer prediction calls from different adapters to have any benefit from batching and to amortize the cost of switching adapters. Depending on your use case, this may or may not be feasible.

Thanks, I suspected that it would be a big reactor... I will probably do as you say initially. Might run the base model as one instance and then peft model with changeable adapters as a second instance for specific tasks.

I will ask the IA^3 authors as well, but do you know of any other implementation that supports mixed-task batches?

I will ask the IA^3 authors as well, but do you know of any other implementation that supports mixed-task batches?

No, I'm not aware, but please let us know if you find (or build) something.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Training or serving multiple LoRA modules in a single batch with the following community implementations:
[https://github.com/S-LoRA/S-LoRA]
[https://github.com/sabetAI/BLoRA]
[https://github.com/TUDB-Labs/multi-lo...