System Info

PyTorch version: 2.2.1
peft version: 0.11.1

Reproduction

The following snippet can trigger this bug:

import multiprocessing
import peft
from torch import nn

def func():
    m = nn.Linear(2, 3).cuda(0)

if __name__ == "__main__":
    proc = multiprocessing.Process(target=func)
    proc.start(), proc.join()

Raise:
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

Issue: early import torch.utils.cpp_extension in peft/tuners/boft/layer.py

How to fix: import cpp_extension module in the get_fbd_cuda function.

Expected behavior

No RuntimeError. CUDA context is not initialized in the main process.