Domain Targeted Synthetic Plant Style Transfer using Stable Diffusion LoRA and ControlNet

Zane K.J. Hartley, Rob J. Lind, Michael P. Pound, Andrew P. French; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 5375-5383

Synthetic images can help alleviate much of the cost in the creation of training data for plant phenotyping-focused AI development. Synthetic-to-real style transfer is of particular interest to users of artificial data because of the domain shift problem created by training neural networks on images generated in a digital environment. In this paper we present a pipeline for synthetic plant creation and image-to-image style transfer with a particular interest in synthetic to real domain adaptation targeting specific real datasets. Utilizing new advances in generative AI we employ a combination of Stable diffusion Low Ranked Adapters (LoRA) and ControlNets to produce an advanced system of style transfer. We focus our work on the core task of leaf instance segmentation exploring both synthetic to real style transfer as well as inter-species style transfer and find that our pipeline makes numerous improvements over CycleGAN for style transfer and the images we produce are comparable to real images when used as training data.