Add new sampler DDIM CFG++ #16035

v0xie · 2024-06-17T01:25:35Z

Description

This PR implements a new sampler "DDIM CFG++" derived from CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models (Chung et al., 2024).

The new sampler is modified from DDIM, with the main change being we use the unconditional noise to guide the denoising instead of the conditional noise.

Major changes:

Adds a new sampler "ddim_cfgpp"
Added a new field in CFGDenoiser: "last_noise_uncond". The new sampler gets the unconditional noise prediction from this field.
CFG scale is divided by 12.5 if the sampler has "CFG++" in name.
- CFG scale maps to CFG++ scale: [1.0, 12.5] -> [0.0, 1.0].

Screenshots/videos:

SD 1.5
Prompt: "A photo of a silver 1998 Toyota Camry."
SD XL
Prompt: "A hyperrealistic portrait close-up photo of a smug man lighting a cigarette in his mouth in the city light at night in the rain with an explosion behind him."

Additional Links:

Official project page: https://cfgpp-diffusion.github.io/
Official code repository: https://github.com/CFGpp-diffusion/CFGpp
ArXiv: https://arxiv.org/abs/2406.08070

Checklist:

I have read contributing wiki page
I have performed a self-review of my own code
My code follows the style guidelines
My code passes tests

AndreyRGW · 2024-06-18T13:07:49Z

Error during generation. Also i see that controlnet is mentioned in the error, although it was not even used during generation.

    Traceback (most recent call last):
      File "F:\WBC\automatic1111_dev\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "F:\WBC\automatic1111_dev\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "F:\WBC\automatic1111_dev\modules\txt2img.py", line 109, in txt2img
        processed = processing.process_images(p)
      File "F:\WBC\automatic1111_dev\modules\processing.py", line 845, in process_images
        res = process_images_inner(p)
      File "F:\WBC\automatic1111_dev\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 59, in processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "F:\WBC\automatic1111_dev\modules\processing.py", line 981, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "F:\WBC\automatic1111_dev\modules\processing.py", line 1328, in sample
        samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
      File "F:\WBC\automatic1111_dev\modules\sd_samplers_timesteps.py", line 159, in sample
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "F:\WBC\automatic1111_dev\modules\sd_samplers_common.py", line 272, in launch_sampling
        return func()
      File "F:\WBC\automatic1111_dev\modules\sd_samplers_timesteps.py", line 159, in <lambda>
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "F:\WBC\automatic1111_dev\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "F:\WBC\automatic1111_dev\modules\sd_samplers_timesteps_impl.py", line 51, in ddim_cfgpp
        alphas_prev = alphas_cumprod[torch.nn.functional.pad(timesteps[:-1], pad=(1, 0))].to(float64(x))
    NameError: name 'float64' is not defined

---

Maybe you should change line 51 in the modules/sd_samplers_timesteps_impl.py file from
"alphas_prev = alphas_cumprod[torch.nn.functional.pad(timesteps[:-1], pad=(1, 0))].to(float64(x))"
to
"alphas_prev = alphas_cumprod[torch.nn.functional.pad(timesteps[:-1], pad=(1, 0))].to(torch.float64 if x.device.type != 'mps' and x.device.type != 'xpu' else torch.float32)"

What do you think?

AndreyRGW · 2024-06-18T13:12:01Z

modules/sd_samplers_timesteps_impl.py

+    """
+    alphas_cumprod = model.inner_model.inner_model.alphas_cumprod
+    alphas = alphas_cumprod[timesteps]
+    alphas_prev = alphas_cumprod[torch.nn.functional.pad(timesteps[:-1], pad=(1, 0))].to(float64(x))


Change to alphas_prev = alphas_cumprod[torch.nn.functional.pad(timesteps[:-1], pad=(1, 0))].to(torch.float64 if x.device.type != 'mps' and x.device.type != 'xpu' else torch.float32) to prevent NameError: name 'float64' is not defined

v0xie · 2024-06-18T22:44:53Z

Error during generation. Also i see that controlnet is mentioned in the error, although it was not even used during generation.

    Traceback (most recent call last):
      File "F:\WBC\automatic1111_dev\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "F:\WBC\automatic1111_dev\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "F:\WBC\automatic1111_dev\modules\txt2img.py", line 109, in txt2img
        processed = processing.process_images(p)
      File "F:\WBC\automatic1111_dev\modules\processing.py", line 845, in process_images
        res = process_images_inner(p)
      File "F:\WBC\automatic1111_dev\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 59, in processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "F:\WBC\automatic1111_dev\modules\processing.py", line 981, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "F:\WBC\automatic1111_dev\modules\processing.py", line 1328, in sample
        samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
      File "F:\WBC\automatic1111_dev\modules\sd_samplers_timesteps.py", line 159, in sample
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "F:\WBC\automatic1111_dev\modules\sd_samplers_common.py", line 272, in launch_sampling
        return func()
      File "F:\WBC\automatic1111_dev\modules\sd_samplers_timesteps.py", line 159, in <lambda>
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "F:\WBC\automatic1111_dev\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "F:\WBC\automatic1111_dev\modules\sd_samplers_timesteps_impl.py", line 51, in ddim_cfgpp
        alphas_prev = alphas_cumprod[torch.nn.functional.pad(timesteps[:-1], pad=(1, 0))].to(float64(x))
    NameError: name 'float64' is not defined

---

Maybe you should change line 51 in the modules/sd_samplers_timesteps_impl.py file from "alphas_prev = alphas_cumprod[torch.nn.functional.pad(timesteps[:-1], pad=(1, 0))].to(float64(x))" to "alphas_prev = alphas_cumprod[torch.nn.functional.pad(timesteps[:-1], pad=(1, 0))].to(torch.float64 if x.device.type != 'mps' and x.device.type != 'xpu' else torch.float32)"

What do you think?

The master branch is missing the float64 method from modules/torch_utils.py. I'd recommend testing on the dev branch for now.

Relevant commit: 9c8075b

v0xie · 2024-06-19T00:52:31Z

It's possible this is something extensible to many of the existing samplers. Samples generated from my more_cfgpp branch https://github.com/v0xie/stable-diffusion-webui/tree/more_cfgpp

viking1304 · 2024-06-20T21:52:39Z

The master branch is missing the float64 method from modules/torch_utils.py. I'd recommend testing on the dev branch for now.

Relevant commit: 9c8075b

Implementation of float64 in torch_utils added in #15815 does not work as intended. This function will always return torch.float64, even on mps and xpu.

I just posted #16058 with proper implementation.

So, if someone getting errors on Mac while testing this, apply my fix for torch_utils first.

Panchovix · 2024-06-25T13:24:44Z

It's possible this is something extensible to many of the existing samplers. Samples generated from my more_cfgpp branch https://github.com/v0xie/stable-diffusion-webui/tree/more_cfgpp

Can you point this fork into the dev branch? Or it is possible to make it like a schedule parameter? To be used like this

EDIT: Did a semi implementation here Panchovix@f8dfe20

…art 1 Re-applied AUTOMATIC1111/stable-diffusion-webui#15333 into Forge, to use any scheduler with any sampler. Also implemented CFG++ from AUTOMATIC1111/stable-diffusion-webui#16035 as a scheduler instead of a sampler. Ported SD Turbo and Variance Preserving from https://github.com/lllyasviel/stable-diffusion-webui-forge/blob/main/ldm_patched/contrib/external_custom_sampler.py

Adds CFG++ Scheduler, making a mirror implementation of how CFG++ Sampler is implemented on AUTOMATIC1111#16035 Modified the code to make it work with k_diffusion samplers and not break compatibility with other schedulers.

add new sampler DDIM CFG++

663a4d8

v0xie requested a review from AUTOMATIC1111 as a code owner June 17, 2024 01:25

AndreyRGW reviewed Jun 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new sampler DDIM CFG++ #16035

Add new sampler DDIM CFG++ #16035

Add new sampler DDIM CFG++ #16035

Are you sure you want to change the base?

Add new sampler DDIM CFG++ #16035

Conversation

Description

Screenshots/videos:

Additional Links:

Checklist:

Choose a reason for hiding this comment