o pi@@sddlmZmZmZmZmZddlZddlmZm Z ddl m Z ddl m Z mZddlmZd d lmZd d lmZd d lmZd d lmZd dlmZdZGdddeZdS))CallableDictListOptionalUnionN) CLIPTextModel CLIPTokenizer)DDPMWuerstchenScheduler) deprecatereplace_example_docstring)DiffusionPipeline) PaellaVQModel)WuerstchenDiffNeXt)WuerstchenPrior)WuerstchenDecoderPipeline)WuerstchenPriorPipelineax Examples: ```py >>> from diffusions import WuerstchenCombinedPipeline >>> pipe = WuerstchenCombinedPipeline.from_pretrained("warp-ai/Wuerstchen", torch_dtype=torch.float16).to( ... "cuda" ... ) >>> prompt = "an image of a shiba inu, donning a spacesuit and helmet" >>> images = pipe(prompt=prompt) ``` c/seZdZdZdZdedededede ded ed e d effd d Z d;de e fddZdWuerstchenCombinedPipelinea Combined Pipeline for text-to-image generation using Wuerstchen This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: tokenizer (`CLIPTokenizer`): The decoder tokenizer to be used for text inputs. text_encoder (`CLIPTextModel`): The decoder text encoder to be used for text inputs. decoder (`WuerstchenDiffNeXt`): The decoder model to be used for decoder image generation pipeline. scheduler (`DDPMWuerstchenScheduler`): The scheduler to be used for decoder image generation pipeline. vqgan (`PaellaVQModel`): The VQGAN model to be used for decoder image generation pipeline. prior_tokenizer (`CLIPTokenizer`): The prior tokenizer to be used for text inputs. prior_text_encoder (`CLIPTextModel`): The prior text encoder to be used for text inputs. prior_prior (`WuerstchenPrior`): The prior model to be used for prior pipeline. prior_scheduler (`DDPMWuerstchenScheduler`): The scheduler to be used for prior pipeline. T tokenizer text_encoderdecoder schedulervqganprior_tokenizerprior_text_encoder prior_priorprior_schedulerc sPt|j||||||||| d t|||| d|_t|||||d|_dS)N) rrrrrrrrr)priorrrr)rrrrr)super__init__register_modulesr prior_piper decoder_pipe) selfrrrrrrrrr __class__y/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/pipelines/wuerstchen/pipeline_wuerstchen_combined.pyr!Is2   z#WuerstchenCombinedPipeline.__init__N attention_opcCs|j|dSN)r$*enable_xformers_memory_efficient_attention)r%r*r(r(r)r,pszEWuerstchenCombinedPipeline.enable_xformers_memory_efficient_attentioncudagpu_iddevicecC$|jj||d|jj||ddS)a Offloads all models to CPU using accelerate, reducing memory usage with a low impact on performance. Compared to `enable_sequential_cpu_offload`, this method moves one whole model at a time to the GPU when its `forward` method is called, and the model remains in GPU until the next model runs. Memory savings are lower than with `enable_sequential_cpu_offload`, but performance is much better due to the iterative execution of the `unet`. r.r/N)r#enable_model_cpu_offloadr$r%r.r/r(r(r)r2sz3WuerstchenCombinedPipeline.enable_model_cpu_offloadcCr0)u Offloads all models (`unet`, `text_encoder`, `vae`, and `safety checker` state dicts) to CPU using 🤗 Accelerate, significantly reducing memory usage. Models are moved to a `torch.device('meta')` and loaded on a GPU only when their specific submodule's `forward` method is called. Offloading happens on a submodule basis. Memory savings are higher than using `enable_model_cpu_offload`, but performance is lower. r1N)r#enable_sequential_cpu_offloadr$r3r(r(r)r5}r4z8WuerstchenCombinedPipeline.enable_sequential_cpu_offloadcCs$|jj||d|jj||ddS)N)iterabletotal)r# progress_barr$)r%r6r7r(r(r)r8sz'WuerstchenCombinedPipeline.progress_barcKs(|jjdi||jjdi|dS)Nr()r#set_progress_bar_configr$)r%kwargsr(r(r)r9sz2WuerstchenCombinedPipeline.set_progress_bar_configi<g@ grpillatentspromptheightwidthprior_num_inference_stepsprior_timestepsprior_guidance_scalenum_inference_stepsdecoder_timestepsdecoder_guidance_scalenegative_prompt prompt_embedsnegative_prompt_embedsnum_images_per_prompt generator output_type return_dictprior_callback_on_step_end(prior_callback_on_step_end_tensor_inputscallback_on_step_end"callback_on_step_end_tensor_inputscKs i}|dddur|d|d<tddd|dddur,tddd|d|d<|jdid | dur7|ndd |d |d |d |d|d| durO| ndd| d| d| d|d|ddddd|d||}|d}|jd||dur~|nd||| | |||||d |}|S)a  Function invoked when calling the pipeline for generation. Args: prompt (`str` or `List[str]`): The prompt or prompts to guide the image generation for the prior and decoder. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings for the prior. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings for the prior. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. height (`int`, *optional*, defaults to 512): The height in pixels of the generated image. width (`int`, *optional*, defaults to 512): The width in pixels of the generated image. prior_guidance_scale (`float`, *optional*, defaults to 4.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598). `prior_guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `prior_guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. prior_num_inference_steps (`Union[int, Dict[float, int]]`, *optional*, defaults to 60): The number of prior denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. For more specific timestep spacing, you can pass customized `prior_timesteps` num_inference_steps (`int`, *optional*, defaults to 12): The number of decoder denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. For more specific timestep spacing, you can pass customized `timesteps` prior_timesteps (`List[float]`, *optional*): Custom timesteps to use for the denoising process for the prior. If not defined, equal spaced `prior_num_inference_steps` timesteps are used. Must be in descending order. decoder_timesteps (`List[float]`, *optional*): Custom timesteps to use for the denoising process for the decoder. If not defined, equal spaced `num_inference_steps` timesteps are used. Must be in descending order. decoder_guidance_scale (`float`, *optional*, defaults to 0.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will ge generated by sampling using the supplied random `generator`. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between: `"pil"` (`PIL.Image.Image`), `"np"` (`np.array`) or `"pt"` (`torch.Tensor`). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.ImagePipelineOutput`] instead of a plain tuple. prior_callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `prior_callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. prior_callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `prior_callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. Examples: Returns: [`~pipelines.ImagePipelineOutput`] or `tuple` [`~pipelines.ImagePipelineOutput`] if `return_dict` is True, otherwise a `tuple`. When returning a tuple, the first element is a list with the generated images. prior_callbackNcallbackz1.0.0ztPassing `prior_callback` as an input argument to `__call__` is deprecated, consider use `prior_callback_on_step_end`prior_callback_stepszzPassing `prior_callback_steps` as an input argument to `__call__` is deprecated, consider use `prior_callback_on_step_end`callback_stepsr?r@rArE timestepsguidance_scalerHrIrJrKrLr>rMptrNFrQrRr) image_embeddingsr?rErWrXrHrLrMrNrQrRr()getpopr r#r$)r%r?r@rArBrCrDrErFrGrHrIrJrKrLr>rMrNrOrPrQrRr: prior_kwargs prior_outputsr[outputsr(r(r)__call__so       z#WuerstchenCombinedPipeline.__call__r+)Nr-)NN)#__name__ __module__ __qualname____doc___load_connected_pipesrrrr rrr!rrr,intrtorchr/strr2r5r8r9no_gradr TEXT2IMAGE_EXAMPLE_DOC_STRINGrfloatTensor Generatorboolrra __classcell__r(r(r&r)r+s  '""       r)typingrrrrrrh transformersrr schedulersr utilsr r pipeline_utilsrmodeling_paella_vq_modelrmodeling_wuerstchen_diffnextrmodeling_wuerstchen_priorrpipeline_wuerstchenrpipeline_wuerstchen_priorrrkrr(r(r(r)s