o piE@sddlmZmZmZmZddlZddlmZddlm Z m Z ddl m Z m Z ddlmZmZddlmZd d lmZmZd d lmZeeZd ZdddZGdddeZdS))CallableListOptionalUnionN)XLMRobertaTokenizer)UNet2DConditionModelVQModel) DDIMScheduler DDPMScheduler)loggingreplace_example_docstring) randn_tensor)DiffusionPipelineImagePipelineOutput)MultilingualCLIPav Examples: ```py >>> from diffusers import KandinskyPipeline, KandinskyPriorPipeline >>> import torch >>> pipe_prior = KandinskyPriorPipeline.from_pretrained("kandinsky-community/Kandinsky-2-1-prior") >>> pipe_prior.to("cuda") >>> prompt = "red cat, 4k photo" >>> out = pipe_prior(prompt) >>> image_emb = out.image_embeds >>> negative_image_emb = out.negative_image_embeds >>> pipe = KandinskyPipeline.from_pretrained("kandinsky-community/kandinsky-2-1") >>> pipe.to("cuda") >>> image = pipe( ... prompt, ... image_embeds=image_emb, ... negative_image_embeds=negative_image_emb, ... height=768, ... width=768, ... num_inference_steps=100, ... ).images >>> image[0].save("cat.png") ``` cCsX||d}||ddkr|d7}||d}||ddkr$|d7}||||fS)Nrrr)hw scale_factornew_hnew_wrrn/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/pipelines/kandinsky/pipeline_kandinsky.py get_new_h_wBs  rc!s4eZdZdZdZdedededee e fde f fdd Z d d Z d&d dZeee        d'deeeefdeejeejfdeejeejfdeeeeefdedededededeeejeejfdeejd eed!eeeeejgd fd"ed#efd$d%ZZS)(KandinskyPipelinea1 Pipeline for text-to-image generation using Kandinsky This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: text_encoder ([`MultilingualCLIP`]): Frozen text-encoder. tokenizer ([`XLMRobertaTokenizer`]): Tokenizer of class scheduler (Union[`DDIMScheduler`,`DDPMScheduler`]): A scheduler to be used in combination with `unet` to generate image latents. unet ([`UNet2DConditionModel`]): Conditional U-Net architecture to denoise the image embedding. movq ([`VQModel`]): MoVQ Decoder to generate the image from the latents. ztext_encoder->unet->movq text_encoder tokenizerunet schedulermovqcs:t|j|||||ddt|jjjd|_dS)N)rrr r!r"rr)super__init__register_moduleslenr"configblock_out_channelsmovq_scale_factor)selfrrr r!r" __class__rrr$bs zKandinskyPipeline.__init__cCsR|dur t||||d}n|j|krtd|jd|||}||j}|S)N) generatordevicedtypezUnexpected latents shape, got z , expected )rshape ValueErrortoinit_noise_sigma)r*r0r/r.r-latentsr!rrrprepare_latentsvs   z!KandinskyPipeline.prepare_latentsNc CsZt|tr t|nd}|j|ddddddd}|j}|j|dddj} | jd |jd krRt|| sR|j| dd|jj dd f} t d |jj d | | |}|j |} |j|| d \} } | j|d d} | j|d d} | j|d d} |r(|durdg|}n;t|t|urtdt|dt|dt|tr|g}n|t|krtd|dt|d|d|d |}|j|ddddddd}|j |}|j |}|j||d \}}|jd}|d|}||||}|jd}|d|d}||||d }|j|d d}t|| g} t|| g} t|| g} | | | fS)Nr max_lengthTMpt)padding truncationr6return_attention_maskadd_special_tokensreturn_tensorslongest)r9r=z\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: ) input_idsattention_maskrdimz?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)r9r6r:r;r<r=) isinstancelistr&rr@r0torchequal batch_decodemodel_max_lengthloggerwarningr2rArrepeat_interleavetype TypeErrorstrr1repeatviewcat)r*promptr.num_images_per_promptdo_classifier_free_guidancenegative_prompt batch_size text_inputstext_input_idsuntruncated_ids removed_text text_mask prompt_embedstext_encoder_hidden_states uncond_tokens uncond_inputuncond_text_input_idsuncond_text_masknegative_prompt_embeds!uncond_text_encoder_hidden_statesseq_lenrrr_encode_prompts  $             z KandinskyPipeline._encode_promptd@rpilTrU image_embedsnegative_image_embedsrXheightwidthnum_inference_stepsguidance_scalerVr-r4 output_typecallbackcallback_steps return_dictc#Cst|trd}nt|trt|}n tdt||j}|| }|dk}|||| ||\}}}t|tr>tj |dd}t|trJtj |dd}|ri|j | dd}|j | dd}tj ||gddj |j |d}|j j||d|j j}|jjj}t|||j\}}|||||f|j || | |j } t||D]\}}|rt | gdn| }||d }|j||||d d d}|r|j| jddd\}}|d\}}|d\}} ||||}tj || gdd}t|j jd r|j jjd vs|j| jddd\}}|j j||| | dj} | dur)||dkr)|t|j dd}!| |!|| q|j j!| ddd}"|"| dvrDtd| | dvrc|"dd}"|"#dd}"|"$%dddd&'}"| dkrm|(|"}"|ss|"fSt)|"dS)a Function invoked when calling the pipeline for generation. Args: prompt (`str` or `List[str]`): The prompt or prompts to guide the image generation. image_embeds (`torch.Tensor` or `List[torch.Tensor]`): The clip image embeddings for text prompt, that will be used to condition the image generation. negative_image_embeds (`torch.Tensor` or `List[torch.Tensor]`): The clip image embeddings for negative text prompt, will be used to condition the image generation. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). height (`int`, *optional*, defaults to 512): The height in pixels of the generated image. width (`int`, *optional*, defaults to 512): The width in pixels of the generated image. num_inference_steps (`int`, *optional*, defaults to 100): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. guidance_scale (`float`, *optional*, defaults to 4.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will ge generated by sampling using the supplied random `generator`. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between: `"pil"` (`PIL.Image.Image`), `"np"` (`np.array`) or `"pt"` (`torch.Tensor`). callback (`Callable`, *optional*): A function that calls every `callback_steps` steps during inference. The function is called with the following arguments: `callback(step: int, timestep: int, latents: torch.Tensor)`. callback_steps (`int`, *optional*, defaults to 1): The frequency at which the `callback` function is called. If not specified, the callback is called at every step. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.ImagePipelineOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.ImagePipelineOutput`] or `tuple` rz2`prompt` has to be of type `str` or `list` but is g?rrB)r/r.)r.r) text_embedsrmF)sampletimestepencoder_hidden_statesadded_cond_kwargsrv variance_type)learned learned_range)r-NorderT)force_not_quantizerx)r8nprlzIOnly the output types `pt`, `pil` and `np` are supported not output_type=)rrlg?rrl)images)*rFrQrGr&r1rO_execution_devicerhrHrTrNr2r/r! set_timesteps timestepsr r' in_channelsrr)r5 enumerate progress_barsplitr0chunkhasattrr|step prev_samplegetattrr"decodemaybe_free_model_hooksclampcpupermutefloatnumpy numpy_to_pilr)#r*rUrmrnrXrorprqrrrVr-r4rsrtrurvrYr.rWr_r`_timesteps_tensornum_channels_latentsitlatent_model_inputr{ noise_pred variance_prednoise_pred_uncondnoise_pred_textvariance_pred_textstep_idximagerrr__call__s I                  zKandinskyPipeline.__call__)N) NririrjrkrNNrlNrT)__name__ __module__ __qualname____doc__model_cpu_offload_seqrrrrr r r r$r5rhrHno_gradr EXAMPLE_DOC_STRINGrQrTensorrintr Generatorrboolr __classcell__rrr+rrLs|  f     r)r)typingrrrrrH transformersrmodelsrr schedulersr r utilsr r utils.torch_utilsrpipeline_utilsrrrr get_loggerrrLrrrrrrrs