o piOx@sddlZddlmZmZmZmZmZddlZddl Z ddl Z ddl Z ddl m Z mZddlmZddlmZmZddlmZddlmZmZmZdd lmZd d lmZmZee Z!d Z"dddZ#ddZ$GdddeeZ%dS)N)CallableDictListOptionalUnion)T5EncoderModel T5Tokenizer)StableDiffusionLoraLoaderMixin)Kandinsky3UNetVQModel) DDPMScheduler) deprecateloggingreplace_example_docstring) randn_tensor)DiffusionPipelineImagePipelineOutputa? Examples: ```py >>> from diffusers import AutoPipelineForImage2Image >>> from diffusers.utils import load_image >>> import torch >>> pipe = AutoPipelineForImage2Image.from_pretrained( ... "kandinsky-community/kandinsky-3", variant="fp16", torch_dtype=torch.float16 ... ) >>> pipe.enable_model_cpu_offload() >>> prompt = "A painting of the inside of a subway train with tiny raccoons." >>> image = load_image( ... "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky3/t2i.png" ... ) >>> generator = torch.Generator(device="cpu").manual_seed(0) >>> image = pipe(prompt, image=image, strength=0.75, num_inference_steps=25, generator=generator).images[0] ``` cCsX||d}||ddkr|d7}||d}||ddkr$|d7}||||fS)Nrr)heightwidth scale_factor new_height new_widthrrx/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/pipelines/kandinsky3/pipeline_kandinsky3_img2img.pydownscale_height_and_width/s  rcCsHt|d}|tjdd}t|gd}t|d}|S)NRGBg_@r)rrrr) nparrayconvertastypefloat32 transposetorch from_numpy unsqueeze) pil_imagearrimagerrr prepare_image9s r,c%seZdZdZgdZdedededede f fdd Z d d Z d d Z e          d7dee jdee jdee jdee jfddZd8ddZddZ      d9ddZeddZed d!Zed"d#Ze eeddd$d%d&dddddddd'ddd(gfd)eeeefd*ee jejjee jeejjfd+e d,e!d-e d.eeeeefd/ee!d0eee j"ee j"fdee jdee jdee jdee jd1eed2e#d3ee$e!e!e%gdfd4eef d5d6Z&Z'S):Kandinsky3Img2ImgPipelineztext_encoder->movq->unet->movq)latents prompt_embedsnegative_prompt_embedsnegative_attention_maskattention_mask tokenizer text_encoderunet schedulermovqcs"t|j|||||ddS)N)r3r4r5r6r7)super__init__register_modules)selfr3r4r5r6r7 __class__rrr9Ks   z"Kandinsky3Img2ImgPipeline.__init__cCs<tt|||}t||d}|jj|d}|||fS)Nr)minintmaxr6 timesteps)r;num_inference_stepsstrengthdevice init_timestept_startrArrr get_timestepsYs z'Kandinsky3Img2ImgPipeline.get_timestepscCs`|r,t||dk||dk<|dd}|ddd|f}|ddd|f}||fS)Nrr)r& zeros_likesumr@)r; embeddingsr2 cut_contextmax_seq_lengthrrr_process_embedsbs z)Kandinsky3Img2ImgPipeline._process_embedsTrNFr/r0r2r1c  Cs|dur|durt|t|urtdt|dt|d|dur&|j}|dur2t|tr2d} n|dur@t|tr@t|} n|jd} d} |dur||j|d| d d d } | j |}| j |} |j || d }|d}| || |\}} || d }|j dur|j j}nd}|j ||d}|j\}}}|d|d}||||d}| |d} |r4|dur4|durdg| }n$t|tr|g}n| t|krtd|dt|d|d| d |}|dur*|j|ddd d d d}|j |}|j |} |j || d }|d}|ddd|jdf}| ddd|jdf} || d }n t|}t| } |ra|jd}|j ||d}|j|jkr`|d|d}|| ||d}| |d} nd}d} ||| | fS)aY Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*): prompt to be encoded device: (`torch.device`, *optional*): torch device to place the resulting embeddings on num_images_per_prompt (`int`, *optional*, defaults to 1): number of images that should be generated per prompt do_classifier_free_guidance (`bool`, *optional*, defaults to `True`): whether to use classifier free guidance or not negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds`. instead. If not defined, one has to pass `negative_prompt_embeds`. instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. attention_mask (`torch.Tensor`, *optional*): Pre-generated attention mask. Must provide if passing `prompt_embeds` directly. negative_attention_mask (`torch.Tensor`, *optional*): Pre-generated negative attention mask. Must provide if passing `negative_prompt_embeds` directly. Nz?`negative_prompt` should be the same type to `prompt`, but got z != .rr max_lengthTpt)paddingrQ truncationreturn_tensors)r2rdtyperDrHz`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)rSrQrTreturn_attention_maskrU)type TypeError_execution_device isinstancestrlistlenshaper3 input_idstor2r4rNr(rWrepeatview ValueErrorr&rI)r;promptdo_classifier_free_guidancenum_images_per_promptrDnegative_promptr/r0 _cut_contextr2r1 batch_sizerQ text_inputstext_input_idsrWbs_embedseq_len_ uncond_tokens uncond_inputrrr encode_promptks*                    z'Kandinsky3Img2ImgPipeline.encode_promptc sttjtjjtfstdtj||d||}j ddkr)}nCttrAt |krAtdt d|dttr[fdd t |D}tj |d d }n j j}j jj|}tj |gd d }|j } t| ||d } j|| |}|} | S) NzK`image` has to be of type `torch.Tensor`, `PIL.Image.Image` or list but is )rDrWrz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.cs0g|]}j||dj|qS)r)r7encode latent_distsample.0i generatorr+r;rr s$z=Kandinsky3Img2ImgPipeline.prepare_latents..rdim)r}rDrW)r]r&TensorPILImager_rfrZrcrar`rangecatr7rvrwrxconfigscaling_factorrr6 add_noise) r;r+timesteprlrirWrDr} init_latentsranoiser.rr|rprepare_latentss6   z)Kandinsky3Img2ImgPipeline.prepare_latentscCsXdtt|jjjv}i}|r||d<dtt|jjjv}|r*||d<|S)Netar})setinspect signaturer6step parameterskeys)r;r}r accepts_etaextra_step_kwargsaccepts_generatorrrrprepare_extra_step_kwargs,sz3Kandinsky3Img2ImgPipeline.prepare_extra_step_kwargsc s |durt|tr |dkrtd|dt|d|dur;tfdd|Ds;tdjdfd d |D|durN|durNtd |d |d |durZ|durZtd|durqt|tsqt|tsqtdt||dur|durtd|d|d |dur|dur|j|jkrtd|jd|jd|dur|durtd|dur|dur|jdd|jkrtd|jddd|jd|dur|durtd|dur|dur|jdd|jkrtd|jddd|jddSdSdS)Nrz5`callback_steps` has to be a positive integer but is z of type rOc3|]}|jvVqdSN_callback_tensor_inputsrzkr;rr N  z9Kandinsky3Img2ImgPipeline.check_inputs..2`callback_on_step_end_tensor_inputs` has to be in , but found cg|] }|jvr|qSrrrrrrr~Rz:Kandinsky3Img2ImgPipeline.check_inputs..zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` zLPlease provide `negative_attention_mask` along with `negative_prompt_embeds`rz`negative_prompt_embeds` and `negative_attention_mask` must have the same batch_size and token length when passed directly, but got: `negative_prompt_embeds` z != `negative_attention_mask` z:Please provide `attention_mask` along with `prompt_embeds`z`prompt_embeds` and `attention_mask` must have the same batch_size and token length when passed directly, but got: `prompt_embeds` z != `attention_mask` ) r]r?rfrZallrr^r_ra) r;rgcallback_stepsrjr/r0"callback_on_step_end_tensor_inputsr2r1rrr check_inputs=sz    z&Kandinsky3Img2ImgPipeline.check_inputscC|jSr_guidance_scalerrrrguidance_scalez(Kandinsky3Img2ImgPipeline.guidance_scalecCs |jdkS)Nrrrrrrrhs z5Kandinsky3Img2ImgPipeline.do_classifier_free_guidancecCrr)_num_timestepsrrrr num_timestepsrz'Kandinsky3Img2ImgPipeline.num_timestepsg333333?g@pilr.rgr+rCrBrrjrir} output_type return_dictcallback_on_step_endrc& s|dd}|dd}|durtddd|dur tddd|durAtfdd|DsAtd jd fd d |Dd }|||| | || | |_|dur^t|tr^d}n|durlt|t rlt |}n| j d}j }j |j|||| | || | d \} } } } jrt| | g} t| | g} t|t s|g}tdd|Dstddd |Ddtjdd |Ddd}|j| j|d}jj||d|||\}}j|d}|j|dd}|dd||}||||| j||}tdrjdurjt ||jj}t |_ j!|d}t"|D]\}}jr@t|gdn|}j#||| | dd}jrb|$d\} }!|d|!|| }jj%||||dj&}|duri}"|D] }#t'|#|"|#<qv||||"}$|$d|}|$d | } |$d!| } |$d"| } |$d#| } |t |dks|d|kr|djjdkr|(|dur||dkr|t)jd$d}%||%||q0| d%vrtd&| | d'ks'jj*|d d(d)}| d*vr|d+d+}|+dd}|,-ddd,d./}| d-kr&0|}n|}1|s:|fWdSt2|d.WdS1sJwYdS)/a# Function invoked when calling the pipeline for generation. Args: prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`. instead. image (`torch.Tensor`, `PIL.Image.Image`, `np.ndarray`, `List[torch.Tensor]`, `List[PIL.Image.Image]`, or `List[np.ndarray]`): `Image`, or tensor representing an image batch, that will be used as the starting point for the process. strength (`float`, *optional*, defaults to 0.8): Indicates extent to transform the reference `image`. Must be between 0 and 1. `image` is used as a starting point and more noise is added the higher the `strength`. The number of denoising steps depends on the amount of noise initially added. When `strength` is 1, added noise is maximum and the denoising process runs for the full number of iterations specified in `num_inference_steps`. A value of 1 essentially ignores `image`. num_inference_steps (`int`, *optional*, defaults to 50): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. guidance_scale (`float`, *optional*, defaults to 3.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. attention_mask (`torch.Tensor`, *optional*): Pre-generated attention mask. Must provide if passing `prompt_embeds` directly. negative_attention_mask (`torch.Tensor`, *optional*): Pre-generated negative attention mask. Must provide if passing `negative_prompt_embeds` directly. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.stable_diffusion.IFPipelineOutput`] instead of a plain tuple. callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. Examples: Returns: [`~pipelines.ImagePipelineOutput`] or `tuple` callbackNrz1.0.0zhPassing `callback` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`znPassing `callback_steps` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`c3rrrrrrrrrz5Kandinsky3Img2ImgPipeline.__call__..rrcrrrrrrrr~rz6Kandinsky3Img2ImgPipeline.__call__..Trr)rirDrjr/r0rkr2r1css$|] }t|tjjtjfVqdSr)r]rrr&rryrrrr+s"zInput is in incorrect format: cSg|]}t|qSr)rZryrrrr~-z:. Currently, we only support PIL image and pytorch tensorcSrr)r,ryrrrr~0rrrV)rDr.text_encoder_offload_hook)totalr)encoder_hidden_statesencoder_attention_maskg?)r}r/r0r2r1order)rRr rlatentzSOnly the output types `pt`, `pil`, `np` and `latent` are supported not output_type=r)force_not_quantizerx)r rg?r r)images)3poprrrfrrrr]r^r_r`rar\rtrhr&rboolrcrWr6 set_timestepsrGr7rvrepeat_interleaverdrhasattrroffloadrr progress_bar enumerater5chunkr prev_samplelocalsupdategetattrdecodeclampcpupermutefloatnumpy numpy_to_pilmaybe_free_model_hooksr)&r;rgr+rCrBrrjrir}r/r0r2r1rrrrkwargsrrrLrlrDrAr.latent_timestepnum_warmup_stepsrr{tlatent_model_input noise_prednoise_pred_uncondnoise_pred_textcallback_kwargsrcallback_outputsstep_idxrrr__call__s X              6        A&z"Kandinsky3Img2ImgPipeline.__call__) TrNNNNFNNr)NNNNNN)(__name__ __module__ __qualname__model_cpu_offload_seqrrrr r r r9rGrNr&no_gradrrrtrrrpropertyrrhrrEXAMPLE_DOC_STRINGrr^rrrrr? Generatorrrrr __classcell__rrr<rr-As     + H   "     r-)r)&rtypingrrrrrrr r PIL.Imager& transformersrrloadersr modelsr r schedulersr utilsrrrutils.torch_utilsrpipeline_utilsrr get_loggerrloggerrrr,r-rrrrs$