o Giv@sddlZddlmZddlZddlZddlZddlmZmZddl m Z ddl m Z ddl mZmZddlmZdd lmZmZmZmZdd lmZd d lmZmZeraddlmmZd Z ndZ e!e"Z#dZ$Gdddee Z%dS)N)Callable)T5EncoderModel T5Tokenizer)VaeImageProcessor)StableDiffusionLoraLoaderMixin)Kandinsky3UNetVQModel) DDPMScheduler) deprecateis_torch_xla_availableloggingreplace_example_docstring) randn_tensor)DiffusionPipelineImagePipelineOutputTFa? Examples: ```py >>> from diffusers import AutoPipelineForImage2Image >>> from diffusers.utils import load_image >>> import torch >>> pipe = AutoPipelineForImage2Image.from_pretrained( ... "kandinsky-community/kandinsky-3", variant="fp16", torch_dtype=torch.float16 ... ) >>> pipe.enable_model_cpu_offload() >>> prompt = "A painting of the inside of a subway train with tiny raccoons." >>> image = load_image( ... "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky3/t2i.png" ... ) >>> generator = torch.Generator(device="cpu").manual_seed(0) >>> image = pipe(prompt, image=image, strength=0.75, num_inference_steps=25, generator=generator).images[0] ``` c$seZdZdZgdZdedededede f fdd Z d d Z d d Z e          d6de jdBde jdBde jdBde jdBfddZd7ddZddZ      d8ddZeddZedd Zed!d"Ze eeddd#d$d%dddddddd&ddd'gfd(eeeBd)e jejjBee jBeejjBd*ed+ed,ed-eeeBdBd.edBd/e j ee j BdBde jdBde jdBde jdBde jdBd0edBd1e!d2e"eegdfdBd3eef d4d5Z#Z$S)9Kandinsky3Img2ImgPipelineztext_encoder->movq->unet->movq)latents prompt_embedsnegative_prompt_embedsnegative_attention_maskattention_mask tokenizer text_encoderunet schedulermovqcstt|j|||||dt|ddr dt|jjjdnd}t|ddr-|jjjnd}t ||ddd|_ dS) N)rrrrrrrbicubic)vae_scale_factorvae_latent_channelsresample reducing_gap) super__init__register_modulesgetattrlenrconfigblock_out_channelslatent_channelsrimage_processor)selfrrrrrmovq_scale_factormovq_latent_channels __class__n/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/kandinsky3/pipeline_kandinsky3_img2img.pyr'Bs  & z"Kandinsky3Img2ImgPipeline.__init__cCs<tt|||}t||d}|jj|d}|||fS)Nr)minintmaxr timesteps)r/num_inference_stepsstrengthdevice init_timestept_startr9r4r4r5 get_timestepsXs z'Kandinsky3Img2ImgPipeline.get_timestepscCs`|r,t||dk||dk<|dd}|ddd|f}|ddd|f}||fS)Nrr)torch zeros_likesumr8)r/ embeddingsr cut_contextmax_seq_lengthr4r4r5_process_embedsas z)Kandinsky3Img2ImgPipeline._process_embedsTrNrrrrc  Cs|dur|durt|t|urtdt|dt|d|dur&|j}|dur2t|tr2d} n|dur@t|tr@t|} n|jd} d} |dur||j|d| d d d } | j |}| j |} |j || d }|d}| || |\}} || d }|j dur|j j}nd}|j ||d}|j\}}}|d|d}||||d}| |d} |r4|dur4|durdg| }n$t|tr|g}n| t|krtd|dt|d|d| d |}|dur*|j|ddd d d d}|j |}|j |} |j || d }|d}|ddd|jdf}| ddd|jdf} || d }n t|}t| } |ra|jd}|j ||d}|j|jkr`|d|d}|| ||d}| |d} nd}d} ||| | fS)aY Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `list[str]`, *optional*): prompt to be encoded device: (`torch.device`, *optional*): torch device to place the resulting embeddings on num_images_per_prompt (`int`, *optional*, defaults to 1): number of images that should be generated per prompt do_classifier_free_guidance (`bool`, *optional*, defaults to `True`): whether to use classifier free guidance or not negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds`. instead. If not defined, one has to pass `negative_prompt_embeds`. instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. attention_mask (`torch.Tensor`, *optional*): Pre-generated attention mask. Must provide if passing `prompt_embeds` directly. negative_attention_mask (`torch.Tensor`, *optional*): Pre-generated negative attention mask. Must provide if passing `negative_prompt_embeds` directly. Nz?`negative_prompt` should be the same type to `prompt`, but got z != .rr max_lengthTpt)paddingrJ truncationreturn_tensors)rrdtyper<r@z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)rLrJrMreturn_attention_maskrN)type TypeError_execution_device isinstancestrlistr*shaper input_idstorrrG unsqueezerPrepeatview ValueErrorrArB)r/promptdo_classifier_free_guidancenum_images_per_promptr<negative_promptrr _cut_contextrr batch_sizerJ text_inputstext_input_idsrPbs_embedseq_len_ uncond_tokens uncond_inputr4r4r5 encode_promptjs*                    z'Kandinsky3Img2ImgPipeline.encode_promptc sttjtjjtfstdtj||d||}j ddkr)}nCttrAt |krAtdt d|dttr[fdd t |D}tj |d d }n j j}j jj|}tj |gd d }|j } t| ||d } j|| |}|} | S) NzK`image` has to be of type `torch.Tensor`, `PIL.Image.Image` or list but is )r<rPrr z/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.cs0g|]}j||dj|qS)r)rencode latent_distsample.0i generatorimager/r4r5 s$z=Kandinsky3Img2ImgPipeline.prepare_latents..rdim)rur<rP)rVrATensorPILImagerXr_rSr[rYr*rangecatrrnrorpr+scaling_factorrr add_noise) r/rvtimesteprerbrPr<ru init_latentsrYnoiserr4rtr5prepare_latentss6   z)Kandinsky3Img2ImgPipeline.prepare_latentscCsXdtt|jjjv}i}|r||d<dtt|jjjv}|r*||d<|S)Netaru)setinspect signaturerstep parameterskeys)r/rur accepts_etaextra_step_kwargsaccepts_generatorr4r4r5prepare_extra_step_kwargs+sz3Kandinsky3Img2ImgPipeline.prepare_extra_step_kwargsc s |durt|tr |dkrtd|dt|d|dur;tfdd|Ds;tdjdfd d |D|durN|durNtd |d |d |durZ|durZtd|durqt|tsqt|tsqtdt||dur|durtd|d|d |dur|dur|j|jkrtd|jd|jd|dur|durtd|dur|dur|jdd|jkrtd|jddd|jd|dur|durtd|dur|dur|jdd|jkrtd|jddd|jddSdSdS)Nrz5`callback_steps` has to be a positive integer but is z of type rHc3|]}|jvVqdSN_callback_tensor_inputsrrkr/r4r5 M  z9Kandinsky3Img2ImgPipeline.check_inputs..2`callback_on_step_end_tensor_inputs` has to be in , but found cg|] }|jvr|qSr4rrrr4r5rwQz:Kandinsky3Img2ImgPipeline.check_inputs..zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` zLPlease provide `negative_attention_mask` along with `negative_prompt_embeds`rz`negative_prompt_embeds` and `negative_attention_mask` must have the same batch_size and token length when passed directly, but got: `negative_prompt_embeds` z != `negative_attention_mask` z:Please provide `attention_mask` along with `prompt_embeds`z`prompt_embeds` and `attention_mask` must have the same batch_size and token length when passed directly, but got: `prompt_embeds` z != `attention_mask` ) rVr7r_rSallrrWrXrY) r/r`callback_stepsrcrr"callback_on_step_end_tensor_inputsrrr4rr5 check_inputs<sz    z&Kandinsky3Img2ImgPipeline.check_inputscC|jSr_guidance_scalerr4r4r5guidance_scalez(Kandinsky3Img2ImgPipeline.guidance_scalecCs |jdkS)Nrrrr4r4r5ras z5Kandinsky3Img2ImgPipeline.do_classifier_free_guidancecCrr)_num_timestepsrr4r4r5 num_timestepsrz'Kandinsky3Img2ImgPipeline.num_timestepsg333333?g@pilrr`rvr;r:rrcrbru output_type return_dictcallback_on_step_endrc& sV|dd}|dd}|durtddd|dur tddd|durAtfdd|DsAtd jd fd d |Dd }|||| | || | |_|dur^t|tr^d}n|durlt|t rlt |}n| j d}j }j |j|||| | || | d \} } } } jrt| | g} t| | g} t|t s|g}tdd|Dstddd |Ddtjfdd |Ddd}|j| j|d}jj||d|||\}}j|d}|j|dd}|dd||}||||| j||}tdrjdurjt ||jj}t |_ j!|d}t"|D]\}}jrAt|gdn|}j#||| | dd}jrc|$d\} }!|d|!|| }jj%||||dj&}|duri}"|D] }#t'|#|"|#<qw||||"}$|$d|}|$d | } |$d!| } |$d"| } |$d#| } |t |dks|d|kr|djjdkr|(|dur||dkr|t)jd$d}%||%||t*rt+,q1| d%ksjj-|d d&d'}j./|| }n|}0|s|fWdSt1|d(WdS1s$wYdS))a/ Function invoked when calling the pipeline for generation. Args: prompt (`str` or `list[str]`, *optional*): The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`. instead. image (`torch.Tensor`, `PIL.Image.Image`, `np.ndarray`, `list[torch.Tensor]`, `list[PIL.Image.Image]`, or `list[np.ndarray]`): `Image`, or tensor representing an image batch, that will be used as the starting point for the process. strength (`float`, *optional*, defaults to 0.8): Indicates extent to transform the reference `image`. Must be between 0 and 1. `image` is used as a starting point and more noise is added the higher the `strength`. The number of denoising steps depends on the amount of noise initially added. When `strength` is 1, added noise is maximum and the denoising process runs for the full number of iterations specified in `num_inference_steps`. A value of 1 essentially ignores `image`. num_inference_steps (`int`, *optional*, defaults to 50): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. guidance_scale (`float`, *optional*, defaults to 3.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`torch.Generator` or `list[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. attention_mask (`torch.Tensor`, *optional*): Pre-generated attention mask. Must provide if passing `prompt_embeds` directly. negative_attention_mask (`torch.Tensor`, *optional*): Pre-generated negative attention mask. Must provide if passing `negative_prompt_embeds` directly. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.stable_diffusion.IFPipelineOutput`] instead of a plain tuple. callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`list`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. Examples: Returns: [`~pipelines.ImagePipelineOutput`] or `tuple` callbackNrz1.0.0zhPassing `callback` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`znPassing `callback_steps` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`c3rrrrrr4r5rrz5Kandinsky3Img2ImgPipeline.__call__..rrcrr4rrrr4r5rwrz6Kandinsky3Img2ImgPipeline.__call__..Trr)rbr<rcrrrdrrcss$|] }t|tjjtjfVqdSr)rVr{r|rArzrqr4r4r5r*s"zInput is in incorrect format: cSsg|]}t|qSr4)rSrqr4r4r5rw,sz:. Currently, we only support PIL image and pytorch tensorcsg|]}j|qSr4)r. preprocessrqrr4r5rw/srxrO)r<rtext_encoder_offload_hook)totalr)encoder_hidden_statesencoder_attention_maskg?)rurrrrorderlatent)force_not_quantizerp)images)2popr rr_rrrrVrWrXr*rYrUrmrarAr~boolr[rPr set_timestepsr?rrnrepeat_interleaver]rhasattrroffloadrr progress_bar enumeraterchunkr prev_samplelocalsupdater) XLA_AVAILABLExm mark_stepdecoder. postprocessmaybe_free_model_hooksr)&r/r`rvr;r:rrcrbrurrrrrrrrkwargsrrrErer<r9rlatent_timestepnum_warmup_stepsrrstlatent_model_input noise_prednoise_pred_uncondnoise_pred_textcallback_kwargsrcallback_outputsstep_idxr4rr5__call__s X              6  9&z"Kandinsky3Img2ImgPipeline.__call__) TrNNNNTNNr)NNNNNN)%__name__ __module__ __qualname__model_cpu_offload_seqrrrrr r r'r?rGrAno_gradrzrmrrrpropertyrrarrEXAMPLE_DOC_STRINGrWrXr{r|floatr7 Generatorrrr __classcell__r4r4r2r5r8s     + H    "     r)&rtypingrr{ PIL.ImagerA transformersrrr.rloadersrmodelsrr schedulersr utilsr r r rutils.torch_utilsrpipeline_utilsrrtorch_xla.core.xla_modelcore xla_modelrr get_loggerrloggerrrr4r4r4r5s(