o ۷iVU@sddlmZddlZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZmZmZdd lmZd d lmZmZd d lmZerUddlmmZdZndZeeZ dZ!dddZ"GdddeZ#dS))CallableN)XLMRobertaTokenizer)VaeImageProcessor)UNet2DConditionModelVQModel) DDIMScheduler)is_torch_xla_availableloggingreplace_example_docstring) randn_tensor)DiffusionPipelineImagePipelineOutput)MultilingualCLIPTFa Examples: ```py >>> from diffusers import KandinskyImg2ImgPipeline, KandinskyPriorPipeline >>> from diffusers.utils import load_image >>> import torch >>> pipe_prior = KandinskyPriorPipeline.from_pretrained( ... "kandinsky-community/kandinsky-2-1-prior", torch_dtype=torch.float16 ... ) >>> pipe_prior.to("cuda") >>> prompt = "A red cartoon frog, 4k" >>> image_emb, zero_image_emb = pipe_prior(prompt, return_dict=False) >>> pipe = KandinskyImg2ImgPipeline.from_pretrained( ... "kandinsky-community/kandinsky-2-1", torch_dtype=torch.float16 ... ) >>> pipe.to("cuda") >>> init_image = load_image( ... "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main" ... "/kandinsky/frog.png" ... ) >>> image = pipe( ... prompt, ... image=init_image, ... image_embeds=image_emb, ... negative_image_embeds=zero_image_emb, ... height=768, ... width=768, ... num_inference_steps=100, ... strength=0.2, ... ).images >>> image[0].save("red_frog.png") ``` cCsX||d}||ddkr|d7}||d}||ddkr$|d7}||||fS)Nr rr)hw scale_factornew_hnew_wrrn/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/pipelines/kandinsky/pipeline_kandinsky_img2img.py get_new_h_wSs  rc#sLeZdZdZdZdedededede f fdd Z d d Z d d Z d0ddZ dejdejdejdejfddZeee            d1deeeBdejejjBeejBeejjBd ejd!ejd"eeeBdBd#ed$ed%ed&ed'ed(ed)ejeejBdBd*edBd+eeeejgdfdBd,ed-ef d.d/ZZS)2KandinskyImg2ImgPipelinea Pipeline for image-to-image generation using Kandinsky This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: text_encoder ([`MultilingualCLIP`]): Frozen text-encoder. tokenizer ([`XLMRobertaTokenizer`]): Tokenizer of class scheduler ([`DDIMScheduler`]): A scheduler to be used in combination with `unet` to generate image latents. unet ([`UNet2DConditionModel`]): Conditional U-Net architecture to denoise the image embedding. movq ([`VQModel`]): MoVQ image encoder and decoder ztext_encoder->unet->movq text_encodermovq tokenizerunet schedulercsxt|j|||||dt|ddr dt|jjjdnd|_t|ddr.|jjj nd}t |j|ddd|_ dS) N)rrrr rrr rrbicubic)vae_scale_factorvae_latent_channelsresample reducing_gap) super__init__register_modulesgetattrlenrconfigblock_out_channelsmovq_scale_factorlatent_channelsrimage_processor)selfrrrrr movq_latent_channels __class__rrr(ss" $ z!KandinskyImg2ImgPipeline.__init__cCs<tt|||}t||d}|jj|d}|||fS)Nr)minintmaxr timesteps)r1num_inference_stepsstrengthdevice init_timestept_startr8rrr get_timestepss z&KandinskyImg2ImgPipeline.get_timestepsc Csv|dur t||||d}n|j|krtd|jd|||}||j}|j}t||||d}||||}|S)N) generatorr;dtypezUnexpected latents shape, got z , expected )r shape ValueErrortoinit_noise_sigma add_noise) r1latentslatent_timesteprAr@r;r?r noiserrrprepare_latentss   z(KandinskyImg2ImgPipeline.prepare_latentsNc CsZt|tr t|nd}|j|ddddddd}|j}|j|dddj} | jd |jd krRt|| sR|j| dd|jj dd f} t d |jj d | | |}|j |} |j|| d \} } | j|d d} | j|d d} | j|d d} |r(|durdg|}n;t|t|urtdt|dt|dt|tr|g}n|t|krtd|dt|d|d|d |}|j|ddddddd}|j |}|j |}|j||d \}}|jd}|d|}||||}|jd}|d|d}||||d }|j|d d}t|| g} t|| g} t|| g} | | | fS)Nr max_lengthMTpt)paddingrJ truncationreturn_attention_maskadd_special_tokensreturn_tensorslongest)rMrQz\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: ) input_idsattention_maskrdimz?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.) isinstancelistr+rrTrAtorchequal batch_decodemodel_max_lengthloggerwarningrCrUrrepeat_interleavetype TypeErrorstrrBrepeatviewcat)r1promptr;num_images_per_promptdo_classifier_free_guidancenegative_prompt batch_size text_inputstext_input_idsuntruncated_ids removed_text text_mask prompt_embedstext_encoder_hidden_states uncond_tokens uncond_inputuncond_text_input_idsuncond_text_masknegative_prompt_embeds!uncond_text_encoder_hidden_statesseq_lenrrr_encode_prompts  $             z'KandinskyImg2ImgPipeline._encode_promptoriginal_samplesrHr8returnc Cstjdddtjd}d|}tj|dd}|j|j|jd}||j}||d }|}t|j t|j krG| d }t|j t|j ks8d ||d }|}t|j t|j krl| d }t|j t|j ks]||||} | S) Ng-C6?g{Gz?i)r@?rrV)r;r@g?rSr) r\linspacefloat32cumprodrCr;r@flattenr+rA unsqueeze) r1r}rHr8betasalphasalphas_cumprodsqrt_alpha_prodsqrt_one_minus_alpha_prod noisy_samplesrrrrEs"    z"KandinskyImg2ImgPipeline.add_noised333333?@rpilTriimage image_embedsnegative_image_embedsrlheightwidthr9r:guidance_scalerjr? output_typecallbackcallback_steps return_dictc% st|trd}nt|trt|}n tdt|j}|| }| dk}||| ||\}}}t|tr>tj |dd}t|trJtj |dd}|ri|j | dd}|j | dd}tj ||gddj |j |d}t|tsq|g}t dd|Dstd d d |Dd tj fd d |Ddd}|j |j |d}j|d}|j | dd}jj||d|| |\}}tjjj| d}tj|g||j |d}jjj}tj\||||f|j || j}t|D]\}}|r t |gdn|}||d}j||||ddd}|rL|j|jddd\}} | d\}!}"| d\}}#|!| |"|!}tj ||#gdd}t!jjdr\jjj"dvsh|j|jddd\}}jj#|||| dj$}|dur||dkr|t%jdd}$||$||t&rt'(qjj)|ddd}*| dvrtd| j+,|| }|s|fSt-|dS)a Function invoked when calling the pipeline for generation. Args: prompt (`str` or `list[str]`): The prompt or prompts to guide the image generation. image (`torch.Tensor`, `PIL.Image.Image`): `Image`, or tensor representing an image batch, that will be used as the starting point for the process. image_embeds (`torch.Tensor` or `list[torch.Tensor]`): The clip image embeddings for text prompt, that will be used to condition the image generation. negative_image_embeds (`torch.Tensor` or `list[torch.Tensor]`): The clip image embeddings for negative text prompt, will be used to condition the image generation. negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). height (`int`, *optional*, defaults to 512): The height in pixels of the generated image. width (`int`, *optional*, defaults to 512): The width in pixels of the generated image. num_inference_steps (`int`, *optional*, defaults to 100): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. strength (`float`, *optional*, defaults to 0.3): Conceptually, indicates how much to transform the reference `image`. Must be between 0 and 1. `image` will be used as a starting point, adding more noise to it the larger the `strength`. The number of denoising steps depends on the amount of noise initially added. When `strength` is 1, added noise will be maximum and the denoising process will run for the full number of iterations specified in `num_inference_steps`. A value of 1, therefore, essentially ignores `image`. guidance_scale (`float`, *optional*, defaults to 4.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`torch.Generator` or `list[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between: `"pil"` (`PIL.Image.Image`), `"np"` (`np.array`) or `"pt"` (`torch.Tensor`). callback (`Callable`, *optional*): A function that calls every `callback_steps` steps during inference. The function is called with the following arguments: `callback(step: int, timestep: int, latents: torch.Tensor)`. callback_steps (`int`, *optional*, defaults to 1): The frequency at which the `callback` function is called. If not specified, the callback is called at every step. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.ImagePipelineOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.ImagePipelineOutput`] or `tuple` rz2`prompt` has to be of type `str` or `list` but is rrrV)r@r;css$|] }t|tjjtjfVqdSN)rZPILImager\Tensor.0irrr s"z4KandinskyImg2ImgPipeline.__call__..zInput is in incorrect format: cSsg|]}t|qSr)rcrrrr sz5KandinskyImg2ImgPipeline.__call__..z:. Currently, we only support PIL image and pytorch tensorcsg|] }j|qSr)r0 preprocessrrr1rrrrsrF)r;r ) text_embedsrF)sampletimestepencoder_hidden_statesadded_cond_kwargsr variance_type)learned learned_range)r?NorderT)force_not_quantizer)rLnprzIOnly the output types `pt`, `pil` and `np` are supported not output_type=)images).rZrer[r+rBrc_execution_devicer|r\rhrbrCr@allrencoder set_timestepsr>r6r,num_train_timestepstensorr in_channelsrr.rI enumerate progress_barsplitrAchunkhasattrrstep prev_sampler* XLA_AVAILABLExm mark_stepdecodemaybe_free_model_hooksr0 postprocessr)%r1rirrrrlrrr9r:rrjr?rrrrrmr;rkrsrt_rFtimesteps_tensorrGnum_channels_latentsrtlatent_model_inputr noise_pred variance_prednoise_pred_uncondnoise_pred_textvariance_pred_textstep_idxrrr__call__)s O               z!KandinskyImg2ImgPipeline.__call__r) NrrrrrrNrNrT) __name__ __module__ __qualname____doc__model_cpu_offload_seqrrrrrr(r>rIr|r\r IntTensorrEno_gradr EXAMPLE_DOC_STRINGrer[rrr6float Generatorrboolr __classcell__rrr3rr]s  g  "     r)r)$typingr PIL.Imagerr\ transformersrr0rmodelsrr schedulersrutilsr r r utils.torch_utilsr pipeline_utilsrrrrtorch_xla.core.xla_modelcore xla_modelrr get_loggerrr`rrrrrrrs&        )