o piU@sddlmZmZmZmZddlZddlZddl Z ddlm Z ddl m Z ddl mZmZddlmZddlmZmZdd lmZd d lmZmZd d lmZeeZdZdddZ dddZ!GdddeZ"dS))CallableListOptionalUnionN)Image)XLMRobertaTokenizer)UNet2DConditionModelVQModel) DDIMScheduler)loggingreplace_example_docstring) randn_tensor)DiffusionPipelineImagePipelineOutput)MultilingualCLIPa Examples: ```py >>> from diffusers import KandinskyImg2ImgPipeline, KandinskyPriorPipeline >>> from diffusers.utils import load_image >>> import torch >>> pipe_prior = KandinskyPriorPipeline.from_pretrained( ... "kandinsky-community/kandinsky-2-1-prior", torch_dtype=torch.float16 ... ) >>> pipe_prior.to("cuda") >>> prompt = "A red cartoon frog, 4k" >>> image_emb, zero_image_emb = pipe_prior(prompt, return_dict=False) >>> pipe = KandinskyImg2ImgPipeline.from_pretrained( ... "kandinsky-community/kandinsky-2-1", torch_dtype=torch.float16 ... ) >>> pipe.to("cuda") >>> init_image = load_image( ... "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main" ... "/kandinsky/frog.png" ... ) >>> image = pipe( ... prompt, ... image=init_image, ... image_embeds=image_emb, ... negative_image_embeds=zero_image_emb, ... height=768, ... width=768, ... num_inference_steps=100, ... strength=0.2, ... ).images >>> image[0].save("red_frog.png") ``` cCsX||d}||ddkr|d7}||d}||ddkr$|d7}||||fS)Nrrr)hw scale_factornew_hnew_wrrv/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/pipelines/kandinsky/pipeline_kandinsky_img2img.py get_new_h_wOs  rcCs^|j||ftjdd}t|d}|tjdd}t|gd}t | d}|S)Nr)resample reducing_gapRGBg_@)rrrr) resizerBICUBICnparrayconvertastypefloat32 transposetorch from_numpy unsqueeze) pil_imagerrarrimagerrr prepare_imageYs r/c#sXeZdZdZdZdedededede f fdd Z d d Z d d Z d0ddZ dejdejdejdejfddZeee            d1deeeefdeejejjeejeejjfd ejd!ejd"eeeeefd#ed$ed%ed&ed'ed(ed)eeejeejfd*eed+eeeeejgdfd,ed-ef d.d/Z Z!S)2KandinskyImg2ImgPipelinea Pipeline for image-to-image generation using Kandinsky This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: text_encoder ([`MultilingualCLIP`]): Frozen text-encoder. tokenizer ([`XLMRobertaTokenizer`]): Tokenizer of class scheduler ([`DDIMScheduler`]): A scheduler to be used in combination with `unet` to generate image latents. unet ([`UNet2DConditionModel`]): Conditional U-Net architecture to denoise the image embedding. movq ([`VQModel`]): MoVQ image encoder and decoder ztext_encoder->unet->movq text_encodermovq tokenizerunet schedulercs:t|j|||||ddt|jjjd|_dS)N)r1r3r4r5r2rr)super__init__register_moduleslenr2configblock_out_channelsmovq_scale_factor)selfr1r2r3r4r5 __class__rrr7xs z!KandinskyImg2ImgPipeline.__init__cCs<tt|||}t||d}|jj|d}|||fS)Nr)minintmaxr5 timesteps)r=num_inference_stepsstrengthdevice init_timestept_startrCrrr get_timestepss z&KandinskyImg2ImgPipeline.get_timestepsc Csv|dur t||||d}n|j|krtd|jd|||}||j}|j}t||||d}||||}|S)N) generatorrFdtypezUnexpected latents shape, got z , expected )rshape ValueErrortoinit_noise_sigma add_noise) r=latentslatent_timesteprLrKrFrJr5noiserrrprepare_latentss   z(KandinskyImg2ImgPipeline.prepare_latentsNc CsZt|tr t|nd}|j|ddddddd}|j}|j|dddj} | jd |jd krRt|| sR|j| dd|jj dd f} t d |jj d | | |}|j |} |j|| d \} } | j|d d} | j|d d} | j|d d} |r(|durdg|}n;t|t|urtdt|dt|dt|tr|g}n|t|krtd|dt|d|d|d |}|j|ddddddd}|j |}|j |}|j||d \}}|jd}|d|}||||}|jd}|d|d}||||d }|j|d d}t|| g} t|| g} t|| g} | | | fS)Nr max_lengthMTpt)paddingrU truncationreturn_attention_maskadd_special_tokensreturn_tensorslongest)rXr\z\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: ) input_idsattention_maskrdimz?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.) isinstancelistr9r3r_rLr)equal batch_decodemodel_max_lengthloggerwarningrNr`r1repeat_interleavetype TypeErrorstrrMrepeatviewcat)r=promptrFnum_images_per_promptdo_classifier_free_guidancenegative_prompt batch_size text_inputstext_input_idsuntruncated_ids removed_text text_mask prompt_embedstext_encoder_hidden_states uncond_tokens uncond_inputuncond_text_input_idsuncond_text_masknegative_prompt_embeds!uncond_text_encoder_hidden_statesseq_lenrrr_encode_prompts  $             z'KandinskyImg2ImgPipeline._encode_promptoriginal_samplesrSrCreturnc Cstjdddtjd}d|}tj|dd}|j|j|jd}||j}||d }|}t|j t|j krG| d }t|j t|j ks8d ||d }|}t|j t|j krl| d }t|j t|j ks]||||} | S) Ng-C6?g{Gz?i)rK?rra)rFrK?r^r) r)linspacer'cumprodrNrFrKflattenr9rLr+) r=rrSrCbetasalphasalphas_cumprodsqrt_alpha_prodsqrt_one_minus_alpha_prod noisy_samplesrrrrP s"    z"KandinskyImg2ImgPipeline.add_noiserd333333?@rpilTrsr. image_embedsnegative_image_embedsrvheightwidthrDrEguidance_scalertrJ output_typecallbackcallback_steps return_dictc% st|trd}nt|trt|}n tdt||j}|| }| dk}|||| ||\}}}t|tr>tj |dd}t|trJtj |dd}|ri|j | dd}|j | dd}tj ||gddj |j |d}t|tsq|g}t dd|Dstd d d |Dd tj fd d |Ddd}|j |j |d}|j|d}|j | dd}|jj||d||| |\}}t|jjj| d}tj|g||j |d}|jjj}t|j\|||||f|j || |j}t||D]\}}|r t |gdn|}||d}|j||||ddd}|rK|j|jddd\}} | d\}!}"| d\}}#|!| |"|!}tj ||#gdd}t!|jjdr[|jjj"dvsg|j|jddd\}}|jj#|||| dj$}|dur||dkr|t%|jdd}$||$||q|jj&|ddd}|'| dvrtd| | dvr|dd}|(dd}|)*ddd d+,}| d!kr|-|}|s|fSt.|d"S)#a Function invoked when calling the pipeline for generation. Args: prompt (`str` or `List[str]`): The prompt or prompts to guide the image generation. image (`torch.Tensor`, `PIL.Image.Image`): `Image`, or tensor representing an image batch, that will be used as the starting point for the process. image_embeds (`torch.Tensor` or `List[torch.Tensor]`): The clip image embeddings for text prompt, that will be used to condition the image generation. negative_image_embeds (`torch.Tensor` or `List[torch.Tensor]`): The clip image embeddings for negative text prompt, will be used to condition the image generation. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). height (`int`, *optional*, defaults to 512): The height in pixels of the generated image. width (`int`, *optional*, defaults to 512): The width in pixels of the generated image. num_inference_steps (`int`, *optional*, defaults to 100): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. strength (`float`, *optional*, defaults to 0.3): Conceptually, indicates how much to transform the reference `image`. Must be between 0 and 1. `image` will be used as a starting point, adding more noise to it the larger the `strength`. The number of denoising steps depends on the amount of noise initially added. When `strength` is 1, added noise will be maximum and the denoising process will run for the full number of iterations specified in `num_inference_steps`. A value of 1, therefore, essentially ignores `image`. guidance_scale (`float`, *optional*, defaults to 4.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between: `"pil"` (`PIL.Image.Image`), `"np"` (`np.array`) or `"pt"` (`torch.Tensor`). callback (`Callable`, *optional*): A function that calls every `callback_steps` steps during inference. The function is called with the following arguments: `callback(step: int, timestep: int, latents: torch.Tensor)`. callback_steps (`int`, *optional*, defaults to 1): The frequency at which the `callback` function is called. If not specified, the callback is called at every step. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.ImagePipelineOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.ImagePipelineOutput`] or `tuple` rz2`prompt` has to be of type `str` or `list` but is rrra)rKrFcss$|] }t|tjjtjfVqdSN)rePILrr)Tensor.0irrr s"z4KandinskyImg2ImgPipeline.__call__..zInput is in incorrect format: cSsg|]}t|qSr)rmrrrr sz5KandinskyImg2ImgPipeline.__call__..z:. Currently, we only support PIL image and pytorch tensorcsg|]}t|qSr)r/rrrrrrsrQ)rFr) text_embedsrF)sampletimestepencoder_hidden_statesadded_cond_kwargsr variance_type)learned learned_range)rJNorderT)force_not_quantizer)rWr#rzIOnly the output types `pt`, `pil` and `np` are supported not output_type=)r#rrrr)images)/rerorfr9rMrm_execution_devicerr)rrrlrNrKallr2encoder5 set_timestepsrIrAr:num_train_timestepstensorr4 in_channelsrr<rT enumerate progress_barsplitrLchunkhasattrrstep prev_samplegetattrdecodemaybe_free_model_hooksclampcpupermutefloatnumpy numpy_to_pilr)%r=rsr.rrrvrrrDrErrtrJrrrrrwrFrur}r~_rQtimesteps_tensorrRnum_channels_latentsrtlatent_model_inputr noise_pred variance_prednoise_pred_uncondnoise_pred_textvariance_pred_textstep_idxrrr__call__%s O                   z!KandinskyImg2ImgPipeline.__call__r) NrrrrrrNrNrT)"__name__ __module__ __qualname____doc__model_cpu_offload_seqrr rr r r7rIrTrr)r IntTensorrPno_gradr EXAMPLE_DOC_STRINGrrorrrrrAr Generatorrboolr __classcell__rrr>rr0bs  g "     r0)r)rr)#typingrrrrrr# PIL.Imagerr)r transformersrmodelsr r schedulersr utilsr r utils.torch_utilsrpipeline_utilsrrr1r get_loggerrrjrrr/r0rrrrs"       )