o ۷iE@sddlmZddlZddlmZddlmZmZddlm Z m Z ddl m Z m Z mZddlmZd d lmZmZd d lmZe rMddlmmZd ZndZe eZdZdddZGdddeZ dS))CallableN)XLMRobertaTokenizer)UNet2DConditionModelVQModel) DDIMScheduler DDPMScheduler)is_torch_xla_availableloggingreplace_example_docstring) randn_tensor)DiffusionPipelineImagePipelineOutput)MultilingualCLIPTFav Examples: ```py >>> from diffusers import KandinskyPipeline, KandinskyPriorPipeline >>> import torch >>> pipe_prior = KandinskyPriorPipeline.from_pretrained("kandinsky-community/Kandinsky-2-1-prior") >>> pipe_prior.to("cuda") >>> prompt = "red cat, 4k photo" >>> out = pipe_prior(prompt) >>> image_emb = out.image_embeds >>> negative_image_emb = out.negative_image_embeds >>> pipe = KandinskyPipeline.from_pretrained("kandinsky-community/kandinsky-2-1") >>> pipe.to("cuda") >>> image = pipe( ... prompt, ... image_embeds=image_emb, ... negative_image_embeds=negative_image_emb, ... height=768, ... width=768, ... num_inference_steps=100, ... ).images >>> image[0].save("cat.png") ``` cCsX||d}||ddkr|d7}||d}||ddkr$|d7}||||fS)Nr rr)hw scale_factornew_hnew_wrrf/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/pipelines/kandinsky/pipeline_kandinsky.py get_new_h_wGs  rc!seZdZdZdZdedededee Bde f fdd Z d d Z d&d dZ eee        d'deeeBdejeejBdejeejBdeeeBd BdedededededejeejBd Bdejd Bd ed Bd!eeeejgd fd Bd"ed#efd$d%ZZS)(KandinskyPipelinea, Pipeline for text-to-image generation using Kandinsky This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: text_encoder ([`MultilingualCLIP`]): Frozen text-encoder. tokenizer ([`XLMRobertaTokenizer`]): Tokenizer of class scheduler (`DDIMScheduler` | `DDPMScheduler`): A scheduler to be used in combination with `unet` to generate image latents. unet ([`UNet2DConditionModel`]): Conditional U-Net architecture to denoise the image embedding. movq ([`VQModel`]): MoVQ Decoder to generate the image from the latents. ztext_encoder->unet->movq text_encoder tokenizerunet schedulermovqcs:t|j|||||ddt|jjjd|_dS)N)rrrrr r r)super__init__register_moduleslenr configblock_out_channelsmovq_scale_factor)selfrrrrr  __class__rrr"gs zKandinskyPipeline.__init__cCsR|dur t||||d}n|j|krtd|jd|||}||j}|S)N) generatordevicedtypezUnexpected latents shape, got z , expected )r shape ValueErrortoinit_noise_sigma)r(r.r-r,r+latentsrrrrprepare_latents{s   z!KandinskyPipeline.prepare_latentsNc CsZt|tr t|nd}|j|ddddddd}|j}|j|dddj} | jd |jd krRt|| sR|j| dd|jj dd f} t d |jj d | | |}|j |} |j|| d \} } | j|d d} | j|d d} | j|d d} |r(|durdg|}n;t|t|urtdt|dt|dt|tr|g}n|t|krtd|dt|d|d|d |}|j|ddddddd}|j |}|j |}|j||d \}}|jd}|d|}||||}|jd}|d|d}||||d }|j|d d}t|| g} t|| g} t|| g} | | | fS)Nr max_lengthTMpt)padding truncationr4return_attention_maskadd_special_tokensreturn_tensorslongest)r7r;z\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: ) input_idsattention_maskrdimz?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)r7r4r8r9r:r;) isinstancelistr$rr>r.torchequal batch_decodemodel_max_lengthloggerwarningr0r?rrepeat_interleavetype TypeErrorstrr/repeatviewcat)r(promptr,num_images_per_promptdo_classifier_free_guidancenegative_prompt batch_size text_inputstext_input_idsuntruncated_ids removed_text text_mask prompt_embedstext_encoder_hidden_states uncond_tokens uncond_inputuncond_text_input_idsuncond_text_masknegative_prompt_embeds!uncond_text_encoder_hidden_statesseq_lenrrr_encode_prompts  $             z KandinskyPipeline._encode_promptd@rpilTrS image_embedsnegative_image_embedsrVheightwidthnum_inference_stepsguidance_scalerTr+r2 output_typecallbackcallback_steps return_dictc#Cst|trd}nt|trt|}n tdt||j}|| }|dk}|||| ||\}}}t|tr>tj |dd}t|trJtj |dd}|ri|j | dd}|j | dd}tj ||gddj |j |d}|j j||d|j j}|jjj}t|||j\}}|||||f|j || | |j } t||D]\}}|rt | gdn| }||d }|j||||d d d}|r|j| jddd\}}|d\}}|d\}} ||||}tj || gdd}t|j jd r|j jjd vs|j| jddd\}}|j j||| | dj} | dur)||dkr)|t|j dd}!| |!|| t r0t!"q|j#j$| ddd}"|%| dvrKtd| | dvrj|"dd}"|"&dd}"|"'(dddd)*}"| dkrt|+|"}"|sz|"fSt,|"dS)a Function invoked when calling the pipeline for generation. Args: prompt (`str` or `list[str]`): The prompt or prompts to guide the image generation. image_embeds (`torch.Tensor` or `list[torch.Tensor]`): The clip image embeddings for text prompt, that will be used to condition the image generation. negative_image_embeds (`torch.Tensor` or `list[torch.Tensor]`): The clip image embeddings for negative text prompt, will be used to condition the image generation. negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). height (`int`, *optional*, defaults to 512): The height in pixels of the generated image. width (`int`, *optional*, defaults to 512): The width in pixels of the generated image. num_inference_steps (`int`, *optional*, defaults to 100): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. guidance_scale (`float`, *optional*, defaults to 4.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`torch.Generator` or `list[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will be generated by sampling using the supplied random `generator`. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between: `"pil"` (`PIL.Image.Image`), `"np"` (`np.array`) or `"pt"` (`torch.Tensor`). callback (`Callable`, *optional*): A function that calls every `callback_steps` steps during inference. The function is called with the following arguments: `callback(step: int, timestep: int, latents: torch.Tensor)`. callback_steps (`int`, *optional*, defaults to 1): The frequency at which the `callback` function is called. If not specified, the callback is called at every step. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.ImagePipelineOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.ImagePipelineOutput`] or `tuple` rz2`prompt` has to be of type `str` or `list` but is g?rr@)r-r,)r,r ) text_embedsrkF)sampletimestepencoder_hidden_statesadded_cond_kwargsrt variance_type)learned learned_range)r+NorderT)force_not_quantizerv)r6nprjzIOnly the output types `pt`, `pil` and `np` are supported not output_type=)rrjg?rrj)images)-rDrOrEr$r/rM_execution_devicerfrFrRrLr0r-r set_timesteps timestepsrr% in_channelsrr'r3 enumerate progress_barsplitr.chunkhasattrrzstep prev_samplegetattr XLA_AVAILABLExm mark_stepr decodemaybe_free_model_hooksclampcpupermutefloatnumpy numpy_to_pilr)#r(rSrkrlrVrmrnrorprTr+r2rqrrrsrtrWr,rUr]r^_timesteps_tensornum_channels_latentsitlatent_model_inputry noise_pred variance_prednoise_pred_uncondnoise_pred_textvariance_pred_textstep_idximagerrr__call__s I                  zKandinskyPipeline.__call__)N) NrgrgrhrirNNrjNrT)__name__ __module__ __qualname____doc__model_cpu_offload_seqrrrrrrr"r3rfrFno_gradr EXAMPLE_DOC_STRINGrOrETensorintr Generatorrboolr __classcell__rrr)rrQs| f      r)r)!typingrrF transformersrmodelsrr schedulersrrutilsr r r utils.torch_utilsr pipeline_utilsrrrrtorch_xla.core.xla_modelcore xla_modelrr get_loggerrrJrrrrrrrs"