o GiPK@sddlZddlZddlZddlmZddlmZm Z m Z m Z ddl m Z mZddlmZddlmZmZddlmZd d lmZmZmZd d lmZerYddlmmZd Z ndZ e!e"Z#GdddeeZ$dS)N) functional)CLIPImageProcessorCLIPTextModelWithProjection CLIPTokenizerCLIPVisionModelWithProjection)UNet2DConditionModel UNet2DModel)UnCLIPScheduler)is_torch_xla_availablelogging) randn_tensor)DeprecatedPipelineMixinDiffusionPipelineImagePipelineOutput)UnCLIPTextProjModelTFcsVeZdZUdZdZeed<eed<eed<e ed<e ed<e ed<e ed <e ed <e ed <e ed <d Zdedede dede de d e d e d e d e ffdd ZddZddZd*dejdBfddZe           d+dejjeejjBejBdBded ed!ed"ejdBd#ejdBd$ejdBdejdBd%ed&edBd'efd(d)ZZS),UnCLIPImageVariationPipelinea Pipeline to generate image variations from an input image using UnCLIP. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.). Args: text_encoder ([`~transformers.CLIPTextModelWithProjection`]): Frozen text-encoder. tokenizer ([`~transformers.CLIPTokenizer`]): A `CLIPTokenizer` to tokenize text. feature_extractor ([`~transformers.CLIPImageProcessor`]): Model that extracts features from generated images to be used as inputs for the `image_encoder`. image_encoder ([`~transformers.CLIPVisionModelWithProjection`]): Frozen CLIP image-encoder ([clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14)). text_proj ([`UnCLIPTextProjModel`]): Utility class to prepare and combine the embeddings before they are passed to the decoder. decoder ([`UNet2DConditionModel`]): The decoder to invert the image embedding into an image. super_res_first ([`UNet2DModel`]): Super resolution UNet. Used in all but the last step of the super resolution diffusion process. super_res_last ([`UNet2DModel`]): Super resolution UNet. Used in the last step of the super resolution diffusion process. decoder_scheduler ([`UnCLIPScheduler`]): Scheduler used in the decoder denoising process (a modified [`DDPMScheduler`]). super_res_scheduler ([`UnCLIPScheduler`]): Scheduler used in the super resolution denoising process (a modified [`DDPMScheduler`]). z0.33.1decoder text_proj text_encoder tokenizerfeature_extractor image_encodersuper_res_firstsuper_res_lastdecoder_schedulersuper_res_schedulerzPtext_encoder->image_encoder->text_proj->decoder->super_res_first->super_res_lastc s,t|j||||||||| | d dS)N) rrrrrrrrrr)super__init__register_modules) selfrrrrrrrrrr __class__n/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/unclip/pipeline_unclip_image_variation.pyr Ys  z%UnCLIPImageVariationPipeline.__init__cCsR|dur t||||d}n|j|krtd|jd|||}||j}|S)N) generatordevicedtypezUnexpected latents shape, got z , expected )r shape ValueErrortoinit_noise_sigma)r"r*r)r(r'latents schedulerr%r%r&prepare_latentsvs   z,UnCLIPImageVariationPipeline.prepare_latentscCstt|tr t|nd}|j|d|jjdd}|j}|j|}| ||} | j } | j } | j |dd} | j |dd} |j |dd}|rdg|} |j d} |j| d| d dd }|j|}| |j|}|j }|j }|j d}|d|}||||}|j d}|d|d}||||d}|j |dd}t|| g} t|| g} t||g}| | |fS) Nr max_lengthpt)paddingr1return_tensorsrdimT)r3r1 truncationr4) isinstancelistlenrmodel_max_length input_idsattention_maskboolr,r text_embedslast_hidden_staterepeat_interleaver*repeatviewtorchcat)r"promptr(num_images_per_promptdo_classifier_free_guidance batch_size text_inputstext_input_ids text_masktext_encoder_output prompt_embedstext_encoder_hidden_states uncond_tokensr1 uncond_inputuncond_text_mask*negative_prompt_embeds_text_encoder_outputnegative_prompt_embeds!uncond_text_encoder_hidden_statesseq_lenr%r%r&_encode_promptsT       z+UnCLIPImageVariationPipeline._encode_promptNimage_embeddingscCs`t|jj}|dur't|tjs|j|ddj}|j ||d}||j }|j |dd}|S)Nr2)imagesr4)r(r)rr5) nextr parametersr)r:rFTensorr pixel_valuesr, image_embedsrC)r"imager(rIrZr)r%r%r& _encode_images  z*UnCLIPImageVariationPipeline._encode_imager @pilTrarIdecoder_num_inference_stepssuper_res_num_inference_stepsr'decoder_latentssuper_res_latentsdecoder_guidance_scale output_type return_dictc (Cs |durt|tjjrd} nt|trt|} n |jd} n|jd} dg| } |j}| |} | dk}|| |||\}}}|||||}|j ||||d\}}|j dkro| t j }t j||j jdfdd}| t j}n t j||j jdfd d}|jj||d |jj}|jjj}|jjj}|jjj}|dur|| |||f|j||||j}t||D]l\}}|rt |gd n|}|j|||||d j}|r|d \}}|j|jddd \}}|j|jddd \}} || ||}t j|| gdd }|d|jdkr d}!n||d}!|jj ||||!|dj!}q|"dd}|}"|j#j||d |j#j}#|j$jjd }$|j$jj}|j$jj}|durX|| |$||f|"j||||j#}|j dkrht j%|"||gd}%ni}&dt&'t j%j(vrxd |&d<t j%|"f||gddd|&}%t||#D]L\}}||#jddkr|j)}'n|j$}'t j||%gdd }|'||dj}|d|#jdkrd}!n|#|d}!|j#j ||||!|dj!}t*rt+,q|}|-|dd}|"dd}|./dd dd01}| dkr|2|}| s |fSt3|dS)a The call function to the pipeline for generation. Args: image (`PIL.Image.Image` or `list[PIL.Image.Image]` or `torch.Tensor`): `Image` or tensor representing an image batch to be used as the starting point. If you provide a tensor, it needs to be compatible with the [`CLIPImageProcessor`] [configuration](https://huggingface.co/fusing/karlo-image-variations-diffusers/blob/main/feature_extractor/preprocessor_config.json). Can be left as `None` only when `image_embeddings` are passed. num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. decoder_num_inference_steps (`int`, *optional*, defaults to 25): The number of denoising steps for the decoder. More denoising steps usually lead to a higher quality image at the expense of slower inference. super_res_num_inference_steps (`int`, *optional*, defaults to 7): The number of denoising steps for super resolution. More denoising steps usually lead to a higher quality image at the expense of slower inference. generator (`torch.Generator`, *optional*): A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. decoder_latents (`torch.Tensor` of shape (batch size, channels, height, width), *optional*): Pre-generated noisy latents to be used as inputs for the decoder. super_res_latents (`torch.Tensor` of shape (batch size, channels, super res height, super res width), *optional*): Pre-generated noisy latents to be used as inputs for the decoder. decoder_guidance_scale (`float`, *optional*, defaults to 4.0): A higher guidance scale value encourages the model to generate images closely linked to the text `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`. image_embeddings (`torch.Tensor`, *optional*): Pre-defined image embeddings that can be derived from the image encoder. Pre-defined image embeddings can be passed for tasks like image interpolations. `image` can be left as `None`. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generated image. Choose between `PIL.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.ImagePipelineOutput`] instead of a plain tuple. Returns: [`~pipelines.ImagePipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.ImagePipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list with the generated images. Nrrr7g?)rZrPrQrJmps)valueT)r(r)sampletimestepencoder_hidden_states class_labelsr?r5) prev_timestepr'r8)size antialiasbicubicF)rumode align_corners)rprqg?rrf)r[)4r:PILImager;r<r*_execution_devicerYrbrtyperFintFpadclip_extra_context_tokensr@r set_timesteps timestepsrconfig in_channels sample_sizer0r) enumerate progress_barrGrpchunksplitstep prev_sampleclamprr interpolateinspect signaturer]r XLA_AVAILABLExm mark_stepmaybe_free_model_hookscpupermutefloatnumpy numpy_to_pilr)(r"rarIrgrhr'rirjrZrkrlrmrKrHr(rJrPrQrNadditive_clip_time_embeddingsdecoder_text_maskdecoder_timesteps_tensornum_channels_latentsheightwidthitlatent_model_input noise_prednoise_pred_uncondnoise_pred_text_predicted_variancert image_smallsuper_res_timesteps_tensorchannelsimage_upscaledinterpolate_antialiasunetr%r%r&__call__s7                             z%UnCLIPImageVariationPipeline.__call__)N) NrrcrdNNNNrerfT) __name__ __module__ __qualname____doc___last_supported_versionr__annotations__rrrrrr r model_cpu_offload_seqr r0rYrFr^rbno_gradrzr{r;r~ Generatorrstrr@r __classcell__r%r%r#r&r-s     ?    r)%r PIL.ImagerzrFtorch.nnrr transformersrrrrmodelsrr schedulersr utilsr r utils.torch_utilsr pipeline_utilsrrrrrtorch_xla.core.xla_modelcore xla_modelrr get_loggerrloggerrr%r%r%r&s"