o Gi@s8ddlZddlmZmZmZmZmZddlmZddl m Z ddl m Z m Z ddlmZmZddlmZmZmZdd lmZdd lmZmZmZmZd d lmZmZd d lm Z m!Z!m"Z"m#Z#ddlm$Z$e%e&Z' ddej(dej)dBde*fddZ+GdddeZ,GdddeZ-GdddeZ.GdddeZ/dS)N)CLIPImageProcessor CLIPTextModelCLIPTextModelWithProjection CLIPTokenizerCLIPVisionModelWithProjection) FrozenDict)ClassifierFreeGuidance)PipelineImageInputVaeImageProcessor) StableDiffusionXLLoraLoaderMixinTextualInversionLoaderMixin) AutoencoderKLImageProjectionUNet2DConditionModel)adjust_lora_scale_text_encoder)USE_PEFT_BACKENDloggingscale_lora_layersunscale_lora_layers)ModularPipelineBlocks PipelineState) ComponentSpec ConfigSpec InputParam OutputParam) StableDiffusionXLModularPipelinesampleencoder_output generator sample_modecCsRt|dr|dkr|j|St|dr|dkr|jSt|dr%|jStd)N latent_distrargmaxlatentsz3Could not access latents of provided encoder_output)hasattrr#rmoder%AttributeError)r r!r"r)l/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/modular_pipelines/stable_diffusion_xl/encoders.pyretrieve_latents-s   r+c@seZdZdZedefddZedeefddZ edee fddZ edee fd d Z edd d ZddZedededefddZd S)StableDiffusionXLIPAdapterStepstable-diffusion-xlreturncCs dS)NaIP Adapter step that prepares ip adapter image embeddings. Note that this step only prepares the embeddings - in order for it to work correctly, you need to load ip adapter weights into unet via ModularPipeline.load_ip_adapter() and pipeline.set_ip_adapter_scale(). See [ModularIPAdapterMixin](https://huggingface.co/docs/diffusers/api/loaders/ip_adapter#diffusers.loaders.ModularIPAdapterMixin) for more detailsr)selfr)r)r* description=sz*StableDiffusionXLIPAdapterStep.descriptionc CsBtdttdttdddddtdttdttd d iddgS) N image_encoderfeature_extractor)size crop_size from_configconfigdefault_creation_methodunetguiderguidance_scale@)rrrrrr r/r)r)r*expected_componentsGs  z2StableDiffusionXLIPAdapterStep.expected_componentscCstdtdddgS)Nip_adapter_imageTz%The image(s) to be used as ip adapter)requiredr1)rr r/r)r)r*inputsZsz%StableDiffusionXLIPAdapterStep.inputscCs tdtjddtdtjddgS)Nip_adapter_embedszIP adapter image embeddings type_hintr1negative_ip_adapter_embedsz$Negative IP adapter image embeddingsrtorchTensorr/r)r)r*intermediate_outputsesz3StableDiffusionXLIPAdapterStep.intermediate_outputsNc Cst|jj}t|tjs|j|ddj}|j ||d}|rH|j|ddj d}|j |dd}|jt |ddj d}|j |dd}||fS||j }|j |dd}t |} || fS) Npt)return_tensorsdevicedtypeToutput_hidden_statesrdim)nextr2 parametersrO isinstancerHrIr3 pixel_valuesto hidden_statesrepeat_interleave zeros_like image_embeds) componentsimagerNnum_images_per_promptrQrOimage_enc_hidden_statesuncond_image_enc_hidden_statesr]uncond_image_embedsr)r)r* encode_imageps(    z+StableDiffusionXLIPAdapterStep.encode_imagecCsng}|rg}|durft|ts|g}t|t|jjjkr/tdt|dt|jjjdt||jjjD]-\} } t| t } | || |d| \} } | | dddf|rd| | dddfq7n|D]} |rx| d\} } | | | | qhg}t |D]0\}} t j| g|dd} |rt j||g|dd} t j| | gdd} | j|d} | | q|S) NzK`ip_adapter_image` must have same length as the number of IP Adapters. Got z images and z IP Adapters.rrrrS)rN)rWlistlenr;encoder_hid_projimage_projection_layers ValueErrorziprrdappendchunk enumeraterHcatrY)r0r^r@ip_adapter_image_embedsrNr`prepare_unconditional_embedsr]negative_image_embedssingle_ip_adapter_imageimage_proj_layeroutput_hidden_statesingle_image_embedssingle_negative_image_embedsir)r)r*prepare_ip_adapter_image_embedssH          z>StableDiffusionXLIPAdapterStep.prepare_ip_adapter_image_embedsr^statecCs||}|jjdk|_|j|_|j||jd|jd|jd|_|jrAg|_ t |jD]\}}| d\}}|j |||j|<q*| ||||fS)Nr)r@rorNr`rpr)get_block_stater<num_conditionsrp_execution_devicerNrxr@rCrFrmrlrkset_block_state)r0r^ry block_staterwr]rqr)r)r*__call__s&    z'StableDiffusionXLIPAdapterStep.__call__)N)__name__ __module__ __qualname__ model_namepropertystrr1rerr?rrBrrJ staticmethodrdrxrHno_gradrrrr)r)r)r*r,:s   3r,c@s2eZdZdZedefddZedeefddZ edee fddZ edee fd d Z edeefd d Zed dZe            d%dededBdejdBdedededBdedBdejdBdejdBdejdBdejdBdedBdedBfdd Zed!ed"edefd#d$ZdS)& StableDiffusionXLTextEncoderStepr-r.cCdS)NzMText Encoder step that generate text_embeddings to guide the image generationr)r/r)r)r*r1z,StableDiffusionXLTextEncoderStep.descriptionc Cs:tdttdttdttdttdttddidd gS) N text_encodertext_encoder_2 tokenizer tokenizer_2r<r=r>r7r8)rrrrr rr/r)r)r*r?s z4StableDiffusionXLTextEncoderStep.expected_componentscCs tddgS)Nforce_zeros_for_empty_promptT)rr/r)r)r*expected_configss z1StableDiffusionXLTextEncoderStep.expected_configscCs(tdtdtdtdtdtdgS)Npromptprompt_2negative_promptnegative_prompt_2cross_attention_kwargs clip_skip)rr/r)r)r*rBsz'StableDiffusionXLTextEncoderStep.inputsc CsDtdtjdddtdtjdddtdtjdddtd tjdd dgS) N prompt_embedsdenoiser_input_fieldsz2text embeddings used to guide the image generation)rE kwargs_typer1negative_prompt_embedsz;negative text embeddings used to guide the image generationpooled_prompt_embedsz9pooled text embeddings used to guide the image generationnegative_pooled_prompt_embedszBnegative pooled text embeddings used to guide the image generationrGr/r)r)r*rJs2z5StableDiffusionXLTextEncoderStep.intermediate_outputscCsx|jdurt|jtst|jtstdt|j|jdur6t|jts8t|jts:tdt|jdSdSdS)Nz2`prompt` has to be of type `str` or `list` but is z4`prompt_2` has to be of type `str` or `list` but is )rrWrrerityper)r~r)r)r* check_inputss       z-StableDiffusionXLTextEncoderStep.check_inputsNrTrrrNr`rprrrrrr lora_scalerc! Cs|p|j}| dur9t|tr9| |_|jdur%tst|j| nt|j| |jdur9ts3t|j| nt|j| t|t rA|gn|}|durLt |}n|j d}|j dur\|j |j gn|j g}|jdurk|j|jgn|jg}|dur |pw|}t|t r|gn|}g}||g}t|||D]u\}}}t|tr|||}||d|jddd}|j}||dddj}|j d |j d krt||s||dd|jd d f}td |jd ||||dd }|d} | dur|jd}n|j| d }||qtj|d d}|duo|jj}|r*| dur*|r*t|} t| } n|r| dur|p6d}|p;|}t|t rG||gn|}t|t rT||gn|}|durst|t|urst dt|dt|d|t |krt!d|dt |d|d|d ||g}g}t|||D]8\}}}t|tr|||}|j d }||d|ddd}||j|dd } | d} | jd} || qtj|d d} |jdur|j|jj"|d}n |j|j#j"|d}|j \}}} |$d |d }|%|||d }|r9| j d }|jdur | j|jj"|d} n | j|j#j"|d} | $d |d } | %|||d } | $d |%||d } |rT| $d |%||d } |jdurit|tritrit&|j| |jdur~t|tr~tr~t&|j| || | | fS)ae Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `list[str]`, *optional*): prompt to be encoded prompt_2 (`str` or `list[str]`, *optional*): The prompt or prompts to be sent to the `tokenizer_2` and `text_encoder_2`. If not defined, `prompt` is used in both text-encoders device: (`torch.device`): torch device num_images_per_prompt (`int`): number of images that should be generated per prompt prepare_unconditional_embeds (`bool`): whether to use prepare unconditional embeddings or not negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). negative_prompt_2 (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the image generation to be sent to `tokenizer_2` and `text_encoder_2`. If not defined, `negative_prompt` is used in both text-encoders prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. pooled_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, pooled text embeddings will be generated from `prompt` input argument. negative_pooled_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, pooled negative_prompt_embeds will be generated from `negative_prompt` input argument. lora_scale (`float`, *optional*): A lora scale that will be applied to all LoRA layers of the text encoder if LoRA layers are loaded. clip_skip (`int`, *optional*): Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that the output of the pre-final layer will be used for computing the prompt embeddings. Nr max_lengthTrK)paddingr truncationrLlongest)rrLrz\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: rPrRrrSz?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)rOrN)'r|rWr _lora_scalerrrrrrrfshaperrrjr maybe_convert_promptmodel_max_length input_idsrHequal batch_decodeloggerwarningrYrZrkconcatr9rr\r TypeErrorrirOr;repeatviewr)!r^rrrNr`rprrrrrrrr batch_size tokenizers text_encodersprompt_embeds_listpromptsrr text_inputstext_input_idsuntruncated_ids removed_textzero_out_negative_prompt uncond_tokensnegative_prompt_embeds_listr uncond_inputbs_embedseq_len_r)r)r* encode_prompt%s ;                                     z.StableDiffusionXLTextEncoderStep.encode_promptr^rycCs||}|||jjdk|_|j|_|jdur!|jddnd|_ |j ||j |j |jd|j|j |jdddd|j |jd\|_|_|_|_|||||fS)Nrscale)rrrrrr)rzrr<r{rpr|rNrgettext_encoder_lora_scalerrrrrrrrrrr})r0r^ryr~r)r)r*rs>     z)StableDiffusionXLTextEncoderStep.__call__) NNrTNNNNNNNN)rrrrrrr1rerr?rrrrBrrJrrrHrNintboolrIfloatrrrrrr)r)r)r*rsr         src@seZdZdZedefddZedeefddZ edee fddZ edee fd d Z d ejd ejfd dZedededefddZdS)StableDiffusionXLVaeEncoderStepr-r.cCr)NzIVae Encoder step that encode the input image into a latent representationr)r/r)r)r*r1Drz+StableDiffusionXLVaeEncoderStep.descriptioncCs"tdttdttddiddgS)Nvaeimage_processorvae_scale_factorr7r8rrr rr/r)r)r*r?Hs z3StableDiffusionXLVaeEncoderStep.expected_componentsc Cs>tdddtdtdtdtdtjdd td tdBd d gS) Nr_TrAheightwidthr!rOz Data type of model tensor inputsrDpreprocess_kwargszA kwargs dictionary that if specified is passed along to the `ImageProcessor` as defined under `self.image_processor` in [diffusers.image_processor.VaeImageProcessor])rrHrOdictr/r)r)r*rBTs z&StableDiffusionXLVaeEncoderStep.inputscCstdtjddgS)N image_latentszUThe latents representing the reference image for image-to-image/inpainting generationrDrGr/r)r)r*rJcs z4StableDiffusionXLVaeEncoderStep.intermediate_outputsr_r!csjd}}tjjdr jjjdur tjjjdddd}tjjdr<jjjdur|"zEStableDiffusionXLVaeEncoderStep._encode_vae_image..rrSrrMr&rr9rrHtensorrrrO force_upcastrrYfloat32rWrerangerrnr+rrNscaling_factorr0r^r_r!rrrOrr)rr*_encode_vae_imageo2      z1StableDiffusionXLVaeEncoderStep._encode_vae_imager^rycCs||}|jp i|_|j|_|jdur|jn|jj|_|jj|jf|j |j d|j}|j |j|jd}|j d|_ t|jtrYt|j|j krYtdt|jd|j d|j|||jd|_|||||fS)N)rrrMrz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.r_r!)rzrr|rNrOrr preprocessr_rrrYrrrWr!rerfrirrr})r0r^ryr~r_r)r)r*rs,    z(StableDiffusionXLVaeEncoderStep.__call__N)rrrrrrr1rerr?rrBrrJrHrI Generatorrrrrrr)r)r)r*rAs  "rc@seZdZdZedeefddZedefddZ edee fddZ edee fd d Z d ejd ejfd dZddZedededefddZdS)&StableDiffusionXLInpaintVaeEncoderStepr-r.c Cs>tdttdttddiddtdttddd d d ddgS) Nrrrrr7r8mask_processorFT) do_normalizer do_binarizedo_convert_grayscalerr/r)r)r*r?s   z:StableDiffusionXLInpaintVaeEncoderStep.expected_componentscCr)NzLVae encoder step that prepares the image and mask for the inpainting processr)r/r)r)r*r1rz2StableDiffusionXLInpaintVaeEncoderStep.descriptionc Cs>tdtdtdddtdddtdtdtjd d td gS) Nrrr_Tr mask_imagepadding_mask_croprOzThe dtype of the model inputsrDr!)rrHrOr/r)r)r*rBs  z-StableDiffusionXLInpaintVaeEncoderStep.inputscCsFtdtjddtdtjddtdtjddtdtttfdBd dgS) Nrz-The latents representation of the input imagerDmaskz*The mask to use for the inpainting processmasked_image_latentsz^The masked image latents to use for the inpainting process (only for inpainting-specifid unet) crops_coordszPThe crop coordinates to use for the preprocess/postprocess of the image and mask)rrHrItuplerr/r)r)r*rJsz;StableDiffusionXLInpaintVaeEncoderStep.intermediate_outputsr_r!csjd}}tjjdr jjjdur tjjjdddd}tjjdr<jjjdur.rrSrrMrrr)rr*rrz8StableDiffusionXLInpaintVaeEncoderStep._encode_vae_imagec Cs@tjjj|||j||jfd}|j||d}|jd|krC||jddks6td|d|jdd|||jdddd}|durQ|jddkrQ|} nd} |dur| durj|j||d}|j ||| d } | jd|kr|| jddkstd |d| jdd | || jdddd} | j||d} || fS) N)r5rMrzvThe passed mask and the required batch size don't match. Masks are supposed to be duplicated to a total batch size of z, but zo masks were passed. Make sure the number of masks that you pass is divisible by the total requested batch size.rrrzyThe passed images and the required batch size don't match. Images are supposed to be duplicated to a total batch size of zq images were passed. Make sure the number of images that you pass is divisible by the total requested batch size.) rHnn functional interpolaterrYrrirr) r0r^r masked_imagerrrrOrNr!rr)r)r*prepare_mask_latents sFz;StableDiffusionXLInpaintVaeEncoderStep.prepare_mask_latentsr^ryc Cs^||}|jdur |jn|jj|_|j|_|jdur|j|_|jdur(|j|_|j dur@|j j |j |j|j|j d|_ d|_nd|_ d|_|jj|j|j|j|j |jd}|jtjd}|j j|j |j|j|j|j d}||dk|_|jd|_|j|j|jd }|j|||jd |_||||j|j|j|j|j|j|j \|_|_|||||fS) N)padfilldefault)rrr resize_moder)rrrrg?rrMr)rzrOrr|rNrdefault_heightr default_widthrrget_crop_regionrrrrrr_rYrHrrrrrr!rrrrr})r0r^ryr~r_rr)r)r*r;s\      z/StableDiffusionXLInpaintVaeEncoderStep.__call__N)rrrrrrerr?rr1rrBrrJrHrIrrrrrrrr)r)r)r*rs $/r)Nr)0rH transformersrrrrrconfiguration_utilsrguidersr rr r loadersr r modelsrrr models.lorarutilsrrrrmodular_pipelinerrmodular_pipeline_utilsrrrrr get_loggerrrrIrrr+r,rrrr)r)r)r*s8       kk