o ۷i0@sLddlZddlZddlmZmZddlmZddlmZddl m Z ddl m Z ddl mZmZd d lmZmZd d lmZmZmZd d lmZerL eeZ d(dededeeeBdejdedeejf ddZ d)dejdej!dBdefddZ" d*dejde dej!dejd ej#d!ef d"d#Z$Gd$d%d%eZ%Gd&d'd'eZ&dS)+N)Qwen2Tokenizer Qwen3Model) FrozenDict)ClassifierFreeGuidance)VaeImageProcessor) AutoencoderKL)is_ftfy_availablelogging)ModularPipelineBlocks PipelineState) ComponentSpec InputParam OutputParam)ZImageModularPipeline text_encoder tokenizerpromptdevicemax_sequence_lengthreturnc Cst|tr|gn|}t|D]\}}d|dg}|j|dddd}|||<q||d|ddd}|j|} |j|} || | dd jd } g} t t | D] }| | || |qO| S) Nuser)rolecontentFT)tokenizeadd_generation_promptenable_thinking max_lengthpt)paddingr truncationreturn_tensors) input_idsattention_maskoutput_hidden_states) isinstancestr enumerateapply_chat_templater%tor&bool hidden_statesrangelenappend) rrrrri prompt_itemmessages text_inputstext_input_ids prompt_masks prompt_embedsprompt_embeds_listr;b/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/modular_pipelines/z_image/encoders.pyget_qwen_prompt_embeds$s>  r=sampleencoder_output generator sample_modecCsRt|dr|dkr|j|St|dr|dkr|jSt|dr%|jStd)N latent_distr>argmaxlatentsz3Could not access latents of provided encoder_output)hasattrrBr>moderDAttributeError)r?r@rAr;r;r<retrieve_latentsSs   rH image_tensorvaedtypelatent_channelscsttjstdtdttr.tjdkr.tdtdjddj||dttrRfddt jdD}tj |dd }n t  d }|j jj j}|S) Nz*Expected image_tensor to be a tensor, got .rz/You have passed a list of generators of length z), but it is not same as number of images rrLc s.g|]}t||d|dqS)rr@)rHencode).0r3r@rJrKr;r< ss z$encode_vae_image..)dimrP)r)torchTensor ValueErrortypelistr1shaper-r0catrHrQconfig shift_factorscaling_factor)rJrKr@rrLrM image_latentsr;rSr<encode_vae_image`s   rac @seZdZdZedefddZedeefddZ edee fddZ edee fd d Z ed d Ze  ddedejd Bdeded Bdef ddZedededefddZd S)ZImageTextEncoderStepz-imagercCdS)NzMText Encoder step that generate text_embeddings to guide the video generationr;selfr;r;r< descriptionz!ZImageTextEncoderStep.descriptionc Cs,tdttdttdttdddddgS) Nrrguiderg@F)guidance_scaleenabled from_configr]default_creation_method)rrrrrrer;r;r<expected_componentss z)ZImageTextEncoderStep.expected_componentscCstdtdtdddgS)Nrnegative_promptrr)default)rrer;r;r<inputss zZImageTextEncoderStep.inputscCs,tdttjdddtdttjdddgS)Nr9denoiser_input_fieldsz2text embeddings used to guide the image generation) type_hint kwargs_typergnegative_prompt_embedsz;negative text embeddings used to guide the image generation)rrZrVrWrer;r;r<intermediate_outputssz*ZImageTextEncoderStep.intermediate_outputscCsB|jdurt|jtst|jtstdt|jdSdSdS)Nz2`prompt` has to be of type `str` or `list` but is )rr)r*rZrXrY) block_stater;r;r< check_inputss    z"ZImageTextEncoderStep.check_inputsNTrrrprepare_unconditional_embedsrprc Cs|p|j}t|ts |g}t|}t|j|j|||d}d}|ro|p#d}t|tr.||gn|}|durKt|t|urKt dt|dt|d|t|krdt d|dt|d |d|d t|j|j|||d}||fS) aC Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `list[str]`, *optional*): prompt to be encoded device: (`torch.device`): torch device prepare_unconditional_embeds (`bool`): whether to use prepare unconditional embeddings or not negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). max_sequence_length (`int`, defaults to `512`): The maximum number of text tokens to be used for the generation process. )rrrrrNz?`negative_prompt` should be the same type to `prompt`, but got z != rNz`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.) _execution_devicer)rZr1r=rrr*rY TypeErrorrX) componentsrrrzrpr batch_sizer9rvr;r;r< encode_promptsJ    z#ZImageTextEncoderStep.encode_promptr~statecCsX||}|||j|_|j||j|j|j|j|jd\|_ |_ | ||||fS)N)r~rrrzrpr) get_block_stateryr|rrrrequires_unconditional_embedsrprr9rvset_block_state)rfr~rrxr;r;r<__call__s   zZImageTextEncoderStep.__call__)NTNr)__name__ __module__ __qualname__ model_namepropertyr*rgrZrrorrrrrw staticmethodryrVrr.intrno_gradrr rr;r;r;r<rbs<   Arbc@seZdZdZedefddZedeefddZ edee fddZ edee fd d Z ed d Zd ededefddZdS)ZImageVaeImageEncoderSteprcrcCrd)NzcVae Image Encoder step that generate condition_latents based on image to guide the image generationr;rer;r;r<rgrhz%ZImageVaeImageEncoderStep.descriptioncCs"tdttdttddiddgS)NrKimage_processorvae_scale_factorrIrlrm)rrrrrer;r;r<ros z-ZImageVaeImageEncoderStep.expected_componentscCs&tdtjjddtdtdtdgS)NimageT)rtrequiredheightwidthr@)rPILImagerer;r;r<rr#s z ZImageVaeImageEncoderStep.inputscCstdtjddgS)Nr`z@video latent representation with the first frame image condition)rtrg)rrVrWrer;r;r<rw,s z.ZImageVaeImageEncoderStep.intermediate_outputscCs^|jdur |j|jdks|jdur+|j|jdkr-td|jd|jd|jddSdS)Nrz-`height` and `width` have to be divisible by z but are z and rN)rvae_scale_factor_spatialrrX)r~rxr;r;r<ry6s z&ZImageVaeImageEncoderStep.check_inputsr~rc Cs||}||||j}|j}tj}|jj}|jj ||j |j dj ||d}t ||j|j|||jd|_|||||fS)N)rrrO)rJrKr@rrLrM)rryrr|rVfloat32rKrLr preprocessrrr-rar@num_channels_latentsr`r) rfr~rrxrrrL vae_dtyperJr;r;r<r?s*    z"ZImageVaeImageEncoderStep.__call__N)rrrrrr*rgrZrrorrrrrwrryrr rr;r;r;r<rs  r)r)Nr>)rI)'rrV transformersrrconfiguration_utilsrguidersrrrmodelsrutilsr r modular_pipeliner r modular_pipeline_utilsrrrr get_loggerrloggerr*rZrrrWr= GeneratorrHrLrarbrr;r;r;r<sj        0