o ۷ixp @sddlmZddlmZddlmZddlZddlmZddl mm Z ddl m Z mZddlmZmZddlmZdd lmZdd lmZerZdd lmZmZmZmZmZmZd d lm Z d dl!m"Z"d dl#m$Z$d dl%m&Z&d dl'm(Z(ddl)m*Z*m+Z+m,Z,e-e.Z/dej0dej1dej1fddZ2dej0dej1dej1fddZ3dej0dej1de4dej1fddZ5 d2de6d e4de4d!e4d"e4dej0f d#d$Z7e2e3e5d%Z8eGd&d'd'eZ9Gd(d)d)ej0Z:Gd*d+d+ej0Z;Gd,d-d-ej0ZdS)3) dataclass)sqrt)AnyN) ConfigMixinregister_to_config) BaseOutputlogging)apply_forward_hook)is_transformers_available) randn_tensor)Dinov2WithRegistersConfigDinov2WithRegistersModelSiglipVisionConfigSiglipVisionModel ViTMAEConfig ViTMAEModel)get_activation)AttentionMixin) Attention)get_2d_sincos_pos_embed) ModelMixin)AutoencoderMixin DecoderOutput EncoderOutputmodelimagesreturncCs&||dd}d}|jdd|dfS)NT)output_hidden_stateslast_hidden_state)rroutputsunused_token_numr&c/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_rae.py_dinov2_encoder_forward:s r(cCs||ddd}|jS)NT)r interpolate_pos_encodingr")rrr$r&r&r'_siglip2_encoder_forward@sr* patch_sizecCs|jd|jd}}t|||d}||d||kr#tdt|d|jdd|j|j }|||dd}|j ddddfS) Nrrz-Image size should be divisible by patch size.rT)r)r) shapeint ValueErrortorcharange unsqueezeexpandtodevicedtyper#)rrr+hw patch_numnoiser$r&r&r'_mae_encoder_forwardEs.r;@ encoder_type hidden_sizenum_hidden_layershead_dimcCs||}|dkrt||d||d}t|}d|j_d|j_n?|dkr:t||d||d}t|}d|jj_d|jj_n#|dkrUt ||d||d d }t |}d|j_d|j_nt d |d | d |S)zr+ image_sizenum_attention_headsr?Nsiglip2mae)r>r+rBrCr? mask_ratioUnknown encoder_type='z"'. Available: dinov2, siglip2, maeF) r r layernormweightbiasrr vision_modelpost_layernormrrr/requires_grad_)r=r>r+r?r@rCconfigrr&r&r'_build_encoderTsL     rR)rArDrFc@seZdZUdZejed<dS)RAEDecoderOutputz Output of `RAEDecoder`. Args: logits (`torch.Tensor`): Patch reconstruction logits of shape `(batch_size, num_patches, patch_size**2 * num_channels)`. logitsN)__name__ __module__ __qualname____doc__r0Tensor__annotations__r&r&r&r'rSs rScsBeZdZd dededeffdd Zdejdejfd d ZZ S) ViTMAEIntermediategelur>intermediate_size hidden_actcs&tt|||_t||_dSN)super__init__nnLineardenserintermediate_act_fn)selfr>r]r^ __class__r&r'ras zViTMAEIntermediate.__init__ hidden_statesrcCs||}||}|Sr_)rdre)rfrir&r&r'forwards  zViTMAEIntermediate.forward)r\) rUrVrWr.strrar0rYrj __classcell__r&r&rgr'r[sr[csHeZdZd dededeffdd Zdejdejd ejfd d ZZ S) ViTMAEOutputrHr>r]hidden_dropout_probcs(tt|||_t||_dSr_)r`rarbrcrdDropoutdropout)rfr>r]rnrgr&r'ras zViTMAEOutput.__init__ri input_tensorrcCs ||}||}||}|Sr_)rdrp)rfrirqr&r&r'rjs  zViTMAEOutput.forward)rH) rUrVrWr.floatrar0rYrjrlr&r&rgr'rms$rmcsfeZdZdZdddddddeded ed ed ed ed edeffddZde j de j fddZ Z S) ViTMAELayerz^ This matches the naming/parameter structure used in RAE-main (ViTMAE decoder block). T-q=rHr\)qkv_biaslayer_norm_epsrnattention_probs_dropout_probr^r>rCr]rurvrnrwr^c st||dkrtd|d|t||||||d|_t|||d|_t|||d|_t j ||d|_ t j ||d|_ dS)Nrz hidden_size=z* must be divisible by num_attention_heads=) query_dimheadsdim_headrprM)r>r]r^)r>r]rneps) r`rar/r attentionr[ intermediatermoutputrb LayerNormlayernorm_beforelayernorm_after) rfr>rCr]rurvrnrwr^rgr&r'ras( zViTMAELayer.__init__rircCs<|||}||}||}||}|||}|Sr_)r}rrr~r)rfriattention_output layer_outputr&r&r'rjs    zViTMAELayer.forward) rUrVrWrXr.boolrrrkrar0rYrjrlr&r&rgr'rss2   !rscseZdZdZ          d1d edededededededededededededeffdd Zde j de j fdd Z d!e j de j fd"d#Z d2d%e j d&e eefd$Bfd'd(Zd)d)d d*d+e j d,ed-ed.edee e j Bf d/d0ZZS)3 RAEDecodera  Decoder implementation ported from RAE-main to keep checkpoint compatibility. Key attributes (must match checkpoint keys): - decoder_embed - decoder_pos_embed - decoder_layers - decoder_norm - decoder_pred - trainable_cls_token rErTrtrHr\r>decoder_hidden_sizedecoder_num_hidden_layersdecoder_num_attention_headsdecoder_intermediate_size num_patchesr+ num_channelsrBrurvrnrwr^c st|_||_||_| |_||_tj|dd|_ t |d}t |dddd}|j d| dd d tfd d t|D|_tjd |_tj|d|dd|_d |_ttdd|_dS)NT)rM?rpt) cls_token extra_tokens output_typedecoder_pos_embedrF persistentc s$g|]}tdqS))r>rCr]rurvrnrwr^)rs).0_rwrrrr^rnrvrur&r' s z'RAEDecoder.__init__..r{r)r`rarr+rrBrrbrc decoder_embedr.rregister_bufferr2rr ModuleListrangedecoder_layersr decoder_norm decoder_predgradient_checkpointing Parameterr0zerostrainable_cls_token)rfr>rrrrrr+rrBrurvrnrwr^ grid_size pos_embedrgrr'ras*    zRAEDecoder.__init__ embeddingsrcCs|jdd}|jjdd}|jdddddf}|jddddddf}|jjd}|ddd|dddd}tj|d||fddd}|dddddd|}tj| d|fdd S) Nrrr,rrbicubicF) scale_factormode align_cornersdim) r-rreshapepermuteF interpolateviewr0catr2)rfrembeddings_positions num_positionsclass_pos_embedpatch_pos_embedrr&r&r'r)&s  z#RAEDecoder.interpolate_pos_encodingxcCs|j\}}}||jkr |St|d}}|||||dddd}t|jdt|jdf}tj||ddd}|dddd||j|}|S) NrrrrrbilinearFsizerr) r-rr.rrrr contiguousr)rfrblcr7r8 target_sizer&r&r'interpolate_latent8s   zRAEDecoder.interpolate_latentNpatchified_pixel_valuesoriginal_image_sizec Cs|j|j}}|dur |n|j|jf}|\}}||}||}|||jdkr8td|jdd|d||jd} || |||||}td|}|| |||||} | S)Nrz5The number of patches in the patchified pixel values z9, does not match the number of patches on original image *rznhwpqc->nchpwq)r+rrBr-r/rr0einsum) rfrrr+roriginal_heightoriginal_width num_patches_h num_patches_w batch_size pixel_valuesr&r&r' unpatchifyCs6  zRAEDecoder.unpatchifyF)r)drop_cls_token return_dictrir)rrc Cs||}|r|ddddddf}||}n||}|j|jddd}tj||gdd}|rA|s;td||}n|j }||j |j |j d}|j D]} | |}qR||}||} | ddddddf} |su| fSt| dS)Nrrr,rz:interpolate_pos_encoding only supports drop_cls_token=Truer5r6)rT)rrrr3r-r0rr/r)rr4r5r6rrrrS) rfrir)rrrx_rr layer_modulerTr&r&r'rjbs*         zRAEDecoder.forward)rrrrrrErrrETrtrHrHr\r_)rUrVrWrXr.rrrrkrar0rYr)rtuplerrSrjrlr&r&rgr'rs|     5" # rc*seZdZdZdZdgZdgZe              dCdede de de de de de de de de de dBde de dBd e dBd!e e Be j BdBd"e e Be j BdBd#ed$ed%ed&ef(fd'd( ZdDd)e j d*e jdBd+e j fd,d-Zd)e j d+e j fd.d/Zd)e j d+e j fd0d1Zd2e j d+e j fd3d4Zd2e j d+e j fd5d6ZdDd)e j d*e jdBd+e j fd7d8Ze dEd)e j d9ed*e jdBd+ee e j Bfd:d;Zd2e j d+e j fdd?Z dEd@e j d9ed*e jdBd+ee e j BfdAdBZZS)GAutoencoderRAEa Representation Autoencoder (RAE) model for encoding images to latents and decoding latents to images. This model uses a frozen pretrained encoder (DINOv2, SigLIP2, or MAE) with a trainable ViT decoder to reconstruct images from learned representations. This model inherits from [`ModelMixin`]. Check the superclass documentation for its generic methods implemented for all models (such as downloading or saving). Args: encoder_type (`str`, *optional*, defaults to `"dinov2"`): Type of frozen encoder to use. One of `"dinov2"`, `"siglip2"`, or `"mae"`. encoder_hidden_size (`int`, *optional*, defaults to `768`): Hidden size of the encoder model. encoder_patch_size (`int`, *optional*, defaults to `14`): Patch size of the encoder model. encoder_num_hidden_layers (`int`, *optional*, defaults to `12`): Number of hidden layers in the encoder model. patch_size (`int`, *optional*, defaults to `16`): Decoder patch size (used for unpatchify and decoder head). encoder_input_size (`int`, *optional*, defaults to `224`): Input size expected by the encoder. image_size (`int`, *optional*): Decoder output image size. If `None`, it is derived from encoder token count and `patch_size` like RAE-main: `image_size = patch_size * sqrt(num_patches)`, where `num_patches = (encoder_input_size // encoder_patch_size) ** 2`. num_channels (`int`, *optional*, defaults to `3`): Number of input/output channels. encoder_norm_mean (`list`, *optional*, defaults to `[0.485, 0.456, 0.406]`): Channel-wise mean for encoder input normalization (ImageNet defaults). encoder_norm_std (`list`, *optional*, defaults to `[0.229, 0.224, 0.225]`): Channel-wise std for encoder input normalization (ImageNet defaults). latents_mean (`list` or `tuple`, *optional*): Optional mean for latent normalization. Tensor inputs are accepted and converted to config-serializable lists. latents_std (`list` or `tuple`, *optional*): Optional standard deviation for latent normalization. Tensor inputs are accepted and converted to config-serializable lists. noise_tau (`float`, *optional*, defaults to `0.0`): Noise level for training (adds noise to latents during training). reshape_to_2d (`bool`, *optional*, defaults to `True`): Whether to reshape latents to 2D (B, C, H, W) format. use_encoder_loss (`bool`, *optional*, defaults to `False`): Whether to use encoder hidden states in the loss (for advanced training). Frszdecoder.decoder_pos_embedrAr rrrrrGNrrHT?r=encoder_hidden_sizeencoder_patch_sizeencoder_num_hidden_layersrrrrr+encoder_input_sizerBrencoder_norm_meanencoder_norm_std latents_mean latents_std noise_tau reshape_to_2duse_encoder_lossscaling_factorc st|tvrtd|dttdtdtffdd dtjt Bt BdBdtjdBfdd}||}|j ||d | |_ t ||_t||_t||_t|}|j |d krmtd |j d |d t| }|d krytdt||||d|_t||_|j |d}tt|}|||krtd|d||}| dur|} nt| } | |krtd| d|d|d|d | durgd} |durgd}tj| tjddddd}tj|tjddddd}|jd|dd|jd|dd||}|durtd}|jd |dd|dur"td}|jd!|ddtt|t|t|t|t|t|t|t| t| d" |_t||_t||_ t| |_!d#|_"dS)$NrJz'. Available: valuercsXt|tjr|St|trfdd|DSt|tr*fdd|DS|S)Ncg|]}|qSr&r&rv_to_config_compatibler&r'rzJAutoencoderRAE.__init__.._to_config_compatible..crr&r&rrr&r'rr) isinstancer0rYdetachcputolistrlistrrr&r'rs   z6AutoencoderRAE.__init__.._to_config_compatiblecSs4|durdSt|tjr|Stj|tjdS)Nr6)rr0rYrclonetensorfloat32rr&r&r'_as_optional_tensors   z4AutoencoderRAE.__init__.._as_optional_tensor)rrrzencoder_input_size=z) must be divisible by encoder_patch_size=.zCpatch_size must be a positive integer (this is decoder_patch_size).)r=r>r+r?rzComputed num_patches=z must be a perfect square.z image_size=z1 must equal decoder_patch_size*sqrt(num_patches)=z for patch_size=z and computed num_patches=)g ףp= ?gv/?gCl?)gZd;O?gy&1?g?rrr encoder_meanTr encoder_std _latents_mean _latents_std) r>rrrrrr+rrBF)#r`ra_ENCODER_FORWARD_FNSr/sortedkeysrr0rYrrrrrrrrrrr.rRencoder_encoder_forward_fnrrrrrronesrdecoderrdecoder_patch_sizedecoder_image_size use_slicing)rfr=rrrrrrrr+rrBrrrrrrrrrrlatents_std_tensorrrgridderived_image_sizeencoder_mean_tensorencoder_std_tensorlatents_mean_tensorrgrr'ras &              zAutoencoderRAE.__init__r generatorrcCsP|jtj|dfd|jd|j|j|d}||t|j||j|jdS)Nr)rr)r5r6r)rr5r6) rr0randrndimr5r6r r-)rfrr noise_sigmar&r&r'_noisingDs"zAutoencoderRAE._noisingcCsp|j\}}}}||jks||jkrtj||j|jfddd}|jj|j|jd}|jj|j|jd}|||S)NrFrr) r-rrrrr4r5r6r)rfrrr7r8meanstdr&r&r'_resize_and_normalizeKs z$AutoencoderRAE._resize_and_normalizecCs4|jj|j|jd}|jj|j|jd}|||S)Nr)rr4r5r6r)rfrrrr&r&r'_denormalize_imageUs z!AutoencoderRAE._denormalize_imagezcCs8|jj|j|jd}|jj|j|jd}|||dSNrgh㈵>rr4r5r6rrfrrrr&r&r'_normalize_latentsZz!AutoencoderRAE._normalize_latentscCs8|jj|j|jd}|jj|j|jd}||d|Srrrr&r&r'_denormalize_latents_rz#AutoencoderRAE._denormalize_latentsc Cs||}|jjdkr||j||jj}n||j|}|jr,|jdkr,|j||d}|j rX|j \}}}t t |}|||krIt d|d|dd||||}n|}||}|jjdkrk||jj}|S) NrFrrzToken length n=z/ is not a perfect square; cannot reshape to 2D.rrr)rrQr=rrrtrainingrrrr-r.rr/ transposerrrr) rfrrtokensrnrsiderr&r&r'_encodeds"        zAutoencoderRAE._encodercs\jr|jddkrtjfdd|dDdd}nj|d}|s)|fSt|dS)Nrrcsg|] }j|dqS)r)r$)rx_slicerrfr&r'rsz)AutoencoderRAE.encode..rr)latent)rr-r0rsplitr$r)rfrrrlatentsr&r&r'encodes & zAutoencoderRAE.encodec Cs|jjdkr ||jj}||}|jr+|j\}}}}|||||dd}n|}|j|ddj }|j |}| |}|j |j dS)NrrrTr)r5)rQrrrr-rr rrrTrrr4r5) rfrrrr7r8r!rTx_recr&r&r'_decodes      zAutoencoderRAE._decodecsVjr|jddkrtjfdd|dDdd}n|}|s&|fSt|dS)Nrrcsg|]}|qSr&)r-)rz_slicerfr&r'rsz)AutoencoderRAE.decode..rsample)rr-r0rr(r-r)rfrrdecodedr&r/r'decodes $  zAutoencoderRAE.decoder1cCs:|j|d|dd}|j|ddd}|s|fSt|dS)NF)rrrr+r0)r*r3r)rfr1rrr)r2r&r&r'rjs  zAutoencoderRAE.forward)rArrrrrrrrrGNrNNNNrHTFrr_)TN)T) rUrVrWrX _supports_gradient_checkpointing_no_split_modules"_keys_to_ignore_on_load_unexpectedrrkr.rrr0rYrrrra Generatorrrrrrr$r rr*r-rr3rjrlr&r&rgr'rs/     " "   &  r)r<)? dataclassesrmathrtypingrr0torch.nnrbtorch.nn.functional functionalrconfiguration_utilsrrutilsrr utils.accelerate_utilsr utils.import_utilsr utils.torch_utilsr transformersr rrrrr activationsrr}rattention_processorrrrmodeling_utilsrvaerrr get_loggerrUloggerModulerYr(r*r.r;rkrRrrSr[rmrsrrr&r&r&r'sb                4   0&