o -ic@sUddlZddlmZddlmZddlmZddlZddlm Z ddl m m Z ddl mZddlmZddlmZddlmZdd lmZmZmZeeeefBZeed <eeeefejBZeed <d d Z e dZ!e dZ"e dZ#e dZ$e Z%deeefdedefddZ&de'eeefdede'efddZ(Gddde j)Z*Gddde j)Z+Gddde j)Z,Gd d!d!e j-Z.Gd"d#d#eZ/Gd$d%d%eZ0Gd&d'd'eZ1Gd(d)d)e j)Z2Gd*d+d+e j)Z3dS),N)Iterable)repeat) TypeAlias) rearrange)PretrainedConfig)QuantizationConfig)default_weight_loader)InternParallelAttentionInternVisionEncoderInternVisionEncoderLayer input_dim_tnorm_tcsfdd}|S)Ncs*t|trt|tst|Stt|SN) isinstancerstrtupler)xn]/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/model_executor/models/radio.pyparse#sz_ntuple..parser)rrrrr_ntuple"s rsize patch_sizereturncCs|\}}||||Srr)rrhwrrr calc_seq_len2sr"sizescsfdd|DS)Ncsg|]}t|qSr)r").0rrrr 8sz!calc_seq_lens..r)r#rrr%r calc_seq_lens7sr'c sTeZdZ    ddededededBdedBf fd d Zd ejfd d ZZ S)ClsTokenrTNndim num_tokensenabledregister_multiple num_registerscst||_||_d|_||_|r6|r||_n |r"||||_|d}tt ||j|||_ nd|_ |j|j|_ dS)Nr) super__init__r)r+r-r*nn Parametertorchrandntoken num_patches)selfr)r*r+r,r-scale __class__rrr0<s$ zClsToken.__init__rcCs@|jdur|S|jd|jddd}tj||gdd}|SNrrdim)r5 unsqueezeexpandshaper3cat)r7rr5rrrforward\s zClsToken.forward)rTNN) __name__ __module__ __qualname__intboolr0r3TensorrC __classcell__rrr9rr(;s" r(cseZdZ            ddej d3ej dBd4e eefdBdej fd5d6Z  d>d7ed3ej dBd4e eefdBdej fd8d9Zd7ede eeffd:d;ZZS)?ViTPatchGeneratorTFNrr embed_dim input_dimsabs_posnormalize_patches cls_tokenmax_input_dims pos_dropoutreturn_pos_encnum_cls_tokensr,r- patch_biascsTtt|tr||f}|dur|}t|tr||f}tfdd|D}||k|_||_| |_t||d}|_ ||_ ||_ |d|_ |d|_ tfdd|D|_|j |j |_||_t|_t|fd| i||_|r|d}ttjd|j|fi|||_t|| || | d |_|rt||_dSt|_dS) Nc3s&|]}tt|VqdSr)rGmathceilr$dr%rr s z-ViTPatchGenerator.__init__..)devicedtyperrc3s|]}|VqdSrrrYr%rrr[biasr.)r*r+r,r-)r/r0rrGrcpe_moderSrTdictrrOrMnum_rowsnum_colsrNr6rR Im2Patches im_to_patchesViTPatchLinearembedderr1r2r3r4 pos_embedr(rQ LayerNormIdentitypatch_normalizer)r7rrMrNrOrPrQrRrSrTrUr,r-rVr\r]factoryr8r9r%rr0ms\        zViTPatchGenerator.__init__r imgs_sizesrcCs|dur||}|j||d\}}|j||d}n||}|j||jddd\}}||}||}|jr>||fS|S)Nrmr input_size) rgapply_pos_enc_dynamiccls_token_dynamic embed_patches apply_pos_encrArQrkrT)r7rrmpatchespos_encrrrrCs     zViTPatchGenerator.forwardruc Cs|js|dfSd}g}|D]P}t||j}|dd|||ddf}|j|jd|d}||} tj|ddd|ddf| |dd||dddfgdd}||||7}q |rgtj|ddnd} || fS)Nrrorr=)rOr"r get_pos_encrAr3rBappend) r7rurmcurrent_length pos_enc_listr seq_length img_patchesrvimg_patches_with_pos full_pos_encrrrrqs(   z'ViTPatchGenerator.apply_pos_enc_dynamiccCs|jjs|Sg}d}t||jD],}|jjd|jddd}||||dd|||ddf||7}qt j |ddSr;) rQr+r'rr5r?r@rArxr3rB)r7rurmoutryseq_len class_tokenrrrrrs  $ z#ViTPatchGenerator.cls_token_dynamiccC|jjSr)rQr+r7rrrapply_cls_tokenz!ViTPatchGenerator.apply_cls_tokencCrr)rQr*rrrrrUrz ViTPatchGenerator.num_cls_tokenscCrr)rQr6rrrrnum_cls_patchesrz!ViTPatchGenerator.num_cls_patchescCrr)rQr-rrrrr-rzViTPatchGenerator.num_registerscCs |j|jSr)rUr-rrrrnum_skips zViTPatchGenerator.num_skip src_embed targ_embedcCs|j|jkr8tt|jd}|d|jdksJdt|d||d}tj||j|jfdddd }t|d }|j |dS) Nrrz*Unable to interpolate non-square embeddingzb (h w) c -> b c h w)r r!bicubicTFrmode align_corners antialiaszb c h w -> b (h w) c) rArGrWsqrtrF interpolaterbrcdatacopy_)r7rrsrc_sizerrr _load_embed s"   zViTPatchGenerator._load_embedsrc_proj_weighttarg_proj_weightcCs|j|jkr=tt|jdd}|dd|jdks!Jdt|dd||d}tj||j|jfddd d }t|d }|j |dS) Nrrrz+Unable to interpolate non-square patch sizezb (c h w) -> b c h w)cr r!rTFrzb c h w -> b (c h w)) rArGrWrrrrrrr)r7rrsrc_patch_sizerrr_load_projection s*   z"ViTPatchGenerator._load_projectioncCs||}||}|Sr)rerg)r7rrurrrrs;s  zViTPatchGenerator.embed_patches patch_idxsrpcCsr|js|S||jd||}|jr1|jdkr1tj|jddd|j|jd|jk}t ||d}n|}|||fS)Nrrr]r\) rOrwrAtrainingrSr3randr]r\where)r7rurrprvkeeps pos_enc_droprrrrt@s zViTPatchGenerator.apply_pos_enc batch_sizecs||durj}n tfdd|D}||}|dur|S|ddd|jd}tj||jdddd|d}|S)Nc3s|]}|jVqdSrr%rYrrrr[asz0ViTPatchGenerator.get_pos_enc..r<rr)r>index)rNr_get_pos_embeddingsr?r@rAr3gather)r7rrrprNrhexp_patch_idxsrrrrwXs zViTPatchGenerator.get_pos_enccs<|j|jfkr |jS|jd|j|jddddd}fdd}|jr|jrtd}t j |dd|j d d||}t d }| }t t j |dd|j d |||} || } |d| } t j| | gdd dd} t j |ddd|j d d| } t jddd|j d d |dd}t jddd|j d ddddf|dd}t j||gdd }|| | }|ddtj||ddd|dddd|j}nt}tj|||fddd|j}||}n||}|jddkrtj|ddd|j}|dddd}|S)Nrr<rrrcs\d|jdkr|dddddf}d|jdkr,|dddddf}|S)Nr.rr<)rA)rhrNrr window_selectws z.window_selectg?r\g?r=)stepsr\NNbilinearzerosT)gridr padding_moder)rrrr)rbrcrhreshapepermuter`rrWrr3rr\logexpstackclamp_linspacer@mul_sub_r grid_samplefloattor]maxrrAflatten)r7rrNrhr min_scaler8 aspect_min aspect_maxaspectscale_xscale_yscale_xypos_xylin_xlin_ylin_xygrid_xymax_dimrrrros        z%ViTPatchGenerator._get_pos_embeddings) TFFNrLFrNNFNNrr)rDrErFrGr rHrr0r3rIlistrrCrqrrpropertyrrUrr-rr1r2rrrsrtrwrrJrrr9rrKls     H           "rKcs8eZdZdeffdd ZdejdejfddZZS)rdrcst||_dSrr/r0r)r7rr9rrr0s  zIm2Patches.__init__rrcCsb|jdkr|d}|ddd}|S|jd|j}|jd|j}t|d||j||jd}|S)Nrrrrr<z*b c (py yy) (px xx) -> b (py px) (c yy xx))pyyypxxx)rrrrAr)r7rrurrrrrrCs  zIm2Patches.forward) rDrErFrGr0r3rIrCrJrrr9rrdsrdcs,eZdZddededeffdd ZZS)rfFrrMr_c s,tjd|d|fd|i|||_dS)Nrrr_r)r7rrMr_rlr9rrr0s" zViTPatchLinear.__init__)F)rDrErFrGrHr0rJrrr9rrfs$rfcs8eZdZ ddejdejdBdejffdd ZZS)RadioParallelAttentionNr attn_maskrc s|dur t|S|j\}}}||\}}|jddd\}}} |jr,|||\}}||||j|j }||||j|j }| |||j|j } dd||| fD\}}} t j ||| ||j d} | dd||d} || \} }| S) Nrr<r=css|] }|ddVqdS)rrN) transposer$trrrr[sz1RadioParallelAttention.forward..)rr8rr)r/rCrAqkvchunkqk_normalization_apply_qk_normviewnum_heads_per_partitionhead_dimrscaled_dot_product_attentionr8rrproj) r7rrBN_rqkvrr9rrrCs"   zRadioParallelAttention.forwardr)rDrErFr3rIrCrJrrr9rrsrc<eZdZd fdd Z d dejdejdBfddZZS) RadioVisionEncoderLayerrNctj|dti|dS)Nattn_cls)r/r0rr7argskwargsr9rrr0z RadioVisionEncoderLayer.__init__ hidden_statesrcCs<||j|||d|j}|||||j}|SN)r)attnnorm1ls1mlpnorm2ls2)r7rrrrrrCs zRadioVisionEncoderLayer.forwardrNrrDrErFr0r3rIrCrJrrr9rrrcr) RadioVisionEncoderrNcr)N layer_cls)r/r0rrr9rrr0rzRadioVisionEncoder.__init__ inputs_embedsrcCs |}|jD]}|||d}q|Sr)layers)r7rrr encoder_layerrrrrCs zRadioVisionEncoder.forwardrrrrrr9rr rrcseZdZddgiZ  ddddddededBdedBd ed ed df fd d Zdee eefBfddZ ddZ de e eefde jd e jfddZ dde jde jdBd e jfddZZS)RadioInternVisionModelrNrnum_hidden_layers_overridenum_dummy_headsprefixconfig quant_configrrrrc st||_|t|j|j\|_|_|_ t t |j |j|j}t dd|jD}t|j|j|j|d|jr?t|nd|jd|_t|||||dd|_dS)Ncss|]}|dVqdS)nameNrrrrrr[3r^z2RadioInternVisionModel.__init__..Tr)rNrRrQrUr,z.encoderrrrrr)r/r0r_init_img_size to_2tupler image_sizeimg_size grid_sizer6rGround cpe_max_sizesetteachersrK hidden_sizecls_token_per_teacherlenr,patch_generatorrencoder)r7rrrrr max_img_sizeunique_teachersr9rrr0!s2    zRadioInternVisionModel.__init__r cCsF|durdSt|}tddt||D}|d|d}|||fS)N)NNNcSsg|]\}}||qSrr)r$sprrrr&Jsz9RadioInternVisionModel._init_img_size..rr)r rzip)r7rr r r6rrrr Fs  z%RadioInternVisionModel._init_img_sizecCs|jSr) embeddingsrrrrget_input_embeddingsNsz+RadioInternVisionModel.get_input_embeddingsrmr\c sz|jj}|jjt||}fdd|D}t|}tj||tj|d}d}|D]} || } d||| || f<| }q(|S)Ncsg|]}|qSrr)r$rrrrr&XszLRadioInternVisionModel.create_inter_image_attention_mask..rrT)rrrr'sumr3rrH) r7rmr\rseq_lens patch_counts total_patchesmask start_idx patch_countend_idxrrr!create_inter_image_attention_maskQs  z8RadioInternVisionModel.create_inter_image_attention_maskrcCsH|j||d}d}|durt|dkr|j||jd}|j||d}|S)Nrnrr)rr)rrr'r\r)r7rrmrrencoder_outputsrrrrCjszRadioInternVisionModel.forwardrr)rDrErFpacked_modules_mappingrrrGrr0rr rrr3r\rIr' FloatTensorrCrJrrr9rrsN% rcseZdZddgiZ ddddddededBdedBd ed ed df fd d Z  dddde j dBde j dBde j dBd e e j e j ffddZ d eefddZ dde j dee eefdBd e e j e j ffddZZS) RadioModelrNrrrrrrrrrcsjt||_t|||||d|_d}|jr0tddt|jD}| dkr0| d|||_ dS)NrcSs g|] \}}|ddr|qS) use_summaryT)get)r$irrrrr&s z'RadioModel.__init__..r summary_idxs) r/r0rrmodelrr3tensor enumeratenumelregister_bufferr/)r7rrrrrr/r9rrr0s"    zRadioModel.__init__rn pixel_values pixel_embedsrmcCs|j||d}|j||dS)Nrn)r0_extract_final)r7r5r6rmyrrrrCszRadioModel.forwardcCsPt}t|}t|trt|}nt|}|D]\}}|ds%q|tdd}|dvr2q|dr8qd}|drKd|ddd}n@|dr\d|ddd}n/|d r|d} t| d kr| d} d | d d} | d vs| d rqd| d| }|r||vr||} t | dt } | | || |q|S)Nz radio_model.>r/zinput_conditioner.zmodel.patch_generator..rr<rz model.blocks.rr>rr)zls1.zls2.zmodel.encoder.layers. weight_loader) rranamed_parametersrritems startswithrsplitjoingetattrradd)r7weights loaded_params params_dict weights_listrweightsubvllm_keyparts layer_idxsuffixparamr:rrr load_weightssB              zRadioModel.load_weightsr8cCs|jjj}|jjj}|jjj}|dur(|ddd|f}|dd|df}nNg}g} d} t||D]4} |dd| || || ddf} || |dd| | |ddf} | | | || 7} q3tj| dd}tj|dd}|j dur|dd|j f}n|}| d|fS)Nrrr=) r0rrrrUr'rxr3rBr/r)r7r8rmrrrU all_summaryall_feat all_patches summaries current_posr6rusummary bb_summaryrrrr7s.       zRadioModel._extract_finalrr)rDrErFr)rrrGrr0r3rIrr*rCrrMrr7rJrrr9rr+zsV  2r+)4rWcollections.abcr itertoolsrtypingrr3torch.nnr1torch.nn.functional functionalreinopsr transformersr'vllm.model_executor.layers.quantizationr-vllm.model_executor.model_loader.weight_utilsr%vllm.model_executor.models.intern_vitr r r rGrr __annotations__rrIr r to_1tupler  to_3tuple to_4tuple to_ntupler"rr'Moduler(rKrdLinearrfrrrrr+rrrrsB         &1X^