o ei1@szddlZddlmZddlmZddlmZmZddlZddl m Z ddl m m Z ddl mZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZddlm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,ddl-m.Z.m/Z/m0Z0m1Z1ddl2m3Z3m4Z4m5Z5ddl6m7Z7ddl8m9Z9m:Z:m;Z;edGddde j<Z=Gddde j<Z>Gddde j<Z?Gdd d e j<Z@Gd!d"d"e j<ZAGd#d$d$e j<ZBd%d&ZCd'ejDd(ejDd)ejDd*ejDd+eEejDejDff d,d-ZFd.ejDd/eGd+ejDfd0d1ZH 2d_d3e jd>e j<ZKGd?d@d@e ZLGdAdBdBe j<ZMdCdDZNd`dEdFZOGdGdHdHe j<ZPGdIdJdJe j<ZQGdKdLdLe ZRee/dMdNGdOdPdPe$ZSe/GdQdRdRe*ZTGdSdTdTeTZUe/GdUdVdVeTZVe/GdWdXdXeTZWee/dYdNGdZd[d[e$ZXGd\d]d]eTeZYgd^ZZdS)aN)Callable) dataclass)AnyOptional) LayerNorm)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPooling ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupletorch_compilable_check)is_flash_attention_requestedmaybe_autocastmerge_with_config_defaults)capture_outputs) Glm4vConfigGlm4vTextConfigGlm4vVisionConfigRMSNormcsFeZdZd deddffdd Zdejdejfdd Zd d ZZ S) Glm4vRMSNormư>epsreturnNcs&ttt||_||_dS)z; Glm4vRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizer( __class__f/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/glm4v/modeling_glm4v.pyr+2s  zGlm4vRMSNorm.__init__ hidden_statescCsJ|j}|tj}|djddd}|t||j}|j||S)NT)keepdim) dtypetor.float32powmeanrsqrtr1r0)r2r8 input_dtypevariancer6r6r7forward:s  zGlm4vRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler0shaper1r2r6r6r7 extra_reprAszGlm4vRMSNorm.extra_repr)r') __name__ __module__ __qualname__floatr+r.TensorrDrH __classcell__r6r6r4r7r&0sr&cs,eZdZddeffdd ZddZZS) Glm4VisionMlpFbiascslt|j|_|j|_tj|j|j|d|_tj|j|j|d|_tj|j|j|d|_ t |j |_ dSNrP) r*r+r3out_hidden_sizeintermediate_sizer,Linear gate_projup_proj down_projr hidden_actact_fn)r2configrPr4r6r7r+Fs zGlm4VisionMlp.__init__cCs ||||||SN)rXrZrVrWr2 hidden_stater6r6r7rDOs zGlm4VisionMlp.forwardF)rIrJrKboolr+rDrNr6r6r4r7rOEs rOcs<eZdZdeddffdd ZdejdejfddZZS) Glm4vVisionPatchEmbedr[r)NcsVt|j|_|j|_|j|_|j|_|j|j|jg}tj|j|j||d|_ dS)N) kernel_sizestride) r*r+ patch_sizetemporal_patch_size in_channelsr3 embed_dimr,Conv3dproj)r2r[rbr4r6r7r+Ts zGlm4vVisionPatchEmbed.__init__r8cCsD|jjj}|d|j|j|j|j}||j|dd|j}|S)Nr:r<) rir0r<viewrfrerdr=rg)r2r8 target_dtyper6r6r7rD^s zGlm4vVisionPatchEmbed.forward rIrJrKr$r+r.rMrDrNr6r6r4r7raSs racsLeZdZUejed<d dededdffdd Zd edejfd d Z Z S) Glm4vVisionRotaryEmbeddinginv_freq@dimthetar)NcsJt||_||_d|tjd|dtjd|}|jd|dddS)N?rr9rjroF persistent)r*r+rqrrr.arangerLregister_buffer)r2rqrrror4r6r7r+js  z#Glm4vVisionRotaryEmbedding.__init__seqlencCs*tj||jj|jjd}t||j}|S)Ndevicer<)r.rvrorzr<outer)r2rxseqfreqsr6r6r7rDqsz"Glm4vVisionRotaryEmbedding.forward)rp) rIrJrKr.rM__annotations__intrLr+rDrNr6r6r4r7rngs rnc sJeZdZd dededededdf fdd Zd ejdejfd d Z Z S)Glm4vVisionPatchMergerFrq context_dimrYrPr)Ncstttj|||d|_t||_tj|||d|_tj|||d|_tj|||d|_ t |_ t ||_ dSrQ)r*r+r,rUrirpost_projection_normrVrWrXGELUact1r rZ)r2rqrrYrPr4r6r7r+xs   zGlm4vVisionPatchMerger.__init__r^cCs:||}|||}||||||Sr\)rirrrXrZrVrWr]r6r6r7rDs  zGlm4vVisionPatchMerger.forwardr_) rIrJrKrstrr`r+r.rMrDrNr6r6r4r7rws$ rcs2eZdZdeffdd ZdejfddZZS)Glm4vVisionEmbeddingsr[cs^t||_|j|_|j|_|j|_|j|jd|_|j|_t |j|j|_ d|_ dS)Nr9bicubic) r*r+r[r3rg image_sizerd num_patches num_positionsr, Embeddingposition_embeddinginterpolated_methodr2r[r4r6r7r+s  zGlm4vVisionEmbeddings.__init__r)csd|jj}|jd}|j}ttrtj|tjd|jd} t | d} | | | | ddd dj |tjd} tfddttDj |tjd} tfddttDj |tjd} |d| dd}|d| dd}tj||fd d  d d}tj| ||jd d d }|dd  dd}| |j |j}||}|S)a Forward pass with integrated position encoding adaptation using 2D interpolation. Args: embeddings: Input embeddings tensor lengths (torch.Tensor): Sequence lengths for each image in the batch. image_shapes (torch.Tensor): Tensor of shape [batch_size, 3] representing the image shapes (t, h, w). h_coords (torch.Tensor): Tensor of shape [total_seq] representing the h coordinate for each patch. w_coords (torch.Tensor): Tensor of shape [total_seq] representing the w coordinate for each patch. Returns: torch.Tensor: Embeddings with adapted position encoding added. r!ryrg?r9c"g|] }|df|qSr!repeat.0i image_shapeslengthsr6r7 "z1Glm4vVisionEmbeddings.forward..cr)r9rrrr6r7rrr:rqFborder)mode align_corners padding_mode)rr0rFrz isinstancelistr.tensorlongrrkpermute unsqueezer=r>catrangelenstackF grid_samplersqueezer<)r2 embeddingsrrh_coordsw_coordspos_embed_weightr3rz orig_size_sq orig_size pos_embed_2dtarget_htarget_wnorm_wnorm_hgridinterpolated_embed_fp32adapted_pos_embed_fp32adapted_pos_embedr6rr7rDs:      "" zGlm4vVisionEmbeddings.forwardrmr6r6r4r7rs rcCsH|dd|jddf}|d|jdddf}tj| |fddS)*Rotates half the hidden dims of the input..Nr:r9r)rFr.rxx1x2r6r6r7 rotate_halfsrqkcossinr)cCs|j}|j}||}}|d|d}}||t||}||t||}||}||}||fS)N)r<rLrrr=)rrrr orig_q_dtype orig_k_dtypeq_embedk_embedr6r6r7apply_rotary_pos_emb_visions  rr8n_repcCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r!N)rFexpandreshape)r8rbatchnum_key_value_headsslenhead_dimr6r6r7 repeat_kvs 0rmodulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur | |} tjj| dtjd |j } tjj | ||j d} t| | } | dd } | | fS)Nr9rr:rqr<)ptrainingr!)rnum_key_value_groupsr.matmul transposer, functionalsoftmaxr>r=r<rr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputr6r6r7eager_attention_forwards  rc sfeZdZdeddffdd Z  d dejdejdejdBd eejejfdBdejf d d ZZ S) Glm4vVisionAttentionr[r)Ncst|j|_|j|_|j|j|_d|_tj|j|jd|j d|_ tj|j|jdd|_ |jd|_ ||_ |j|_d|_dS)Nr!rrRF)r*r+r3rq num_headsrrr,rUattention_biasqkvrirr[attention_dropout is_causalrr4r6r7r+s   zGlm4vVisionAttention.__init__r8 cu_seqlensrotary_pos_embposition_embeddingsc sr|jd}||djdddddd\}}} |\} } t||| | \}}|ddd}|ddd}| ddd} t j j t tj r~|dd|dd} ||| fdjjsndnj||| | dd\} }n,|dd|ddfd d ||| fD}fd d t|D}tj|dd } | |d} | } | S) Nrrr:r!r9rF)rrr cu_seq_lens_q cu_seq_lens_k max_length_q max_length_krcs g|] }tj|ddqS)r9r)r.splittolist)rr)rr6r7rGsz0Glm4vVisionAttention.forward..c sDg|]\}}}|||fdjjsdnjdddqS)NrF)rrrrr)rrr)rrrv)attention_interfacerr2r6r7rKs$   r)rFrrrrunbindrrrr get_interfacer[_attn_implementationrrmaxrrrzipr.rrri)r2r8rrrr seq_length query_statesrrrr max_seqlenr_splits attn_outputsr6)rrrr2r7rDsR (     zGlm4vVisionAttention.forwardNN) rIrJrKr$r+r.rMrErDrNr6r6r4r7r src s^eZdZd fdd Z  d dejdejdejdBdeejejfdBdejf d d ZZS) Glm4vVisionBlockr)NcsJtt|j|jd|_t|j|jd|_t||_t |dd|_ dS)Nr(FrR) r*r+r&r3 rms_norm_epsnorm1norm2rattnrOmlprr4r6r7r+as  zGlm4vVisionBlock.__init__r8rrrcKs<||j||f|||d|}||||}|S)N)rrr)rr rr )r2r8rrrrr6r6r7rDhszGlm4vVisionBlock.forwardr)Nr) rIrJrKr+r.rMrErDrNr6r6r4r7r `s r c seZdZUejed<ddeffdd Ze   ddedBde dde dBd e d e ffd d Z eed dZddZZS)Glm4vTextRotaryEmbeddingroNr[cst|j|_|j|_||_|jjd|_|j}|jdkr$t |j}||j|\}|_ |j d|dd|j d| dd|j dgd|_dS) N rope_typedefaultroFrtoriginal_inv_freq mrope_section) r)r*r+max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr[rope_parametersrcompute_default_rope_parametersrattention_scalingrwclonegetr)r2r[rz rope_init_fnror4r6r7r+~s   z!Glm4vTextRotaryEmbedding.__init__rzz torch.deviceseq_lenr)z torch.Tensorc Cst|jd}|jdd}t|ddp|j|j}t||}d}d|tjd|dtjdj |tj d |}||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorrsrNrr9rjry) rrgetattrr3num_attention_headsrr.rvint64r=rL) r[rzr!baser#rrqattention_factorror6r6r7rs  &z8Glm4vTextRotaryEmbedding.compute_default_rope_parametersc Cs|jdddddfd|jddd}|dddddddf}t|jjtr7|jjdkr7|jjnd}t|dd2|| dd}| ||j }t j ||fdd }||j}||j} Wdn1stwY|j|jd | j|jd fS) Nrr!r:mpscpuF) device_typeenabledr9rrj)rorLrrFrrztyperrr apply_mroperr.rrrrr=r<) r2r position_idsinv_freq_expandedposition_ids_expandedr+r}embrrr6r6r7rDs, &z Glm4vTextRotaryEmbedding.forwardcCs2|}|j|dd}tjddt|Ddd}|S)Nr:rcSsg|] \}}||dqS)rr6)rrchunkr6r6r7rsz8Glm4vTextRotaryEmbedding.apply_mrope..)rr.r enumerate)r2r}rsectionchunksresultr6r6r7r.sz$Glm4vTextRotaryEmbedding.apply_mroper\)NNN)rIrJrKr.rMr~r#r+ staticmethodrrrErLrno_gradrrDr.rNr6r6r4r7r{s(    rcCs>|ddddf}|ddddf}tj| |fdddS) r.rNr9r!r:rr)r.rflattenrr6r6r7rotate_half_llmsr;c Cs||}||}|dd|jddfjddd}|dd|jddfjddd}|jd}|dd|f|d|df}}|dd|f|d|df}} ||t||} ||t||} tj| |gdd} tj| | gdd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .Nr:r9r)rrFrepeat_interleaver;r.r) rrrr unsqueeze_dim rotary_dimq_rotq_passk_rotk_passrrr6r6r7apply_rotary_pos_embs  $$ ""rCcseZdZdZddededBffdd Z    ddejde ejejfdBd ejdBd e dBd ej dBd e e d e ejejdBe ejdBffddZZS)Glm4vTextAttentionz Multi-headed attention from 'Attention Is All You Need' paper. and "Generating Long Sequences with Sparse Transformers". Nr[ layer_idxcst||_||_|j|_|j|_|j|j|_|j|_|j|j|_ d|_ |j |_ |j |_ |jd|_ tj|j|j|jdd|_tj|j|j|jdd|_tj|j|j|jdd|_tj|j|j|jdd|_dS)NTrrRF)r*r+r[rEr3r%rrrrrrrrr,rUq_projk_projv_projo_projr2r[rEr4r6r7r+s   zGlm4vTextAttention.__init__r8rrpast_key_valuescache_positionrr)cKs"|\}}} ||} ||} ||} | ||d|jdd} | ||d|jdd} | ||d|jdd} |\} }t| | | |\} } |dur_|| |d}|| | |j |\} } t |j j t}||| | | |f|jssdn|j|jd|\}}|||d}||}||fS)Nr:r!r9)rrrLr)rr)sizerFrGrHrkrrrCupdaterErrr[rrrrrrrrI)r2r8rrrKrLrbszq_lenrrrrrr cache_kwargsrrrr6r6r7rDs<      zGlm4vTextAttention.forwardr\NNNN)rIrJrK__doc__r#rr+r.rMrEr LongTensorrrrDrNr6r6r4r7rDs,rDcs2eZdZfddZdejdejfddZZS) Glm4vTextMLPcsPt||_tj|jd|jdd|_tj|j|jdd|_t |j |_ dS)Nr9FrR) r*r+r[r,rUr3rT gate_up_projrXr rY activation_fnrr4r6r7r+@s zGlm4vTextMLP.__init__r8r)cCs4||}|jddd\}}|||}||S)Nr9r:r)rVr3rWrX)r2r8 up_statesgater6r6r7rDHs  zGlm4vTextMLP.forward)rIrJrKr+r. FloatTensorrDrNr6r6r4r7rU?s rUcseZdZdedeffdd Ze      ddejde ejejfdBd ejdBd ej dBd e dBd e dBd ej dBde ej e ej ej fdBffddZZS)Glm4vTextDecoderLayerr[rEcstt|j|_t|||_t||_t|j|jd|_ t|j|jd|_ t|j|jd|_ t|j|jd|_ dS)Nr ) r*r+r3rD self_attnrUrr&r input_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernormrJr4r6r7r+Rs   zGlm4vTextDecoderLayer.__init__NFr8rrr/rK use_cacherLr)c Ksr|} ||}|jd|||||||d|\}} ||}| |}|} ||}||}||}| |}|S)N)r8rrr/rKrarLr6)r]r\r_r^rr`) r2r8rrr/rKrarLrresidualrr6r6r7rD\s*     zGlm4vTextDecoderLayer.forward)NNNNFN)rIrJrKr#rr+rr.rMrErTr r`rZrDrNr6r6r4r7r[Qs4  r[zJ Base class for Llava outputs, with hidden states and attentions. ) custom_introc@sreZdZUdZdZejdBed<dZe dBed<dZ e ejdBed<dZ e ejdBed<dZ ejdBed<dS)Glm4vModelOutputWithPasta[ past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Nlast_hidden_staterKr8 attentions rope_deltas)rIrJrKrSrer.rZr~rKr r8rErfrgrTr6r6r6r7rds  rdcsXeZdZUeed<dZdZdZddgZdZ dZ dZ dZ dZ eedZfd d ZZS) Glm4vPreTrainedModelr[model)imagevideotextTr[r rKr8rfcsRt|t|tr'd|jtjd|jdtjd|j}t |j |dSdS)Nrsrr9rj) r* _init_weightsrrnrrr.rvrqrLinitcopy_ro)r2rror4r6r7rns  &z"Glm4vPreTrainedModel._init_weights)rIrJrKr"r~base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_attention_backendr[rD_can_record_outputsrnrNr6r6r4r7rhs rhc sveZdZUeed<dZdgZeedZ dfdd Z d d Z e e ed ejd ejd eedeeBfddZZS)Glm4vVisionModelr[)rjrkr rmr)Ncstj|_j|_t|_t|_jj }t |d|_ t fddtjD|_tjjjd|_tjjd|_t jjjjjd|_tjjd|_d|_|dS)Nr9csg|]}tqSr6)r )rrr[r6r7rsz-Glm4vVisionModel.__init__..)rqrrYr )rf out_channelsrbrcF)r*r+spatial_merge_sizerdrrra patch_embedr3rrnrr, ModuleListrdepthblocksrrSrTrYmergerr&r post_conv_layernormConv2d downsamplepost_layernormgradient_checkpointing post_init)r2r[rr4r|r7r+s*       zGlm4vVisionModel.__init__c Csg}|D]e\}}}t|dd|}|||j|j||j|j}|dddd}|}t|d|d}|||j|j||j|j}|dddd}|}|tj ||gdd |dqtj |dd}|ddddf }| |} | |d} | |fS)Nr!r:rr9rr)r.rvrrrr~rr:appendrrrrr) r2grid_thwpos_idsthwhpos_idswpos_ids max_grid_sizerotary_pos_emb_fullrr6r6r7 rot_pos_embs4" zGlm4vVisionModel.rot_pos_embr8rrc Ks|||}||}||\}}tj||fdd}||f}t|dddf|dddf|dddfjdtj rE|j ntj d}t j|ddd }|dd|dd} ||| ||dddf|j|dddf|j}|jD] } | |f||d |}q||}|d|j|j|jd}|dd dd}||d|jj}||} t|| d S) a\ hidden_states (`torch.Tensor` of shape `(seq_len, hidden_size)`): The final hidden states of the model. grid_thw (`torch.Tensor` of shape `(num_images_or_videos, 3)`): The temporal, height and width of feature shape of each image in LLM. Returns: `torch.Tensor`: hidden_states. r:rNr!r9rr)r!r)r)rrr)re pooler_output)rrrr.rrrr<cumsumjit is_tracingr<int32rpadrrr=rzrrrkr~rFrrr[rSrr) r2r8rrrimage_type_idsr2rrseqlensblkmerged_hidden_statesr6r6r7rDsL  4   zGlm4vVisionModel.forwardr)rIrJrKr$r~rrrtr rrzr+rrr rr.rMrrrErrDrNr6r6r4r7r{s* r{cseZdZUeed<dZdeffdd Zeee       dde j dBde j dBde j dBd e dBd e jdBd edBd e j dBd eedeeBfddZZS)Glm4vTextModelr[)rlcstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr6)r[)rrEr|r6r7rCz+Glm4vTextModel.__init__..r r|F)r*r+ pad_token_id padding_idx vocab_sizer,rr3 embed_tokensrrnum_hidden_layerslayersr&r normr rotary_embrrrr4r|r7r+<s   zGlm4vTextModel.__init__N input_idsrr/rK inputs_embedsrarLrr)c Ksv|du|duAr td|r|durtjst|jd}|dur&||}|durB|dur2|nd} tj| | |j d|j d}|durV| ddd d|j dd}n|j dkrg|d  d|j dd}|j dkr~|j dd kr~|d} |dd}nd} |j||||| d } tdi| } |} |j| |d }|jD]}|| f| | |||d |}|} q|| } t| |dS)N:You must specify exactly one of input_ids or inputs_embedsr|rr!rzr:rr9N.)r[rrrLrKr/)r/)rr/rKrLr)rerKr6) ValueErrorr.rrr r[rget_seq_lengthrvrFrzrkrndimrrrrr)r2rrr/rKrrarLrpast_seen_tokenstext_position_ids mask_kwargs causal_maskr8r decoder_layer layer_outputsr6r6r7rDLs^        zGlm4vTextModel.forward)NNNNNNN)rIrJrKr#r~rrr+rrr r.rTrMr rZr`rrrErrDrNr6r6r4r7r7sB   rcs(eZdZdZiZdZddgZfddZddZd d Z d'd e j d Bd e j d Bde j d Bde j d Bde e j e j ff ddZee d(de jde j d Bdeede eBfddZee d(de jd e j d Bdeede eBfddZ d)d e j de jde jd Bde jd BfddZ d'd e j d Bde j d Bd e j d Bde j d Bde j d Bde j d Bde j d Bfd d!Zee d*d e j d Bde j d Bd"e j d Bded Bde jd Bde j d Bde jd Bd e j d Bde j d Bd#e j d Bd$e j d Bdeede eBfd%d&ZZS)+ Glm4vModelriFr[r cs:t|t|j|_t|j|_d|_ | dSr\) r*r+r{ _from_config vision_configvisualr text_configlanguage_modelrgrrr4r6r7r+s  zGlm4vModel.__init__cC |jSr\)rget_input_embeddingsrGr6r6r7r zGlm4vModel.get_input_embeddingscC|j|dSr\)rset_input_embeddingsr2rr6r6r7rzGlm4vModel.set_input_embeddingsNrimage_grid_thwvideo_grid_thwrr)c/ Ks|jjj}|jj}|jj}|jj} g} |} |durt| }tjd|j d|j d|j |j d} d\} }d}|dur>| nd}|durH| nd}| | j }t| D]\}}|||dk}| }g}d}|D],}||krtd}n|| krzd}||kr|s|d qk||kr|r|d qk|d qkg}tt|d d D]\}}t|}|dd}|ddd}||||fqg}d}|D]\}}} t|dkr|ddnd}!|d kr>|| \}"}#}$|"|#||$|}%}&}'t|%ddd|&|'}(t|&ddd|%d|'})t|'ddd|%|&d}*|t|(|)|*g|!| d7} d}q|d kr||\}+}#}$|}"|"|#||$|}%}&}'t|%D]C},t|,ddd|&|'}(t|&ddddd|'})t|'dddd|&d}*|t|(|)|*g|!q\|d7}|||dkr|d7}d}|d7}q| |}-|t|-dddd|!d}qtj|dddd}.|. | j | d|||dkf<| |.dt| |qTtj| |j dd} | | fS)aU Calculate the 3D rope index based on image and video's temporal, height and width in LLM. Explanation: Each embedding sequence contains vision embedding and text embedding or just contains text embedding. For pure text embedding sequence, the rotary position embedding has no difference with modern LLMs. Examples: input_ids: [T T T T T], here T is for text. temporal position_ids: [0, 1, 2, 3, 4] height position_ids: [0, 1, 2, 3, 4] width position_ids: [0, 1, 2, 3, 4] For vision and text embedding sequence, we calculate 3D rotary position embedding for vision part and 1D rotary position embedding for text part. Examples: Temporal (Time): 3 patches, representing different segments of the video in time. Height: 2 patches, dividing each frame vertically. Width: 2 patches, dividing each frame horizontally. We also have some important parameters: fps (Frames Per Second): The video's frame rate, set to 1. This means one frame is processed each second. tokens_per_second: This is a crucial parameter. It dictates how many "time-steps" or "temporal tokens" are conceptually packed into a one-second interval of the video. In this case, we have 25 tokens per second. So each second of the video will be represented with 25 separate time points. It essentially defines the temporal granularity. temporal_patch_size: The number of frames that compose one temporal patch. Here, it's 2 frames. interval: The step size for the temporal position IDs, calculated as tokens_per_second * temporal_patch_size / fps. In this case, 25 * 2 / 1 = 50. This means that each temporal patch will be have a difference of 50 in the temporal position IDs. input_ids: [V V V V V V V V V V V V T T T T T], here V is for vision. vision temporal position_ids: [0, 0, 0, 0, 50, 50, 50, 50, 100, 100, 100, 100] vision height position_ids: [0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1] vision width position_ids: [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1] text temporal position_ids: [101, 102, 103, 104, 105] text height position_ids: [101, 102, 103, 104, 105] text width position_ids: [101, 102, 103, 104, 105] Here we calculate the text start position_ids as the max vision position_ids plus 1. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Returns: position_ids (`torch.LongTensor` of shape `(3, batch_size, sequence_length)`) mrope_position_deltas (`torch.Tensor` of shape `(batch_size)`) Nrrr!r<rz)rrFTrjrkrlcSs|dS)Nr!r6)rr6r6r7sz+Glm4vModel.get_rope_index..r:r.r)r[rr~image_token_idvideo_start_token_idvideo_end_token_idr. ones_liker/rFr<rzrr=r4r itertoolsgroupbyrrrrvrkrr:rrrrrr)/r2rrrrrr~rrrmrope_position_deltastotal_input_idsr/ image_index video_indexvideo_group_indeximage_grid_thw_listvideo_grid_thw_listr input_tokensinput_token_typevideo_check_flgtokeninput_type_grouprgroup start_index end_indexllm_pos_ids_listvideo_frame_num modality_type start_idxend_idxst_idxrrr llm_grid_t llm_grid_h llm_grid_wt_indexh_indexw_indexrt_idxtext_len llm_positionsr6r6r7get_rope_indexs <           """  """ $ zGlm4vModel.get_rope_indexpixel_values_videosrcKs||jj}g}|}|D]\}}}td||gd|d} || qtj |dd} |j|f| dd|} | d|jj d} t | j | } | | _ | S)[ pixel_values_videos (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input videos. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. r!rrTr return_dictr:r9)r-rr<rr.rrrrrprodr~rr)r2rrrtemp_frames_hwrrrr repeated_rowflattened_video_grid_thwvision_outputs split_sizes video_embedsr6r6r7get_video_featuresOs$ zGlm4vModel.get_video_features pixel_valuescKsX||jj}|j|f|dd|}|d|jjd}t|j|}||_|S)T pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. Trr:r9) r-rr<rr~rr.rr)r2rrrrr image_embedsr6r6r7get_image_featuresns zGlm4vModel.get_image_featuresrimage_featuresvideo_featuresc Cs.|dur3||tj|jjtj|jdk}|d}||tj|jjtj|jdk}|d}n ||jjk}||jjk}| }| d | |j}|durit |||kd|d|jd| }| d | |j}|durt |||kd|d|jd||fS)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. Nrr:z6Image features and image tokens do not match, tokens: z , features: rz6Video features and video tokens do not match, tokens: )rr.rr[rrrzallvideo_token_idsumr expand_asr=rnumelrF) r2rrrrspecial_image_maskspecial_video_maskn_image_tokensn_video_tokensr6r6r7get_placeholder_masks4     zGlm4vModel.get_placeholder_maskrKcCs.|durdn|}|duo|dup|du}|r1|jdus!|dkr1|j||||d\} } | |_| S|jdur|j\} } } |durb|dd} | |dkd} | d| dddd |j } nt ||| } | ddd d| d |j } |jj| |jjddd}| |j | j d} | Sd} | S)Nr)rrrr:r!rrr)rrgrrFrr masked_fillrkrr=rzr.rvrr<)r2rrrrrrKpast_key_values_lengthcan_compute_mroper/rg batch_sizerrdeltar6r6r7compute_3d_position_idss0   " z"Glm4vModel.compute_3d_position_idsr/rgrLc  Ks(|du|duAr td|dur||}|dur@|j||ddj} tj| dd|j|j} |j ||| d\}}| || }|durj|j || ddj}tj|dd|j|j}|j |||d\}}| ||}|dury|j ||| |||d }|j d d||||| d | }td i|d |jiS) a image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. NrT)rrr)r)r)rrrrrrK)rr/rrKrrLrgr6)rrrrr.rr=rzr<r masked_scatterrrrrdrg)r2rrr/rKrrrrrrgrLrr image_maskrr video_maskoutputsr6r6r7rDsL     zGlm4vModel.forwardrRr\r) NNNNNNNNNNN)rIrJrKrq_checkpoint_conversion_mappingaccepts_loss_kwargsrtr+rrr.rTrMrErrrrZrrrrrr rr rdrDrNr6r6r4r7rs    . %     rzQ Base class for Glm4v causal language model (or autoregressive) outputs. c@seZdZUdZdZejdBed<dZejdBed<dZ e dBed<dZ e ejdBed<dZ e ejdBed<dZejdBed<dS) Glm4vCausalLMOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. NlosslogitsrKr8rfrg)rIrJrKrSrr.rZr~rrKr r8rErfrgrTr6r6r6r7rs rc seZdZiZddiZdZfddZddZdd Ze d,d e j d e j d Bd e edeeBfddZe d,de j de j d Bd e edeeBfddZee d-de j d Bde jd Bde j d Bded Bde j d Bde j d Bde jd Bd e j d Bde j d Bd e j d Bde j d Bdee jBd e edeeBfddZ d.fd!d" Zfd#d$Z d,de j d Bde jd Bdee je jffd%d&Z '  d/d(ed)ede j d Bdee j eeefffd*d+Z Z!S)0Glm4vForConditionalGenerationzlm_head.weightz(model.language_model.embed_tokens.weightFcs<t|t||_tj|jj|jjdd|_ | dS)NFrR) r*r+rrir,rUrr3rlm_headrrr4r6r7r+;s   z&Glm4vForConditionalGeneration.__init__cCrr\)rirrGr6r6r7rBrz2Glm4vForConditionalGeneration.get_input_embeddingscCrr\)rirrr6r6r7rErz2Glm4vForConditionalGeneration.set_input_embeddingsNrrrr)cK|jjd||d|S)r)rrNr6)rir)r2rrrr6r6r7rHs  z0Glm4vForConditionalGeneration.get_video_featuresrrcKr)r)rrNr6)rir)r2rrrr6r6r7rYs z0Glm4vForConditionalGeneration.get_image_featuresrrrr/rKrlabelsrLlogits_to_keepc  Ks|jd|||| | ||||| d | }|d}t| tr"t| dn| }||dd|ddf}d}|durC|j|||jjjd}t |||j |j |j |j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, Glm4vForConditionalGeneration >>> model = Glm4vForConditionalGeneration.from_pretrained("zai-org/GLM-4.1V-9B-Thinking") >>> processor = AutoProcessor.from_pretrained("zai-org/GLM-4.1V-9B-Thinking") >>> messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"}, {"type": "text", "text": "What is shown in this image?"}, ], }, ] >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) >>> inputs = processor(text=[text], images=[image], vision_infos=[vision_infos]) >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "The image shows a street scene with a red stop sign in the foreground. In the background, there is a large red gate with Chinese characters ..." ```) rrrrrr/rrKrrLrN)rrr)rrrKr8rfrgr6)rirrslicer loss_functionr[rrrrKr8rfrg)r2rrr/rKrrrrrrrLrrrr8 slice_indicesrrr6r6r7rDhs8< z%Glm4vForConditionalGeneration.forwardTc  sHtj|f||||||| | | || d | }| s"|r"d|d<d|d<|S)N) rKrrrLr/rrrrrais_first_iterationrr)r*prepare_inputs_for_generation)r2rrKrrrLr/rarrrrr!r model_inputsr4r6r7r"s* z;Glm4vForConditionalGeneration.prepare_inputs_for_generationc s<t||}d}|d}dur|}|dkr*|jjdur*|d|jj}|Sd|vr;|djddkr;|d}t|jdkoJ|jt j t j fv}|ru|ddus[|ddurud d | D}|jj |fi|\}} | |j_n|dd d d }t j|jddt j |jd |j_|d}t j||gdd}|S)NrrKrrr!r9rrcSsi|] \}}|dkr||qS)rr6)rrrr6r6r7 zVGlm4vForConditionalGeneration._prepare_position_ids_for_generation..rr:rr)r*$_prepare_position_ids_for_generationrrrirgrFrr<r.rritemsrrrzerosrzr) r2 inputs_tensor model_kwargstext_positions past_lengthcacher/ is_input_idsvision_positionsrgr4r6r7r&s,   zBGlm4vForConditionalGeneration._prepare_position_ids_for_generationc Cs|durA||tj|jjtj|jdkd}||tj|jjtj|jdkd}||tj|jjtj|jdkd}n||jjk}||jjk}||jjk}tj | | dd}|dk}||@}|j dd} |j dd} | | fS)aa Get the number of images and videos for each sample to calculate the separation length of the sample tensor. These parameters are not passed through the processor to avoid unpredictable impacts from interface modifications. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Returns: image_nums (`torch.LongTensor` of shape `(batch_size, num_images_sample)`) video_nums (`torch.LongTensor` of shape `(batch_size, num_videos_sample)`) Nr).rr!rr) rr.rr[image_start_token_idrrzrrrrr) r2rris_imageis_video_start is_video_end video_level inside_videostandalone_images image_counts video_countsr6r6r7_get_image_nums_and_video_numss>      z._expand_dict_for_generation_visual.._repeat_interleave_samples..rr)r.rrqr)rr repeat_timessamplesr7r6r?r7_repeat_interleave_samplesas zGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual.._repeat_interleave_samplesrcSg|] }tj|ddqSr!rr.rrr=r6r6r7rlr%z{Glm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual..)rrArcSrDrErFr=r6r6r7rxr%r<)rr9r.rr) dict_to_expandrr image_nums video_numsrCrrBr)r:rr*r2r6r7"_expand_dict_for_generation_visualZsF             zgGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visualcs~|D]:}|dkr||jdkr||jdd||<q|dkr<||dur._expand_dict_for_generationrrencoder_outputszMIf `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.)r<rr)r2r:r;rr*rJrLr6)r:rr*r2rKr7_expand_inputs_for_generationIs - z;Glm4vForConditionalGeneration._expand_inputs_for_generationr\) NNNNNNNNNNNr) NNNNNTNNNNF)r!FN)"rIrJrKr_tied_weights_keysrr+rrrr.rZrTrrrErrrrrMr rrrDr"r&r9r`dictrrrNrNr6r6r4r7r5s        ^ (' :r)rrrhrr{)rr)[rcollections.abcr dataclassesrtypingrrr.torch.nnr,torch.nn.functionalrrrrro activationsr cache_utilsr r generationr integrationsr masking_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrrutils.output_capturingr configuration_glm4vr"r#r$Moduler&rOrarnrrrrMrErrrrLrrr rr;rCrDrUr[rdrhr{rrrr__all__r6r6r6r7s             K  SM (H4kv p