o eiؾ@s$ddlZddlmZmZddlmZddlmZddlZddl m Z ddl m m Z ddlmZddlmZddlmZmZdd lmZdd lmZdd lmZmZdd lmZdd lm Z m!Z!m"Z"ddl#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3ddl4m5Z5ddl6m7Z7ddl8m9Z9m:Z:m;Z;mee.ddGdd d e,Z?Gd!d"d"e j@ZAGd#d$d$e j@ZBGd%d&d&e j@ZCGd'd(d(e j@ZDGd)d*d*e j@ZEGd+d,d,e j@ZFGd-d.d.e j@ZGGd/d0d0e j@ZHGd1d2d2e j@ZIGd3d4d4e j@ZJGd5d6d6e(ZKGd7d8d8e jLZMGd9d:d:e j@ZNGd;d<dd>e j@ZPd?d@ZQdAejRdBeSdCejRfdDdEZT F  dndGe j@dHejRdIejRdJejRdKejRdBdLeUdMeUdBdNeUdBdCeVejRejRffdOdPZWdodQejRdRejRdSejRdTeSfdUdVZXeeXGdWdXdXe j@ZYGdYdZdZeZZe.Gd[d\d\e(Z[Gd]d^d^e j@Z\e.d_dGd`dadae[Z]e.dbdGdcdddde[eZ^Gdedfdfe j@Z_e.dgdGdhdidie[Z`e.djdGdkdldle[eZagdmZbdS)pN)CallableSequence) dataclass)Optional)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernelized_func)create_causal_mask!create_sliding_window_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupletorch_compilable_check)maybe_autocastmerge_with_config_defaults)capture_outputs) AutoModel)Gemma3nAudioConfig Gemma3nConfigGemma3nTextConfigGemma3nVisionConfigc@s$eZdZUdZdZejdBed<dS)Gemma3nAudioEncoderModelOutputzz audio_mel_mask (`torch.FloatTensor`, *optional*): A torch.BoolTensor of shape `(batch_size, num_frames)` Naudio_mel_mask)__name__ __module__ __qualname____doc__r(torch BoolTensor__annotations__r0r0j/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/gemma3n/modeling_gemma3n.pyr'6s r'zL Base class for Gemma3n outputs, with hidden states and attentions.  custom_introc@s6eZdZUdZdZejdBed<dZejdBed<dS)Gemma3nModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. audio_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. audio_hidden_states of the model produced by the audio encoder and after projecting the last hidden state. Nimage_hidden_statesaudio_hidden_states) r)r*r+r,r5r- FloatTensorr/r6r0r0r0r1r4As r4zS Base class for Gemma3n causal language model (or autoregressive) outputs. c@seZdZUdZdZejdBed<dZejdBed<dZ e dBed<dZ e ejdBed<dZ e ejdBed<dZejdBed<dZejdBed <dS) Gemma3nCausalLMOutputWithPastaF loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. audio_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. audio_hidden_states of the model produced by the audio encoder and after projecting the last hidden state. Nlosslogitspast_key_values hidden_states attentionsr5r6)r)r*r+r,r9r-r7r/r:r;r r<tupler=r5r6r0r0r0r1r8[s r8csReZdZddededeffdd Zdd Zd ej d ej fd d Z ddZ Z S)Gemma3nRMSNormư>Tdimeps with_scalecsLt||_||_|jrtt||_dS|j dt ddddS)Nweight?F persistent) super__init__rBrCnn Parameterr-onesrDregister_buffertensor)selfrArBrC __class__r0r1rIs zGemma3nRMSNorm.__init__cCs$|t|djddd|jS)Nr T)keepdim)r-sqrtpowmeanrB)rOxr0r0r1_norms$zGemma3nRMSNorm._normrWreturncCs"|||j}||SN)rXfloatrDtype_as)rOrWoutputr0r0r1forwards zGemma3nRMSNorm.forwardcCst|jjd|jS)Nz, eps=)r>rDshaperBrOr0r0r1 extra_reprszGemma3nRMSNorm.extra_repr)r@T) r)r*r+intr[boolrIrXr-Tensorr^ra __classcell__r0r0rPr1r?~s  r?cseZdZdeffdd ZdejdejdejfddZd ejd e d e d e d e de de dejfddZ dejdejdejfddZ Z S)%Gemma3nAudioRelativePositionEmbeddingconfigcst||_|jj|_|jj|_|j|j|_td|jj d|_ |jj |_ t j|j|j|jdd|_d}d}|jd}tt|t|t|dd}|tt|| }|jd|dddd dS) Nrr"FbiasrE@r inv_timescalesrF)rHrIrgconf_num_attention_heads num_heads hidden_sizechannelshead_dimmaxconf_attention_context_left max_backwardconf_attention_context_right max_forwardrJLinearpos_projmathlogr[r-exparangerM unsqueeze)rOrg min_timescale max_timescalenum_timescaleslog_timescale_incrementrkrPr0r1rIs$     $ z.Gemma3nAudioRelativePositionEmbedding.__init__positiondtyperYcCsN|d}||jj|jtjd}tjt|t |gdd}| |S)NrRdevicerrA) r[r|rktorr-float32catsincostype)rOrr scaled_time timing_signalr0r0r1_get_timing_signal_1d_poss z?Gemma3nAudioRelativePositionEmbedding._get_timing_signal_1d_posterm_bd_before_shift batch_sizermnum_query_blocksquery_block_sizekey_context_sizemax_span_plus_1cCsx|d|}d|f} tj|| } | |||||df} | ddddddd||f} | |||||f} | S)aZPerforms the relative shift. Args: term_bd_before_shift: Tensor of shape [B, N, U, W, F_span]. batch_size (B), num_heads (N), num_query_blocks (U), query_block_size (W), key_context_size (C = W+L+R), max_span_plus_1 (F_span = L+R+1). Returns: Tensor of shape [B, N, U, W, C]. r"rN)rJ functionalpadreshape)rOrrrmrrrrpad_amount_last_dim padding_tupleterm_bd_paddedterm_bd_reshapedterm_bd_slicedterm_bd_shiftedr0r0r1_relative_shifts(  $  z5Gemma3nAudioRelativePositionEmbedding._relative_shiftquerieskeysc Cs"|j\}}}}}|j\}}} }}tj|j|j dd|jdd} | jd} |j| |jd} | | } | d| |j |j  d}|ddddd}|ddddd}t||}|ddddd}|ddd}| |||||}t||}| ||||| }||||||| | }||S) Nr"rRrrrrr )r_r-r{rsrurr|rrrwrrmrpsqueezepermutematmulr)rOrrrrrrmrp_r pos_indicesrsin_emb_timing_signalprojected_sin_embsin_emb queries_pkeys_p_tterm_ac q_permuted s_permuted q_reshapedterm_bd_unshifed_matmulterm_bd_unshifedrr0r0r1r^sJ       z-Gemma3nAudioRelativePositionEmbedding.forward) r)r*r+r#rIr-rdrrrbrr^rer0r0rPr1rfs* $=rfcseZdZdeffdd ZddZdejdeded ejfd d Z d ejd ejfd dZ d ejd ejfddZ d ejdej d ejfddZ ZS)Gemma3nAudioAttentionrgcsbt||_|jj|_|jj|_|j|j|_|jj|_|jj |_ t d|jj d|_ |jj|_|j|j |j |_t||_tt|jf|_tj|j|j|jdd|_tj|j|j|jdd|_tj|j|j|jdd|_|jd}dtjjtd}|jd|| !dd |"}|jd |dd |jd t|j#dd dS) Nrr"FrhrEq_scalerFlocal_causal_valid_masksoftcap)$rHrIrgrlrmrnrpconf_attention_chunk_size chunk_sizertmax_future_horizonrqrrmax_past_horizonconf_attention_logit_capattention_logits_soft_cap context_sizerfrelative_position_embeddingrJrKr-zeros per_dim_scalervq_projk_projv_projrsoftplusrNrMclonedetachcreate_local_causal_valid_maskr[)rOrgr r_softplus_0rrPr0r1rIDs2         zGemma3nAudioAttention.__init__cCsvtjtj|j|jftjdddj}tjtj|j|jftjd|j|jd}tj|j|jftjd}|||}|S)Nrr)diagonal) r-trilrLrrrcTrr)rOlower_causal_maskupper_causal_maskrr0r0r1rfs  z4Gemma3nAudioAttention.create_local_causal_valid_maskrWpad_left pad_rightrYc CsL|j^}}}|||g|R}|||g|R}tj|||gdd}|S)Nr"r)r_ new_zerosr-r) rOrWrrbatchr tail_shapeleftrightr0r0r1 _pad_dim1ss zGemma3nAudioAttention._pad_dim1r<cCsx|j}|dd\}}||jd|j}||j|}dkr'||d|}|||jf|dd}||}|S)aETurns a sequence to non overlapping blocks. Args: hidden_states: a tensor of [batch, time, ...]. Returns: A tensor of [batch, num_blocks, block_size, ...], with necessary paddings, where output[:, i, ...] are x[:, i*block_size:(i+1)*block_size, ...]. Nr r"r)r_rrr contiguous)rOr<r_bt num_blocks padding_len permute_dimsr0r0r1_convert_to_blockzs z'Gemma3nAudioAttention._convert_to_blockcCsl|j}|j|jd}||||}|j}|j}|jd||d}|jdkr2|jdkr2tj|ddd}| S)aExtracts temporal context for every block. Args: hidden_states: a tensor of [batch, time, ...]. Returns: A tensor of [batch, num_blocks, context_size, ...], with necessary paddings, where context_size = block_size + left_context + right_context, and output[:, i, ...] are x[:, start-left_context:end+right_context, ...], start = i * block_size, end = (i + 1) * block_size. r") dimensionsizestepr rrR)source destination) rrrrrunfoldndimr-movedimr)rOr<rr frame_len frame_step x_unfoldedr0r0r1_extract_block_contextsz,Gemma3nAudioAttention._extract_block_contextmaskc" Csg|jdd|j|jR}|||}|||}|||}tj j |j }ddd|jf}| |} ||j| }|jdd\} } ||} ||} ||}| jd}|}||}|jdkr|jd|jd|jkr|| ||j}|j| ||jfkrtd|jd| d|d|jd |dd }|jd d d }t|||j}|| | }|j|j}||}t|}||}t||t|jj}tj j j |dtj!d j|jd }|j\}}}}}|jd}|"d ddddd||}|"d ddddd||}t#||} | |||||"d dddd}!|!| ||j$|j|jf}!|!ddd| f}!|!S)NrRr"r rrz%Shape of extracted_valid_mask_blocks z is not (z, z) after potential reshape.rrArr)%r_rmrprrrrrr-rJrrrviewrrrrr ValueErrorr|r logical_andrrrrtanhwherefinforminsoftmaxrrbmmr)"rOr<r qkv_shape query_states key_states value_statesper_dim_scale_spbroadcast_shapeper_dim_scale_sp_broadcastrq_time query_blocks key_blocks value_blocksroriginal_valid_maskextracted_valid_mask_blockscondition_from_input_validitycondition_from_causalityfinal_condition_for_wherer: softcap_val probabilitiesb_dimn_dimu_dimw_dimc_dimh_dimprob_bunv_bun result_bmmcontext_vectorsr0r0r1r^s                zGemma3nAudioAttention.forward)r)r*r+r#rIrr-rdrbrrrr.r^rer0r0rPr1rCs" $0rcsLeZdZdZ d dedeedeffdd Zdej d ej fd d Z Z S) Gemma3nAudioCumulativeGroupNormaApplies Group Normalization cumulatively over the time dimension. This layer normalizes the input by calculating the mean and variance cumulatively over the time dimension (dim 1). The statistics are computed over all feature dimensions (specified by `feature_dims` and `num_channels`) for elements marked as valid by the optional `mask`. If a `mask` is provided (True for valid, False for invalid/padded), invalid time steps do not contribute to the statistics calculation, and their corresponding output values are zeroed out. Scale and bias, if enabled, are applied per-channel (last dimension). This behavior is similar to JAX's `GroupNormalization` with `num_groups=1` and `cumulative=True`. MbP? num_channels feature_dimsrBcsTt||_t||_||_tt ||_ tt ddt |jd|_ dS)Nr r")rHrIrr>rrBrJrKr-rLrDrangelenreduction_axes)rOrrrBrPr0r1rI8s  "z(Gemma3nAudioCumulativeGroupNorm.__init__r<rYcCsL|j|jf}|jdd|krtd|jddd||j}tj}||}tj||d}tj ||j dd}tj |dd }tj ||j dd} tj | dd } tj | d d } || } ||  d} tj | |j dd}tj |dd }|| }|| t||j}|j|}dg|d|jg}|||}||}||S) zApplies cumulative group norm, optionally using a mask. Args: hidden_states: Input tensor, shape [B, T, *feature_dims, C]. Returns: Normalized tensor with the same shape as x. r NzInput tensor shape suffix z> does not match expected suffix (feature_dims + num_channels) rTrArSr"rrE)r)rrr_rrr-rr ones_likesumrcumsumclamprUrsqrtrBrDrAr)rOr<expected_input_suffix input_dtype calc_dtypex_calc mask_calcsum_values_at_tcum_sum_valueselements_in_group_at_tcum_count_elementssafe_cum_count_elementscum_meansquared_diff_from_meansum_sq_diff_at_tcum_sum_sq_diff cum_variance normalized_xscalescale_view_shape final_outputr0r0r1r^Js6    z'Gemma3nAudioCumulativeGroupNorm.forward)r) r)r*r+r,rbrr[rIr-rdr^rer0r0rPr1r'src sXeZdZdZ d dedededeeeeefffdd Zd ej d ej fd d Z Z S)Gemma3nAudioSSCPConvBlockzA single convolution block for the SubSampleConvProjection. This block consists of a 2D convolution, followed by CumulativeGroupNorm, and a ReLU activation. It handles manual padding for the convolution. rrrrrgidxinput_freq_dimmanual_paddingc st||_||_|dkrdn|jj|d}|jj|}|jj|\}}|jj|\} } tj||||f| | fddd|_ ||jd|jd} | || d} t || f|jj d|_ t |_dS)Nrr")rrF) in_channels out_channels kernel_sizestridepaddingri)rrrB)rHrIrgr8sscp_conv_channel_sizesscp_conv_kernel_sizesscp_conv_stride_sizerJConv2dconvrsscp_conv_group_norm_epsnormReLU activation) rOrgr6r7r8r9r:kernel_hkernel_wstride_hstride_w f_in_padded f_out_convrPr0r1rIs2  z"Gemma3nAudioSSCPConvBlock.__init__audio_encodingsrYcCsftj||jddd|jjj}||}|dddd}| |}|dddd}| |S)Nconstantr)modevaluerr rr") Frr8rrBrDrrrrDrF)rOrMaudio_encodings_paddedaudio_encodings_conv x_for_normx_normedaudio_encodings_normedr0r0r1r^s   z!Gemma3nAudioSSCPConvBlock.forward)r5) r)r*r+r,r#rbr>rIr-rdr^rer0r0rPr1r4s +r4c8eZdZdeffdd ZdejdejfddZZS)#Gemma3nAudioSubSampleConvProjectionrgcst||_|j}g}g}tdD]:}|j|\}}|j|\}} d} |d} d} d} | | | | f}|||| | }||| d}|||}qtd|j||dd|_ td|d||dd|_ |j d}|d}|||_ t j|j |jjdd|_dS)Nr rr")r6r7rgr8rRFrh)rHrIrginput_feat_sizerr?r@appendr4conv_0conv_1r>input_proj_in_featuresrJrvrninput_proj_linear)rOrgcurrent_f_for_block_inputcalculated_block_paddingcalculated_f_out_dimsirGrHrIrJ pad_t_top pad_t_bottom pad_f_left pad_f_rightmanual_padding_tuplerKf_out_after_conv final_c_out final_f_outrPr0r1rIsL       z,Gemma3nAudioSubSampleConvProjection.__init__rMrYc Cs`|d}||}||}|j\}}}}|dddd}|||||} || } | S)Nr"rr r)r|r[r\r_rrrr^) rOrMaudio_encodings_reshapedrWrc_outt_outf_out x_permutedoutput_flattenedr]r0r0r1r^s    z+Gemma3nAudioSubSampleConvProjection.forward r)r*r+r#rIr-rdr^rer0r0rPr1rXs9rXc>eZdZdeffdd ZdejdejdejfddZZ S) Gemma3nAudioConformerAttentionrgcsvt||_|jj|_|jdt|jjddt |jj|_ t ||_ t j|j|jjdd|_t |jj|_dS)Ngradient_clippingFrFrh)rHrIrgrnpost_in_featuresrMr-rNrtr? pre_attn_normrattnrJrvpost post_normrOrgrPr0r1rI#s   z'Gemma3nAudioConformerAttention.__init__rMr(rYc Csz|}t||j |j}||}|||}|j\}}}} ||||| } || }t||j |j}|||SrZ) r-rrtrvrwr_rrxry) rOrMr(audio_encodings_input_to_attnaudio_encodings_normaudio_encodings_attn_outrrrmrprkr0r0r1r^-s   z&Gemma3nAudioConformerAttention.forward r)r*r+r#rIr-rdr.r^rer0r0rPr1rs"s$ rscrW) Gemma3nAudioConformerFeedForwardrgcst||_|jdt|jjddt|jj|_ t j |jj|jjddd|_ t j |jjd|jjdd|_ t|jj|_|jj|_dS)NrtFrFrrh)rHrIrgrMr-rNrtr?rnpre_layer_normrJrv ffw_layer_1 ffw_layer_2post_layer_normconf_residual_weightpost_layer_scalerzrPr0r1rI?s z)Gemma3nAudioConformerFeedForward.__init__rMrYcCsn|}t||j |j}||}||}tj|}||}t||j |j}| |}|||j SrZ) r-rrtrrrJrsilurrr)rOrMresidualr0r0r1r^Ks     z(Gemma3nAudioConformerFeedForward.forwardrqr0r0rPr1r>s rcrW) Gemma3nAudioConformerLightConv1drgc st||_t|jj|jjd|_tj|jj|jjddd|_ tj |jj|jj|jj dd|jjdd|_ |j dt|jjdd t|jj|jjd|_tj|jj|jjdd|_|jj d|_dS) NrBr Frhr"r)r9r:r;r<r=groupsrirtrF)rHrIrgr?rn rms_norm_epsrrJrv linear_startConv1dconf_conv_kernel_sizedepthwise_conv1drMr-rNrt conv_norm linear_endcausal_paddingrzrPr0r1rIXs"  z)Gemma3nAudioConformerLightConv1d.__init__rMrYcCs|}||}||}tjjj|dd}|ddd}t||j df}| |}|ddd}t ||j |j }| |}tj|}||}||}|S)NrRrrr r")rrr-rJrglurrQrrrrrtrrr)rOrMaudio_encodings_residualaudio_encodings_permutedaudio_encodings_permuted_paddedr]r0r0r1r^ms      z(Gemma3nAudioConformerLightConv1d.forwardrqr0r0rPr1rWsrcrr) Gemma3nAudioConformerBlockrgcslt||_t|j|_t|j|_t|j|_t|j|_ |j dt |jj ddt|jj|_dS)NrtFrF)rHrIrgrffw_layer_startrs attentionrlconv1d ffw_layer_endrMr-rNrtr?rnrDrzrPr0r1rIs     z#Gemma3nAudioConformerBlock.__init__rMr(rYcCsh||}|||}|}||d|j}||}||}t||j |j }| |}|S)NrR) rrr|rrrrr-rrtrD)rOrMr(validity_mask_for_lconvaudio_encodings_for_lconv_inputr]r0r0r1r^s      z"Gemma3nAudioConformerBlock.forwardr~r0r0rPr1rs$ rc sfeZdZUdZeed<dZdZdeffdd Ze e de j de j deed eeBfd d ZZS) Gemma3nAudioEncoderzx An audio encoder based on the [Universal Speech Model](https://huggingface.co/papers/2303.01037) architecture. rg audio_melaudiocsHt|_t|_tfddtjD|_ | dS)Ncsg|]}tqSr0)r.0rrgr0r1 sz0Gemma3nAudioEncoder.__init__..) rHrIrgrXsubsample_conv_projectionrJ ModuleListrconf_num_hidden_layers conformer post_initrzrPrr1rIs   zGemma3nAudioEncoder.__init__r(kwargsrYc Ks^||}|jd}d}tt|jjD] }||jj|d9}qtj||jd|}tj ||jddd}|j dkrN|j dkrN| d |jdd}n |j |j krn|jddkrn|jddkrn||jdkrn| d}t |d|} |jD]} | || }qx|jjdkr|dddd|jjf}| dddd|jjf} || dd}t|| dS) aEncodes a batch of MELs. Args: audio_mel: a torch.Tensor of shape [batch, num_frames, num_channels, mel_bins]. Returns: audio_encodings: a torch.Tensor of shape `[batch_size, self.config.audio_soft_tokens_per_image, self.config.audio_config.hidden_size]` audio_mel_mask: a torch.BoolTensor of shape [batch, num_frames]. r"rr)rqrRNr)last_hidden_stater()rr_rrrgr@r-r{rrrr|expandgatherrconf_reduction_factor masked_fillr') rOrr(rrMt_subtime_stride_productstride_pair_idxindices current_maskblockr0r0r1r^s2       zGemma3nAudioEncoder.forward)r)r*r+r,r#r/main_input_nameinput_modalitiesrIrrr-rdr.rrr>r'r^rer0r0rPr1rs"  rc sHeZdZdZd dedededeffdd Zd ejffd d Z Z S) Gemma3nTextScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. rEnum_embeddings embedding_dim padding_idx embed_scalecs0t|||||_|jdt|dddS)NrFrF)rHrIscalar_embed_scalerMr-rN)rOrrrrrPr0r1rIsz'Gemma3nTextScaledWordEmbedding.__init__ input_idscst||j|jjSrZ)rHr^rrrDrrOrrPr0r1r^sz&Gemma3nTextScaledWordEmbedding.forward)rE) r)r*r+r,rbr[rIr-rdr^rer0r0rPr1rs rcs<eZdZdZdeffdd ZdejdejfddZZ S) Gemma3nTextLaurelBlockz Learned Augmented Residual Layerrgcs^t||_tj|jj|jjdd|_tj|jj|jjdd|_t |jj|jj d|_ dS)NFrhr) rHrIrgrJrvrn laurel_rank linear_left linear_rightr?rpost_laurel_normrzrPr0r1rIs zGemma3nTextLaurelBlock.__init__r<rYcCs&||}||}||}||SrZ)rrr)rOr<laurel_hidden_statesnormed_laurel_hidden_statesr0r0r1r^s   zGemma3nTextLaurelBlock.forward) r)r*r+r,r%rIr-rdr^rer0r0rPr1rsrcsTeZdZd dedeffdd Zdejdejfdd Zd ejdejfd d Z Z S)Gemma3nTextMLPrrg layer_idxcst||_|j|_|j||_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ |j ||_dSNFrh)rHrIrgrnintermediate_sizerJrv gate_projup_proj down_projrhidden_activationact_fnactivation_sparsity_patternactivation_sparsityrOrgrrPr0r1rIs   zGemma3nTextMLP.__init__r<rYcCsD||}|jdkr||}||}||}|||}|S)Nr)rr_gaussian_topkrrr)rOr<r activationsrrr0r0r1r^s     zGemma3nTextMLP.forwardinputscCsztj|jtj|jd}tjjdd}||}| |j }tj |ddd}tj |dddd}|||}t j||S) Nrrrr"rRTrF)rArSunbiased)r-rNrrr distributionsnormalNormalicdfrrrVstdrJrrelu)rOrtarget_sparsity_tensor normal_diststd_multiplier inputs_mean inputs_stdcutoff_xr0r0r1r#s   zGemma3nTextMLP._gaussian_topk)r) r)r*r+r%rbrIr-rdr^rrer0r0rPr1rs  rcseZdZdZdeffdd ZdejdejfddZd ejdejfd d Z d ejd ejdejfddZ dejdejfddZ dejdejfddZ Z S)Gemma3nTextAltUpaAlternating Updates (AltUp) The AltUp module wraps transformer layers. The `predict` step modifies the input to the transformer layer, and the `correct` step propagates the output of the transformer layer to the sparsely updated dimensions. See more in the research paper: https://proceedings.neurips.cc/paper_files/paper/2023/file/f2059277ac6ce66e7e5543001afa8bb5-Paper-Conference.pdf rgcst||_tt|jj|_tj |jj |jj dd|_ tj |jj |jj ddd|_ tj |jj|jj dd|_ t|jj|jjd|_|jdt|jjddddS)NFrhr rrouter_input_scalerF)rHrIrgrJrKr-rrncorrect_output_scalervaltup_num_inputscorrection_coefsprediction_coefsmodality_routerr?r router_normrMrNrzrPr0r1rI@s "zGemma3nTextAltUp.__init__rWrYcCs.|||j}||}t||SrZ)rrrr-rr[r\)rOrW router_inputsroutedr0r0r1compute_router_modalitiesJs z*Gemma3nTextAltUp.compute_router_modalitiesr<cCs|||jj}|jr |jjdur |jjj|jj |jj||j g|j dd|jj |jj R dddd}t | dddd|}| dddd}||7}||S)aPredicts the output of a layer using a trainable map. Args: hidden_states: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` derived by stacking the input embeddings and preprocessing the last `num_altup_inputs - 1` matrices. Returns: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` containing the predictions. NrRrr"rr )rrgaltup_active_idxtrainingaltup_coef_cliprrDdataclamp_rr_rrr-rrr\)rOr< modalities all_coefs predictionsr0r0r1predictOs$   zGemma3nTextAltUp.predictr activatedcCs||}|||jj}||jjddd}|jr:|jjdur:|jj |jj |jj}t j j j ||ddd}n||d}|dddd}t ||}||7}||S)a_Corrects the predictions relative to the Args: predictions: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` derived by stacking the input embeddings and preprocessing the last `num_altup_inputs - 1` matrices. activated: A 3D tensor of shape `[batch_size, num_tokens, hidden_size]` containing the activated inputs. Returns: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` correcting the original predictions relative to the activated input embeddings. r"NrhrEr rrR)rrgrrepeatrrrrrDrr-rJrlinearrr|mulrr\)rOrrr innovationrDr correctedr0r0r1correctks  zGemma3nTextAltUp.correctrcCs||j|j|S)a  This is only defined as the `forward` so that accelerate hooks can move correctly `correct_output_scale` (which is a nn.Parameter, not a Module) between devices when offloading. It is otherwise only used in `scale_corrected_output` )r\rrOrr0r0r1r^szGemma3nTextAltUp.forwardcCs ||S)zMScales the provided 3D tensor of shape [batch_size, num_tokens, hidden_size].)r^rr0r0r1scale_corrected_outputs z'Gemma3nTextAltUp.scale_corrected_output)r)r*r+r,r%rIr-rdrrrr^rrer0r0rPr1r4s  rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..NrRr r)r_r-r)rWx1x2r0r0r1 rotate_halfsrr<n_reprYcCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r"N)r_rr)r<rrnum_key_value_headsslenrpr0r0r1 repeat_kvs 0rrmodulequerykeyrPattention_maskdropoutscalingrc Ks|dur |jd}t||j} t||j} t|| dd|} |dur2| |} t| } | |} |dur:| |} tjj | dtj d |j } tjj | ||jd} t| | } | dd} | | fS)Nrr rrRr)prr")rprnum_key_value_groupsr-r transposerrJrrrrrr rr) r r r rPr r rrrrr attn_weights attn_outputr0r0r1eager_attention_forwards      rrWrr unsqueeze_dimcCs(||}||}||t||S)a\Applies Rotary Position Embedding to the query and key tensors. Args: x (`torch.Tensor`): The tensor to embed. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )r|r)rWrrrr0r0r1apply_rotary_pos_embs  rcseZdZdZdedeffdd Z    ddejdejd ejdBd e dBd ej dBd e e d e ejejdBe ejdBffddZZS)Gemma3nTextAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrgrcstt|dr|j|nd|_||_||_t|d|j|j |_ |j |j |_ d|_ |jj|_d|_tj|j|j |j |jd|_tj|j|j |j |jd|_tj|j|j |j |jd|_tj|j |j |j|jd|_|jdkry|jnd|_|jdk|_t|j |jd|_t|j |jd|_t|j |jdd |_|jj|jj}||kod kn|_ |jd|}|j rt!|d |ddd "|j||_#d|_$dSd|_#|t!|d |ddd "|j|k|_$dS) N layer_typesrprETrhsliding_attention)rArBF)rArBrCrr"rR)%rHrIhasattrr layer_typergrgetattrrnnum_attention_headsrprrrattention_dropout is_causalrJrvattention_biasrrro_projsliding_window is_slidingr?rq_normk_normv_normnum_hidden_layersnum_kv_shared_layersis_kv_shared_layerrindexkv_shared_layer_indexstore_full_length_kv)rOrgrfirst_kv_shared_layer_idx prev_layersrPr0r1rIsH   ( zGemma3nTextAttention.__init__Nr<position_embeddingsr r;cache_positionrrYcKs|jdd}g|d|jjR}|\} } |||} || } t| | | dd} | dd} |jrM|durM|j |j \} } | | j } | | j } n.| ||} || } t| | | dd} | dd} |||} || } | dd} |dur| | ||jd}|js|| | |j|\} } |jrt|dsi|_ | | f|j |j<t|jjt}||| | | |f|jr|jnd|j|jd|\}}|jg|dR}||}||fS) NrRr )rr")rrr0r" shared_layersr)r rr") r_rgrprrr$rrr)r1r+rrrr%rr&r"updaterr,rr get_interface_attn_implementationrrrrrrr!)rOr<r/r r;r0r input_shape hidden_shaperrrrr cache_kwargsattention_interfacerrr0r0r1r^sf            zGemma3nTextAttention.forwardNNNN)r)r*r+r,r%rbrIr-rdr LongTensorrrr>r^rer0r0rPr1rs,/rcseZdZdedeffdd Z      ddejdejdejd ejdBd ejdBd e dBd ejdBd e e de eje ej ej fdBffddZZS)Gemma3nTextDecoderLayerrgrcst||_|j|_||_|j||_t|||_t ||d|_ t |j|j d|_ t |j|j d|_t |j|j d|_t |j|j d|_|j|_t|j|_t||_t||_tj|j|jdd|_tj|j|jdd|_t |j|j d|_dS)N)rrFrh)rHrIrgrnrrattention_typer self_attnrmlpr?rinput_layernormpost_attention_layernormpre_feedforward_layernormpost_feedforward_layernormhidden_size_per_layer_inputrrrraltuprlaurelrJrvper_layer_input_gateper_layer_projectionpost_per_layer_input_normrrPr0r1rI[s$      z Gemma3nTextDecoderLayer.__init__Nr<r/per_layer_inputr  position_idsr;r0rrYc Ks|j|} | |jj} || } || } |jd| |||||d|\} }|| } | | }|| t d}| |}| |}| |}||}|j | |}||jj}|jjre|j|}||}||}t||}||}||}|dd|7<|S)N)r<r rJr/r;r0r r"r0)rDrrgrr?rEr=r@rxrTrAr>rBrraltup_correct_scalerrFrr-multiplyrGrH)rOr<r/rIr rJr;r0rractive_predictionactive_prediction_normed laurel_outputrwr attn_gated attn_laurel attn_normattn_ffw attn_ffw_normattn_ffw_laurel_gatedcorrected_predictionsfirst_predictionr0r0r1r^qs@             zGemma3nTextDecoderLayer.forward)NNNNNN)r)r*r+r%rbrIr-rdr:r rrr>r7r^rer0r0rPr1r;Zs6  r;csdeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZdZefdd ZZS) Gemma3nPreTrainedModelrgmodelTr;r;)r<r=)imagetextrc s0t|t|trt|jnt|trIt|j |j d}dt j j t d}t|j||t|j|jt|j|nt|trWt|j|jnt|trnt|jt|j|jjdnt|trd\}}|j d}t!"t#|t#|t$|dd}|t %t &|| }t|j'|#(d(dnZt|t)rt|j*|jdt|j+dt!,d n>t|t-r|j.D]4} |j/} |j0| d krt1|j0| } | |j| d \} } tt2|| d | tt2|| d | qt3|drt|j4|jj4dSdS)NrrErr)rErjr r"r@defaultr _inv_freq_original_inv_freqrt)5rH _init_weights isinstancerinitones_rDrzeros_rrpr-rJrrrNcopy_r constant_rrrrrrrrrrrgrnrfrorxryr[rqrzr{rkr|Gemma3nTextModelper_layer_projection_scaleper_layer_input_scalerTGemma3nRotaryEmbeddingrcompute_default_rope_parameters rope_typerrrrt) rOr rrr}r~rrrkr rope_init_fn curr_inv_freqrrPr0r1rasJ               z$Gemma3nPreTrainedModel._init_weights)r)r*r+r$r/base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendr;r_can_record_outputsrr-no_gradrarer0r0rPr1rXs" rXcseZdZUejed<ddeffdd Ze    ddedBde dde dBd e dBd e d e ff d d ZeedddZZS)rkinv_freqNrgcst|j|_|j|_||_tt|j|_i|_ |jD]P}|jj |}|dur+q|d|j |<|j }|j |dkrCt |j |}||j||d\}}|j |d|dd|j |d|ddt||d|qdS) Nrmr]r^r_FrFr`_attention_scaling)rHrImax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrglistsetrrmrope_parametersrlrrMrsetattr)rOrgrr rope_paramsrnrocurr_attention_scalingrPr0r1rIs&   zGemma3nRotaryEmbedding.__init__rz torch.deviceseq_lenrrYz torch.TensorcCs^|j|d}t|ddp|j|j}d}d|tjd|dtjdj|tjd|}||fS) a| Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. layer_type (`str`, *optional*): The current layer type if the model has different RoPE parameters per type. Should not be used unless `config.layer_types is not None` Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetarpNrErr rr) rrrnrr-r{int64rr[)rgrrrbaserAattention_factorr{r0r0r1rls&z6Gemma3nRotaryEmbedding.compute_default_rope_parametersc Cst||d}t||d}|ddddf|jddd|j}|dddddf}t|jjtrE|jjdkrE|jjnd}t |dd )|| dd } t j | | fdd } | |} | |} Wdn1sywY| j|jd | j|jd fS) Nr_r|rrRr"mpscpuF) device_typeenabledr rr)rr[rr_rrrbrstrrrr-rrrr) rOrWrJrr{attention_scalinginv_freq_expandedposition_ids_expandedrfreqsembrrr0r0r1r^!s.& zGemma3nRotaryEmbedding.forwardNNr9rZ)r)r*r+r-rdr/r%rI staticmethodrrbrr>r[rlrzrr^rer0r0rPr1rks,   #rkzBThe base Gemma 3n language model without a language modeling head.cseZdZUeed<dZdeffdd Zeedde        dde j dBd e j dBd e j dBd e j dBd e dBd e jdBdedBde j dBdeedefddZde j de j fddZ dd e j d e j dBde j fddZZS)rhrg)r[csltj_j_tjjjjjdd_t fddt j D_ tjjd_t_d_j_j_tjj jjjdd_t jjj jdd_tjjd_t fddt d jjD_t fd dt d jjD_jd tjd dd jdt tddd !dS)N?)rcsg|]}t|qSr0)r;)rrrr0r1rCsz-Gemma3nTextModel.__init__..rFrhc g|] }tjjjddqSFrhrJrvrnrr`r0r1r] r"crrrrr`r0r1rarrirrFrjr\)"rHrI pad_token_idr vocab_sizerrnrg embed_tokensrJrrr'layersr?rrDrk rotary_embgradient_checkpointingrCvocab_size_per_layer_inputembed_tokens_per_layerrvper_layer_model_projectionper_layer_projection_normraltup_projectionsaltup_unembed_projectionsrMr-rNr rrzrP)rgrOr1rI9sF     zGemma3nTextModel.__init__F)tie_last_hidden_statesNrper_layer_inputsr rJr; inputs_embeds use_cacher0rrYc  Ks|du|duAr td|dur||}||}|||}|r,|dur,t|jd}|durG|dur8|nd} tj|j d|j d| }|durP| d}t |} t sp|j|||||d} tdi| tdi| d} |} tj| d d d d d }td}| g}td|jjD]6}|j|d| }|j| j|j d}tj|d d d d }tt|||j }|||}||qtj|dd}i}|jjD] }||||||<q|jd|jjD](}| |j}|dddd|j ddf}||||j|f||||d| }qtj|dd d d d d }|dg}td|jjD]9}|j!|d||}|j| j|j d}tj|d d d d }tt|||j }|||}||q)t|}tj|dd}|"|}t#||dS)z per_layer_inputs (torch.Tensor, *optional*, defaults to None): Pre-computed per-layer embeddings. If None, they are derived from input_ids if provided. N:You must specify exactly one of input_ids or inputs_embedsrrr"r)rgrr r0r;rJ)full_attentionrr rRTrrgh㈵>rr)r rJr;r0)rr;r0)$rrget_per_layer_inputsproject_per_layer_inputsr rgget_seq_lengthr-r{r_rr|rbdictr rrVrNrrrrrrTmaximumrZstackrrrr'r<rrrDr)rOrrr rJr;rrr0rpast_seen_tokenscausal_mask_mapping mask_kwargshidden_states_0target_magnitudeepsilon_tensortemp_hidden_statesrb altup_projcurrent_hidden_state new_magnituder<r/r decoder_layer causal_maskrIaltup_unemb_projr0r0r1r^js                zGemma3nTextModel.forwardcCs&||jg|j|jj|jRSrZ)rrr_rgr'rCrr0r0r1rs z%Gemma3nTextModel.get_per_layer_inputscCs||}||jj|j|jd9}|jg|jdd|jj|j R}| |}|dur0|S|j|jkrC|dd|jjddf}|||j j|j|jdS)NrrR.) rrirrrrr_rgr'rCrrj)rOrrrGr0r0r1rs&     z)Gemma3nTextModel.project_per_layer_inputs)NNNNNNNNrZ)r)r*r+r%r/rrIrrrr-r:rdr r7rcrrrr^rrrer0r0rPr1rh4sZ 1   n rhz?The base Gemma 3n language model with a language modeling head.cseZdZUddiZddiZddgdgfiZeed<dd iZdeffd d Z e e dde j d Bde jd Bde j d Bded Bde jd Bde j d Bded Bde j d Bdee jBdeedefddZZS)Gemma3nForCausalLMlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputr<r:rgzmodel.language_modelrYcs@t|t||_|j|_tj|j|jdd|_| dSr) rHrIrhrYrrJrvrnrrrzrPr0r1rIs   zGemma3nForCausalLM.__init__Nrrr rJr;rlabelsrr0logits_to_keeprrYc  Ks|jd|||||||d| } | j} t| trt| dn| } || dd| ddf}|jjdurE||jj}t |}||jj}d}|durW|j |||j fi| }t ||| j | j| jdS)a Example: ```python >>> from transformers import AutoTokenizer, Gemma3nForCausalLM >>> model = Gemma3nForCausalLM.from_pretrained("google/gemma-2-9b") >>> tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-9b") >>> prompt = "What is your favorite condiment?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "What is your favorite condiment?" ```)rr rJr;rrr0N)r9r:r;r<r=r0)rYrrbrbslicerrgfinal_logit_softcappingr-r loss_functionrrr;r<r=)rOrr rJr;rrrr0rroutputsr< slice_indicesr:r9r0r0r1r^s8!     zGemma3nForCausalLM.forward) NNNNNNNNr)r)r*r+_tied_weights_keys_tp_plan_pp_planr%r/_checkpoint_conversion_mappingrIrrr-r:rdr r7rcrbrrrr^rer0r0rPr1rsR      rcsXeZdZdZdeeBdeffdd Z  d dej dBdej dBd ej fd d Z Z S) Gemma3nMultimodalEmbedderzQEmbeds token ids or soft tokens for multimodal content into language model space.multimodal_config text_configcst|j|_|j|_|j|_|j|_|j|_t |j|j|_ t |j|jd|_ t |j|jd|_t j|j|jdd|_t |j|jdd|_dS)NrFrh)rBrC)rHrIrnmultimodal_hidden_sizerrB vocab_offsetrtext_hidden_sizerJ Embedding embeddingr?hard_embedding_normsoft_embedding_normrvembedding_projectionembedding_post_projection_norm)rOrrrPr0r1rITs z"Gemma3nMultimodalEmbedder.__init__NrrrYcCsZ|du|duAr td|dur||}n |||j}||}||}||S)aEmbeds token ids or soft tokens for multimodal content into language model space. Args: input_ids: A torch.LongTensor containing the token ids to embed. Values should be in the range `[vocab_offset, vocab_offset + vocab_size)`. inputs_embeds: A torch.Tensor containing the soft tokens to embed. Returns: A torch.Tensor of embeddings with shape `[batch_size, seq_len, self.config.text_config.hidden_size]`. Nr)rrrrrrr)rOrremb_normhard_emb emb_norm_projr0r0r1r^gs    z!Gemma3nMultimodalEmbedder.forwardr) r)r*r+r,r#r&r%rIr-r:rdr^rer0r0rPr1rQs rz The base Gemma 3n model comprising a vision backbone, an audio backbone, and a language model without a language modeling head. c"seZdZiZdZdeffdd ZddZddZe e d d d e j d e ed eeBfddZ    d(de jdBde j dBde j dBde j dBfddZe              d)de jdBd e j dBde j dBde jdBde jdBde jdBdedBde jdBde jdBde j dBde jdBdedBd edBd!edBd"e ed ef d#d$Ze e d%d de jde jd e ed eeBfd&d'ZZS)* Gemma3nModelFrgcs~t|tj|jd|_|jj|_tj|jd}||_|jj |_ t|j |_ t |j|j|_ t |j |j|_|dS)Nr)rHrIr! from_config vision_config vision_towerrrlanguage_modelr audio_config audio_towerr embed_vision embed_audior)rOrgrrPr0r1rIs    zGemma3nModel.__init__cC |jSrZ)rget_input_embeddingsr`r0r0r1r z!Gemma3nModel.get_input_embeddingscC|j|dSrZ)rset_input_embeddingsrOrPr0r0r1rz!Gemma3nModel.set_input_embeddingszOProjects the last hidden state from the vision model into language model space.r2 pixel_valuesrrYcKsj|jd |ddd|}|j}||jd|jjj|jjddd}||jjjd9}|j |d|_ |S) NFT)r do_pooling return_dictrr r"rrr0) rrrr_rgrrnvision_soft_tokens_per_imagerr pooler_output)rOrrvision_outputsrr0r0r1get_image_featuress zGemma3nModel.get_image_featuresNrrimage_featuresaudio_featuresc Cs>|dur1||tj|jjtj|jdk}|d}||tj|jjtj|jdkd}n ||jjk}||jjk}| }| d | |j}|durlt |||kd|d|jd|jd| }| d | |j}|durt |||kd|d|jd|jd||fS) z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. NrrRz6Image features and image tokens do not match, tokens: z , features: rr"z6Audio features and audio tokens do not match, tokens: )rr-rNrgimage_token_idlongrallaudio_token_idrr| expand_asrrnumelr_) rOrrrrspecial_image_maskspecial_audio_maskn_image_tokensn_audio_tokensr0r0r1get_placeholder_masks:    z!Gemma3nModel.get_placeholder_maskinput_featuresr input_features_maskrJr;token_type_idsr0rroutput_attentionsoutput_hidden_states lm_kwargsc*Ks|du| duAr td| dur| n|jj} |dur|n|jj}|dur||} t|dk||jk}t||t |}|j |}t||j j k||jj k}|j j |j jd}t|||| j}|j |d}|| j| j}|d| }t||| } ||jj k}|jj |jjd}t|||| j}|j|d}|| j| j}|d| }t||| } nd}|dur|j|ddj}|| j| j}|j|| |d \}}| ||} |durP|durP|j||dd} | j} | j}tj|jdggtj| jd }!|j|!d}"t|d|"| } | j\}#}$}%|jj|$}&|"|#|&|%}'tj | |'fdd } | | j| j} |j|| | d \}}(| |(| } |j dd||||| | | |d| d |})t!|)j"| rm|)j#nd|)j$|)j%|dury|nd|dur| dSddS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, Gemma3nForConditionalGeneration >>> model = Gemma3nForConditionalGeneration.from_pretrained("google/gemma3n2-3b-mix-224") >>> processor = AutoProcessor.from_pretrained("google/gemma3n2-3b-mix-224") >>> prompt = "Where is the cat standing?" >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs,) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Where is the cat standing?\nsnow" ``` Nrrr")rrRT)r)rrrr)rr) rrr rJr;rrrrrr0)rr;r<r=r5r6r0)&rrgrrrr-rrr zeros_likerrrrrrrrrr|rrrrmasked_scatterget_audio_featuresr(rNrr_audio_soft_tokens_per_imagerrr4rr;r<r=)*rOrrrr rrJr;rr0rrrrrr per_layer_inputs_maskper_layer_inputs_tokensr vision_maskdummy_vision_token_idvision_input_ids vision_embedsexpanded_vision_mask audio_maskdummy_audio_token_idaudio_input_ids audio_embedsexpanded_audio_maskrrrraudio_padding_toksaudio_padding_embsaudio_batch_size audio_seq_lenaudio_embed_dimextra_padding_tokensextra_padding_featuresrrr0r0r1r^s1            zGemma3nModel.forwardzPProjects the last hidden state from the audio encoder into language model space.cKs0|j||fddi|}|j|jd}||_|S)a0 input_features (`torch.FloatTensor]` of shape `(num_images, seq_length, num_features)`): The tensors corresponding to the input audio. input_features_mask (`torch.FloatTensor]` of shape `(num_images, seq_length)`): The attention mask for the input audio. rTr)rrrr)rOrrr audio_outputsrr0r0r1r uszGemma3nModel.get_audio_featuresr9)NNNNNNNNNNNNNN)r)r*r+raccepts_loss_kwargsr$rIrrrrr-r7rrr>rrr:rrdr rcr8r^r'r rer0r0rPr1rs  ,      rz The base Gemma 3n model comprising a vision backbone, an audio backbone, a language model, and a language modeling head. c%sNeZdZiZddiZdeffdd ZddZdd Ze d e j d e e fd d Zee               d&de jdBd e j dBde j dBde jdBde jdBde jdBdedBde jdBde jdBde j dBde jdBdedBdedBdedBdee jBde e def"d d!Z          "   #d'fd$d% ZZS)(Gemma3nForConditionalGenerationrz(model.language_model.embed_tokens.weightrgcs<t|t||_tj|jj|jjdd|_ | dSr) rHrIrrYrJrvrrnrrrrzrPr0r1rIs   z(Gemma3nForConditionalGeneration.__init__cCrrZ)rYrr`r0r0r1rrz4Gemma3nForConditionalGeneration.get_input_embeddingscCrrZ)rYrrr0r0r1rrz4Gemma3nForConditionalGeneration.set_input_embeddingsrrcKs|jj|fi|SrZ)rYr)rOrrr0r0r1rsz2Gemma3nForConditionalGeneration.get_image_featuresNrrrr rrJr;rr0rrrrrrr rYcKs| dur| n|jj} |dur|n|jj}|jd ||||||||| | | | | |dd|}|j}t|tr:t| dn|}||dd|ddf}|j j }dura||}t |}||}d}| dur| }|dddddf}| dddf}|dur|dd|jd df|j}|||jdk}|||jdk}n|}|}t}|d|jjj}|d|j}|||}t|||j|j|j|j|jdS) a input_features_mask (torch.Tensor, *optional*, defaults to None): The attention mask for the input audio. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, Gemma3ForConditionalGeneration >>> model = Gemma3ForConditionalGeneration.from_pretrained("google/gemma-3-4b-it") >>> processor = AutoProcessor.from_pretrained("google/gemma-3-4b-it") >>> messages = [ ... { ... "role": "system", ... "content": [ ... {"type": "text", "text": "You are a helpful assistant."} ... ] ... }, ... { ... "role": "user", "content": [ ... {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"}, ... {"type": "text", "text": "Where is the cat standing?"}, ... ] ... }, ... ] >>> inputs = processor.apply_chat_template( ... messages, ... tokenizer=True, ... return_dict=True, ... return_tensors="pt", ... add_generation_prompt=True ... ) >>> # Generate >>> generate_ids = model.generate(**inputs) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "user\nYou are a helpful assistant.\n\n\n\n\n\nWhere is the cat standing?\nmodel\nBased on the image, the cat is standing in a snowy area, likely outdoors. It appears to" ``` NT)rrrr rrJr;rr0rrrrrr.rRr"r)r9r:r;r<r=r5r6r0)rgrrrYrrbrbrrget_text_configrr-rr[r_rrrrJCrossEntropyLossrrrr8r;r<r=r5r6)rOrrrr rrJr;rr0rrrrrrr rr<rr:rr9 shift_logits shift_labelsshift_attention_maskloss_fct flat_logits flat_labelsr0r0r1r^sjE $ z'Gemma3nForConditionalGeneration.forwardTFc sLtj|f|||||| | | |d |}|s| s$||d<||d<| |d<|S)N) r;rr rJr0rrris_first_iterationrrr)rHprepare_inputs_for_generation)rOrr;rr0rJrrr rrrrrr,r model_inputsrPr0r1r-, s( z=Gemma3nForConditionalGeneration.prepare_inputs_for_generation)NNNNNNNNNNNNNNr) NNNNNNNNNTNNF)r)r*r+rrr$rIrrrr-r7rrrrr:rdr rcrbr8r^r-rer0r0rPr1r#s     r#)rrr#rrXrh)rNN)r")crxcollections.abcrr dataclassesrtypingrr-torch.nnrJtorch.nn.functionalrrQrrcrr cache_utilsr r generationr integrationsr masking_utilsr rmodeling_layersrmodeling_outputsrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrrutils.output_capturingrautor!configuration_gemma3nr#r$r%r&r'r4r8Moduler?rfrrr4rXrsrrrrrrrrrrrdrbrr[r>rrrr;rXrkrhrrrr#__all__r0r0r0r1s            +emEI+Q&c ""wM;QJR2 G