o ei{@sddlmZddlmZmZddlmZddlmZddl m Z e e Z Gdd d eZGd d d eZGd d d eZGdddeZgdZdS))Any)PreTrainedConfiglayer_type_validation)RopeParameters)logging)SiglipVisionConfigc2reZdZdZdZdgZddddddddZdgdgfd d gd gfd gd gfd Zd d dZ                   ! ! ! !d=d"e d!Bd#e d!Bd$e d!Bd%e d!Bd&e d!Bd'e d!Bd(e d!Bd)e d!Bd*e d!Bd+e d!Bd,e d!Bd-e d!Bd.e d!Bd/e d!Bd0e d!Bd1e d!Bd2e d!Bd3e d!Bd4e d!Bd5e e d!Bd6e d!Bd7e d!Bd8eee efBd!Bf.fd9d: Zd>d;d<ZZS)?T5Gemma2TextConfiga This is the configuration class to store the configuration of a [`T5Gemma2TextModel`]. It is used to instantiate the encoder's text model portion of the T5Gemma2 Model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the T5Gemma2Text-7B. e.g. [google/t5gemma2_text-7b](https://huggingface.co/google/t5gemma2_text-7b) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 262208): Vocabulary size of the T5Gemma2Text model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`T5Gemma2TextModel`] hidden_size (`int`, *optional*, defaults to 2304): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 9216): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 26): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 8): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*, defaults to 4): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. head_dim (`int`, *optional*, defaults to 256): The attention head dimension. hidden_activation (`str` or `function`, *optional*, defaults to `"gelu_pytorch_tanh"`): The non-linear activation function (function or string) in the decoder. Will default to `"gelu_pytorch_tanh"` if not specified. `"gelu_pytorch_tanh"` uses an approximation of the `"gelu"` activation function. max_position_embeddings (`int`, *optional*, defaults to 131072): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*, defaults to 0): Padding token id. eos_token_id (`int`, *optional*, defaults to 1): End of stream token id. bos_token_id (`int`, *optional*, defaults to 2): Beginning of stream token id. attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. query_pre_attn_scalar (`float`, *optional*, defaults to 256): Scaling factor used on the attention scores sliding_window (`int`, *optional*, defaults to 4096): In T5Gemma2Text, every other layer uses sliding window attention. This is the size of the sliding window. layer_types (`list`, *optional*): Attention pattern for each layer. final_logit_softcapping (`float`, *optional*): Scaling factor when applying tanh softcapping on the logits. attn_logit_softcapping (`float`, *optional*): Scaling factor when applying tanh softcapping on the attention scores. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. t5gemma2_textpast_key_valuescolwiserowwisezlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.o_projzlayers.*.mlp.gate_projzlayers.*.mlp.up_projzlayers.*.mlp.down_proj input_ids inputs_embeds hidden_statesattention_mask embed_tokenslayersnorm.A@globallocal@ $gelu_pytorch_tanh{Gz?ư>TrrFN vocab_size hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headsnum_key_value_headshead_dimhidden_activationmax_position_embeddingsinitializer_range rms_norm_eps use_cache pad_token_id eos_token_id bos_token_idattention_biasattention_dropoutquery_pre_attn_scalarsliding_window layer_typesfinal_logit_softcappingattn_logit_softcappingrope_parametersc | _|_|_|_| _|_|_|_|_|_ |_ | _ | _ | _ |_|_|_|_|_|_|_|_|dd_jdur[fddtjD_tjj|_tjdi|dS)Nsliding_window_patternc&g|]}t|djrdndqSr)sliding_attentionfull_attentionbool_sliding_window_pattern.0iselfq/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/t5gemma2/configuration_t5gemma2.py z/T5Gemma2TextConfig.__init__..rRr8r:r9r,r4r-r.r/r0r2r1r5r6r7r;r<r3r=r>r@rAr?getrLrangerrBsuper__init__rQr,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBkwargs __class__rPrSrZw<  zT5Gemma2TextConfig.__init__cK|dd}ddiddid}|jdur|jn||_|dur&|jd||jddur5ddi|jd<|jdd|d|jd|jddurTddi|jd<|jdd|d |jd ||j|d |S N rope_scaling rope_typedefault)rHrIrI rope_thetarrHrope_local_base_freqr) ignore_keyspoprBupdaterW setdefault default_thetastandardize_rope_params validate_roperQignore_keys_at_rope_validationr\rbdefault_rope_paramsrRrRrSconvert_rope_params_to_dict(    z.T5Gemma2TextConfig.convert_rope_params_to_dictrrr r!r"r#r$r%r&r'r(Trr)rFr*r$r+NNNNN__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencebase_model_tp_planbase_model_pp_planrlintstrfloatrKlistrdictrZrr __classcell__rRrRr]rSr D         ?r cseZdZdZdZddddZeedZ   ddee e e fBdBdee e e fBdBde dBde dBde dBde dBde dBdedBffdd ZZS)T5Gemma2EncoderConfigaq This is the configuration class to store the configuration of a [`T5Gemma2EncoderForConditionalGeneration`]. It is used to instantiate an T5Gemma2EncoderForConditionalGeneration according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the PaliGemma-2B. e.g. [google/gemma-3-4b](https://huggingface.co/google/gemma-3-4b) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: text_config (`Union[T5Gemma2EncoderTextConfig, dict]`, *optional*): The config object of the text backbone. vision_config (`Union[AutoConfig, dict]`, *optional*): Custom vision config or dict. mm_tokens_per_image (`int`, *optional*, defaults to 256): The number of tokens per image embedding. boi_token_index (`int`, *optional*, defaults to 255999): The begin-of-image token index to wrap the image prompt. eoi_token_index (`int`, *optional*, defaults to 256000): The end-of-image token index to wrap the image prompt. image_token_index (`int`, *optional*, defaults to 262144): The image token index to encode the image prompt. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. tie_word_embeddings (`bool`, *optional*, defaults to `True`): Whether to tie weight embeddings Example: ```python >>> from transformers import T5Gemma2EncoderForConditionalGeneration, T5Gemma2EncoderConfig, SiglipVisionConfig, T5Gemma2EncoderTextConfig >>> # Initializing a Siglip-like vision config >>> vision_config = SiglipVisionConfig() >>> # Initializing a T5Gemma2Encoder Text config >>> text_config = T5Gemma2EncoderTextConfig() >>> # Initializing a T5Gemma2Encoder gemma-3-4b style configuration >>> configuration = T5Gemma2EncoderConfig(vision_config, text_config) >>> # Initializing a model from the gemma-3-4b style configuration >>> model = T5Gemma2EncoderTextConfig(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```t5gemma2_encoderimage_token_indexboi_token_indexeoi_token_index)image_token_id boi_token_id eoi_token_id) text_config vision_configNr$r'Trrmm_tokens_per_imager5tie_word_embeddingsc  s|dur t}tdn t|trtdi|}t|tr&tdi|}n |dur2t}td||_||_||_||_ ||_ ||_ ||_ ||_ tjdi| dS)NzItext_config is None, using default T5Gemma2EncoderTextConfig text config.zFvision_config is None, using default SiglipVisionConfig vision config.rR)r loggerinfo isinstancerr rrrrrrr5rrYrZ) rQrrrrrrr5rr\r]rRrSrZs&     zT5Gemma2EncoderConfig.__init__)NNr$rrrr'T)rwrxryrzr{ attribute_mapr r sub_configsrrrrrrKrZrrRrRr]rSrsF1 rc2r )?T5Gemma2DecoderConfiga  This is the configuration class to store the configuration of a [`T5Gemma2DecoderModel`]. It is used to instantiate the decoder text model portion of the T5Gemma2 Model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the T5Gemma2Decoder-7B. e.g. [google/t5gemma2_text-7b](https://huggingface.co/google/t5gemma2_text-7b) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 262208): Vocabulary size of the T5Gemma2Decoder model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`T5Gemma2DecoderModel`] hidden_size (`int`, *optional*, defaults to 2304): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 9216): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 26): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 8): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*, defaults to 4): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. head_dim (`int`, *optional*, defaults to 256): The attention head dimension. hidden_activation (`str` or `function`, *optional*, defaults to `"gelu_pytorch_tanh"`): The non-linear activation function (function or string) in the decoder. Will default to `"gelu_pytorch_tanh"` if not specified. `"gelu_pytorch_tanh"` uses an approximation of the `"gelu"` activation function. max_position_embeddings (`int`, *optional*, defaults to 131072): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*, defaults to 0): Padding token id. eos_token_id (`int`, *optional*, defaults to 1): End of stream token id. bos_token_id (`int`, *optional*, defaults to 2): Beginning of stream token id. attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. query_pre_attn_scalar (`float`, *optional*, defaults to 256): Scaling factor used on the attention scores sliding_window (`int`, *optional*, defaults to 4096): In T5Gemma2Decoder, every other layer uses sliding window attention. This is the size of the sliding window. layer_types (`list`, *optional*): Attention pattern for each layer. final_logit_softcapping (`float`, *optional*): Scaling factor when applying tanh softcapping on the logits. attn_logit_softcapping (`float`, *optional*): Scaling factor when applying tanh softcapping on the attention scores. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. t5gemma2_decoderr rrrrrrrrrrrrrr r!r"r#r$r%r&r'r(Trr)rFr*r+Nr,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBc rC)NrDrEcrFrGrJrMrPrRrSrTrUz2T5Gemma2DecoderConfig.__init__..rRrVr[r]rPrSrZr_zT5Gemma2DecoderConfig.__init__cKr`rarhrorRrRrSrrrsz1T5Gemma2DecoderConfig.convert_rope_params_to_dictrtrurvrRrRr]rSr7rrcseZdZdZdZdgZeedZdddZ   dd ee e e fBdBdee e e fBdBde dededededede dBffdd ZZS)T5Gemma2ConfigaV This is the configuration class to store the configuration of a [`T5Gemma2Model`]. It is used to instantiate an T5Gemma2 model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to a hypothetical balanced Gemma3 encoder-decoder model. e.g. [google/t5gemma-2-270m-270m](https://huggingface.co/google/t5gemma-2-270m-270m) Configuration objects inherit from [PreTrainedConfig] and can be used to control the model outputs. Read the documentation from [PreTrainedConfig] for more information. Args: encoder (`Union[T5Gemma2EncoderConfig, dict]`, optional, *optional*): Configuration for the encoder. decoder (`Union[T5Gemma2DecoderConfig, dict]`, optional, *optional*): Configuration for the decoder. is_encoder_decoder (bool, optional, *optional*, defaults to `True`): Whether the model is used as an encoder/decoder or not. dropout_rate (`float`, *optional*, defaults to 0.0): The ratio for all dropout layers (following T5). attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for attention. classifier_dropout_rate (`float`, *optional*, defaults to 0.0): The dropout ratio for classifier (following T5). initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. image_token_index (`int`, *optional*, defaults to 256001): The image token index to encode the image prompt. Defaults to 256001, which is right after the eoi_token_index. Note this is different from Gemma 3. tie_word_embeddings (`bool`, *optional*, defaults to `True`): Whether to tie weight embeddings ```python >>> from transformers import T5Gemma2Config, T5Gemma2Model >>> t5gemma2_config = T5Gemma2Config.from_pretrained("google/t5gemma-270m-270m") >>> model = T5Gemma2Model(t5gemma2_config) ``` t5gemma2r )encoderdecoderrr)rrNTr*r'rris_encoder_decoder dropout_rater<classifier_dropout_rater5rc  st|tr td i|}n|durt}tdnt|ts(tt|dt|tr5td i|}n|durBt}tdnt|tsPtt|d|jj |j kretd|jj d|j d|sktd|jj |j krtd|jj d|j d||j_ ||j_ ||j _ ||_||_||_ ||_ ||_d D] } | | vrt|| | | <q||_||_|j|_||_| |_tjd d |i| dS) NzDencoder is None, using default T5Gemma2EncoderConfig encoder config.z is not supported.zDdecoder is None, using default T5Gemma2DecoderConfig decoder config.zBImbalanced encoder-decoder is not supported in T5Gemma2: encoder (z) vs decoder (z).z4T5Gemma2Model only support encoder-decoder modeling.zRImbalanced encoder-decoder vocabulary size is not supported in T5Gemma2: encoder ()r:r8r9r,rrR)rrrrr ValueErrortyperrr-r,rr<rrrrgetattrrr5rrrYrZ) rQrrrrr<rr5rrr\special_token_keyr]rRrSrZsd      zT5Gemma2Config.__init__) NNTr*r*r*r'rT)rwrxryrzr{r|rrrrrrrrKrrrZrrRrRr]rSrsL$  r)rr rrN)typingrconfiguration_utilsrrmodeling_rope_utilsrutilsrsiglipr get_loggerrwrr rrr__all__rRrRrRrSs     6b6 y