o ¾e¦i‚;ã@s`dZddlmZddlmZe e¡ZGdd„deƒZGdd„deƒZ Gdd „d eƒZ d gZ d S) zMllama model configurationé)ÚPreTrainedConfig)Úloggingcs¬eZdZdZdZdZ          d&dedededededededededededede edBde e edBd ef‡fd!d"„ Z e d#efd$d%„ƒZ ‡Z S)'ÚMllamaVisionConfiga+ This is the configuration class to store the configuration of a [`MllamaVisionModel`]. It is used to instantiate an Mllama vision model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Mllama-11B. e.g. [meta-llama/Llama-3.2-11B-Vision](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 1280): Dimensionality of the encoder layers and the pooler layer. hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` `"quick_gelu"` are supported. num_hidden_layers (`int`, *optional*, defaults to 32): Number of hidden layers in the Transformer encoder. num_global_layers (`int`, *optional*, defaults to 8): Number of global layers in the Transformer encoder. Vision model has a second transformer encoder, called global. num_attention_heads (`int`, *optional*, defaults to 16): Number of attention heads for each attention layer in the Transformer encoder. num_channels (`int`, *optional*, defaults to 3): Number of channels in the input image. intermediate_size (`int`, *optional*, defaults to 5120): Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder. vision_output_dim (`int`, *optional*, defaults to 7680): Dimensionality of the vision model output. Includes output of transformer encoder with intermediate layers and global transformer encoder. image_size (`int`, *optional*, defaults to 448): The size (resolution) of each image *tile*. patch_size (`int`, *optional*, defaults to 14): The size (resolution) of each patch. norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the layer normalization layers. max_num_tiles (`int`, *optional*, defaults to 4): Maximum number of tiles for image splitting. intermediate_layers_indices (`list[int]`, *optional*, defaults to [3, 7, 15, 23, 30]): Indices of intermediate layers of transformer encoder from which to extract and output features. These output features are concatenated with final hidden state of transformer encoder. supported_aspect_ratios (`list[list[int]]`, *optional*): List of supported aspect ratios for image splitting. If not specified, the default supported aspect ratios are [[1, 1], [1, 2], [1, 3], [1, 4], [2, 1], [2, 2], [3, 1], [4, 1]] for `max_num_tiles=4`. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. Example: ```python >>> from transformers import MllamaVisionConfig, MllamaVisionModel >>> # Initializing a Llama config >>> config = MllamaVisionConfig() >>> # Initializing a vision model from the mllama-11b style configuration >>> model = MllamaVisionModel(config) >>> # Accessing the model configuration >>> configuration = model.config ```Úmllama_vision_modelÚ vision_configéÚgelué ééréééÀéçñh㈵øä>éNç{®Gáz”?Ú hidden_sizeÚ hidden_actÚnum_hidden_layersÚnum_global_layersÚnum_attention_headsÚ num_channelsÚintermediate_sizeÚvision_output_dimÚ image_sizeÚ patch_sizeÚnorm_epsÚ max_num_tilesÚintermediate_layers_indicesÚsupported_aspect_ratiosÚinitializer_rangec sÌ|dur&| dkr tdƒ‚ddgddgddgddgddgddgddgddgg}| dur.gd¢} ||_||_||_||_||_| |_||_| |_| |_ ||_ | |_ | |_ ||_ ||_||_tƒjdi|¤ŽdS)Nrz;max_num_tiles must be 4 for default supported aspect ratioséér)réééé©)Ú ValueErrorrrrrrrrrrrrrÚattention_headsr r!ÚsuperÚ__init__)Úselfrrrrrrrrrrrrrr r!Úkwargs©Ú __class__r(úm/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/mllama/configuration_mllama.pyr,Xs,4zMllamaVisionConfig.__init__ÚreturncCs t|jƒS)N)Úlenr )r-r(r(r1Úmax_aspect_ratio_id„s z&MllamaVisionConfig.max_aspect_ratio_id)rrr r r rr r rrrrNNr)Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú model_typeÚbase_config_keyÚintÚstrÚfloatÚlistr,Úpropertyr4Ú __classcell__r(r(r/r1rsh>ðþýüûúùø ÷ ö õ ô ó òñð,rc&s¬eZdZdZdZdZdZ            d*dededededededede d Bde ded e d!e d"e d#e ed Bd$e d%ed&ed'ed Bf$‡fd(d)„ Z ‡ZS)+ÚMllamaTextConfigaV This is the configuration class to store the configuration of a [`MllamaTextModel`]. It is used to instantiate an Mllama text model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Mllama-11B. e.g. [meta-llama/Llama-3.2-11B-Vision](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 128256): Vocabulary size of the Mllama text model. Defines the maximum number of different tokens that can be represented by the `inputs_ids` passed when calling [`MllamaTextModel`]. hidden_size (`int`, *optional*, defaults to 4096): Dimensionality of the embeddings and hidden states. hidden_act (`str` or `Callable`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the encoder and pooler. num_hidden_layers (`int`, *optional*, defaults to 40): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 32): Number of attention heads for each attention layer in the Transformer encoder. num_key_value_heads (`int`, *optional*, defaults to 8): This is the number of key_value heads that should be used to implement Grouped Query Attention. If not specified, will default to `num_attention_heads`. intermediate_size (`int`, *optional*, defaults to 14336): Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. max_position_embeddings (`int`, *optional*, defaults to 131072): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings cross_attention_layers (`list[int]`, *optional*): Indices of the cross attention layers. If not specified, will default to [3, 8, 13, 18, 23, 28, 33, 38]. dropout (`float`, *optional*, defaults to 0): The dropout probability for self- and cross-attention layers. bos_token_id (`int`, *optional*, defaults to 128000): The id of the beginning of sentence token. eos_token_id (`int`, *optional*, defaults to 128001): The id of the end of sentence token. pad_token_id (`int`, *optional*, defaults to 128004): The id of the padding token. Example: ```python >>> from transformers import MllamaTextModel, MllamaTextConfig >>> # Initializing a Mllama text config >>> config = MllamaTextConfig() >>> # Initializing a model from the Mllama text configuration >>> model = MllamaTextModel(config) >>> # Accessing the model configuration >>> configuration = model.config ```Úmllama_text_modelÚ text_configg€„AéõéÚsilué(r r é8NrérTFééôéôéôÚ vocab_sizerrrrÚnum_key_value_headsrÚrope_parametersÚ rms_norm_epsÚmax_position_embeddingsr!Ú use_cacheÚtie_word_embeddingsÚcross_attention_layersÚdropoutÚ bos_token_idÚ eos_token_idÚ pad_token_idc s’|durgd¢}||_||_||_||_||_||_| |_| |_| |_||_ ||_ ||_ | |_ ||_ | |_||_||_||_tƒjdi|¤ŽdS)N)rr é ér&éé!é&r()rNrrUrrrOr!rSrQrrVrrRrPrTrYrWrXr+r,)r-rNrrrrrOrrPrQrRr!rSrTrUrVrWrXrYr.r/r(r1r,Ñs*zMllamaTextConfig.__init__)rDrErFrGr r rHNrrIrTFNrJrKrLrM)r5r6r7r8r9r:Ú default_thetar;r<Údictr=Úboolr>r,r@r(r(r/r1rA‰sxCíþýüûúùø ÷ ö õ ô óò ñðïîírAcs>eZdZdZdZddiZeedœZ   d ‡fdd „ Z ‡Z S) Ú MllamaConfigaÍ This is the configuration class to store the configuration of a [`MllamaForConditionalGeneration`]. It is used to instantiate an Mllama model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Mllama-9B. e.g. [meta-llama/Llama-3.2-11B-Vision](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vision_config (`Union[AutoConfig, dict]`, *optional*, defaults to `MllamaVisionConfig`): The config object or dictionary of the vision backbone. text_config (`Union[AutoConfig, dict]`, *optional*, defaults to `MllamaTextConfig`): The config object or dictionary of the text backbone. image_token_index (`int`, *optional*, defaults to 128256): The image token index to encode the image prompt. Example: ```python >>> from transformers import MllamaForConditionalGeneration, MllamaConfig, MllamaVisionConfig, MllamaTextConfig >>> # Initializing a CLIP-vision config >>> vision_config = MllamaVisionConfig() >>> # Initializing a Llama config >>> text_config = MllamaTextConfig() >>> # Initializing a mllama-11b style configuration >>> configuration = MllamaConfig(vision_config, text_config) >>> # Initializing a model from the mllama-11b style configuration >>> model = MllamaForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```ÚmllamaÚimage_token_idÚimage_token_index)rCrNrDc s¬|durtƒ|_t d¡nt|tƒrtdi|¤Ž|_nt|tƒr$||_||_|dur5tƒ|_t d¡nt|tƒrCtdi|¤Ž|_nt|tƒrK||_t ƒj di|¤ŽdS)Nz9vision_config is None, using default mllama vision configz5text_config is None, using default mllama text configr() rrÚloggerÚinfoÚ isinstancer`rerArCr+r,)r-rrCrer.r/r(r1r,.s       zMllamaConfig.__init__)NNrD) r5r6r7r8r9Ú attribute_maprArÚ sub_configsr,r@r(r(r/r1rbs'ÿ ürbN) r8Úconfiguration_utilsrÚutilsrÚ get_loggerr5rfrrArbÚ__all__r(r(r(r1Ús   sw J