o ºiPHã@s‚dZddlmZddlmZe e¡ZGdd„deƒZGdd„deƒZ Gdd „d eƒZ Gd d „d eƒZ Gd d „d eƒZ gd¢Z dS)zBlt model configurationé)ÚPretrainedConfig)ÚloggingcóFeZdZdZdZ          d‡fdd„ Z‡ZS)ÚBltLocalEncoderConfigzB Configuration class for the Blt Local Encoder component. Úblt_local_encoderéFééééNéçñh㈵øä>çé`瀄AÚsilué ç{®Gáz”?c ó¨||_||_||_||_||_||_|p||_|||_|p%td|dƒ|_ ||_ | |_ | |_ | |_ | |_| |_||_||_| dd¡tƒjdi|¤ddi¤ŽdS©NérÚtie_word_embeddingsF©©Ú vocab_sizeÚcross_attn_all_layersÚ cross_attn_kÚhidden_size_globalÚ hidden_sizeÚnum_attention_headsÚnum_key_value_headsÚhead_dimÚintÚintermediate_sizeÚnum_hidden_layersÚ rms_norm_epsÚdropoutÚmax_position_embeddingsÚ rope_thetaÚ rope_scalingÚ hidden_actÚinitializer_rangeÚpopÚsuperÚ__init__©Úselfrrrrrrr r$r%r&r'r(r)r*r#r+Úkwargs©Ú __class__rúf/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/blt/configuration_blt.pyr.ó&   zBltLocalEncoderConfig.__init__)rFrr r r Nr r rrrNrrr©Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú model_typer.Ú __classcell__rrr2r4ró(ïrcr)ÚBltLocalDecoderConfigzB Configuration class for the Blt Local Decoder component. Úblt_local_decoderrTrr r r Né r rrrrrrc rrrr/r2rr4r.Qr5zBltLocalDecoderConfig.__init__)rTrr r r Nr@r rrrNrrrr6rrr2r4r>Jr=r>cs>eZdZdZdZ        d‡fdd„ Z‡ZS)ÚBltGlobalTransformerConfigzG Configuration class for the Blt Global Transformer component. Úblt_global_transformerr r Nér rérrérc  s||_||_|p ||_|||_| ptd|dƒ|_||_||_||_||_ ||_ | |_ | |_ | |_ |  dd¡tƒjdi| ¤ddi¤ŽdSr)rrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.)r0rrr r$r%r&r'r(r)r*r#r+r1r2rr4r.ƒs   z#BltGlobalTransformerConfig.__init__) r r NrCr rrDrNrrErr6rrr2r4rA|s órAcs>eZdZdZdZ        d‡fdd„ Z‡ZS)ÚBltPatcherConfiga¿ Configuration class for the Blt Patcher/Entropy model component. Args: vocab_size (`int`, *optional*, defaults to 260): Vocabulary size of the Blt patcher model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling the patcher model. hidden_size (`int`, *optional*, defaults to 768): Dimension of the hidden representations. num_hidden_layers (`int`, *optional*, defaults to 14): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. rope_theta (`float`, *optional*, defaults to 10000.0): The base period of the RoPE embeddings. intermediate_size (`int`, *optional*, defaults to 2048): Dimension of the MLP representations. rope_scaling (`dict`, *optional*): Dictionary containing the RoPE scaling configuration. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. Ú blt_patcherrééé Né r rçˆÃ@r rc  s ||_||_||_||_|||_|dur|n||_||_||_||_| |_ d|_ | p3t d|jdƒ|_ | |_ | |_|  dd¡tƒjdi| ¤ddi¤ŽdS)NrrrrFr)rrr$rr!r r'r%r&r(r*r"r#r)r+r,r-r.)r0rrr$rr r'r%r&r(r#r)r+r1r2rr4r.Îs   zBltPatcherConfig.__init__) rrHrIrJNrKr rrLr Nrr6rrr2r4rF¦s %órFcsbeZdZdZdZdgZeeee dœZ         d‡fdd„ Z ‡Z S)Ú BltConfigas This is the configuration class to store the configuration of a [`BltModel`]. It is used to instantiate a Blt model according to the specified arguments, defining the model architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 260): Vocabulary size of the Blt model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`BltModel`]. max_position_embeddings (`int`, *optional*, defaults to 4096): The maximum sequence length that this model might ever be used with. patch_in_forward (`bool`, *optional*, defaults to `True`): Whether to perform patching during the forward pass. patch_size (`int`, *optional*, defaults to 4): Size of the patches used in the patching mechanism. patching_mode (`str`, *optional*, defaults to `"entropy"`): The mode used for patching, such as entropy-based patching. patching_threshold (`float`, *optional*, defaults to 1.34): Threshold value used for determining when to apply patches. patching_batch_size (`int`, *optional*, defaults to 1): Batch size used during the patching process. max_patch_length (`int`, *optional*): Maximum length of patches that can be generated. cross_attn_k (`int`, *optional*, defaults to 2): Number of cross-attention heads used in the model. encoder_hash_byte_group_size (`list`, *optional*): List of byte group sizes used in the encoder hash function. encoder_hash_byte_group_vocab (`int`, *optional*, defaults to 500002): Vocabulary size for the encoder hash byte groups. encoder_hash_byte_group_nb_functions (`int`, *optional*, defaults to 1): Number of hash functions used in the encoder byte grouping. patcher_config (`BltPatcherConfig`, *optional*): Configuration for the patcher component of the model. encoder_config (`BltLocalEncoderConfig`, *optional*): Configuration for the local encoder component of the model. decoder_config (`BltLocalDecoderConfig`, *optional*): Configuration for the local decoder component of the model. global_config (`BltGlobalTransformerConfig`, *optional*): Configuration for the global transformer component of the model. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rope_theta (`float`, *optional*, defaults to 500000.0): The base period of the RoPE embeddings. rope_scaling (`dict`, *optional*): Dictionary containing the RoPE scaling configuration. ```python >>> from transformers import BltModel, BltConfig >>> # Initializing a Blt configuration >>> configuration = BltConfig() >>> # Initializing a model from the configuration >>> model = BltModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` Checkpoint: [facebook/blt](https://huggingface.co/facebook/blt) ÚbltÚpast_key_values)Úpatcher_configÚencoder_configÚdecoder_configÚ global_configrrDTéÚentropyç€ø]õ?r Nré"¡Frrc sF||_||_||_||_||_||_||_||_||_||_ ||_ |  dd¡|_ |  dd¡|_ |  d¡|_|  dd¡|_| |_| pDgd¢|_| |_| |_| dur\t|d |_t d ¡nt| tƒrp|  d |¡tdi| ¤Ž|_nt| tƒrx| |_|durˆt|d |_t d ¡nt|tƒrœ| d |¡tdi|¤Ž|_nt|tƒr¤||_|dur´t|d |_t d ¡nt|tƒrÈ| d |¡tdi|¤Ž|_nt|tƒrÐ||_|duràt|d |_ t d¡nt|tƒrô| d |¡tdi|¤Ž|_ nt|tƒrü||_ |jj!|j}||j j!kr |nd|j _"| #dd¡t$ƒj%dd|i|¤ŽdS)NÚpatching_deviceÚcudaÚrealtime_patchingTÚpatching_threshold_addÚ monotonicityF)rrTééér)r+z8patcher_config is None, using default Blt patcher configr+z8encoder_config is None, using default Blt encoder configz8decoder_config is None, using default Blt decoder configz6global_config is None, using default Blt global configrr)&rr'r+r(r)Úpatch_in_forwardÚ patch_sizeÚ patching_modeÚpatching_thresholdÚpatching_batch_sizeÚmax_patch_lengthÚgetrXrZr[r\rÚencoder_hash_byte_group_sizeÚencoder_hash_byte_group_vocabÚ$encoder_hash_byte_group_nb_functionsrFrPÚloggerÚinfoÚ isinstanceÚdictÚ setdefaultrrQr>rRrArSrÚencoder_cross_output_sizer,r-r.)r0rr'r`rarbrcrdrerrgrhrirPrQrRrSrr+r(r)r1ror2rr4r.>sp                     ÿ zBltConfig.__init__)rrDTrTrUrVr NrNrWr NNNNFrrN) r7r8r9r:r;Úkeys_to_ignore_at_inferencerFrr>rAÚ sub_configsr.r<rrr2r4rMòs<Bü ërM)rMrFrr>rAN)r:Úconfiguration_utilsrÚutilsrÚ get_loggerr7rjrr>rArFrMÚ__all__rrrr4Ús   22*L 0