o eiM@sdZddlmZddlmZddlmZeeZ GdddeZ GdddeZ Gd d d eZ Gd d d eZ Gd ddeZgdZdS)zBlt model configuration)PreTrainedConfig)RopeParameters)loggingc seZdZdZdZdZ         d#ded Bded Bded Bded Bded Bded Bded Bded Bded Bded Bded Bde e e e fBd Bde d Bded Bd ed Bffd!d" Z Z S)$BltLocalEncoderConfigzB Configuration class for the Blt Local Encoder component. blt_local_encoderAFNh㈵>`silu {Gz? vocab_sizecross_attn_all_layers cross_attn_khidden_size_global hidden_sizenum_attention_headsnum_key_value_headsnum_hidden_layers rms_norm_epsdropoutmax_position_embeddingsrope_parameters hidden_actintermediate_sizeinitializer_rangec s||_||_||_||_||_||_|p||_|||_|p%td|d|_ ||_ | |_ | |_ | |_ | |_||_| |_|ddtjdi|ddidS)Nrtie_word_embeddingsF)rrrrrrrhead_dimintr!rrrrr r"rpopsuper__init__)selfrrrrrrrrrrrrr r!r"kwargs __class__r%g/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/blt/configuration_blt.pyr* s$   zBltLocalEncoderConfig.__init__)rFr r r r Nr rrrNrrr__name__ __module__ __qualname____doc__ model_type default_thetar'boolfloatrdictstrr* __classcell__r%r%r-r/rsd     rc(seZdZdZdZdZ          d(ded Bded Bded Bded Bded Bded Bded Bded Bded Bded Bded Bde e e e fBd Bde d Bd ed Bd!ed Bd"ed Bd#ed Bd$ed Bd%ed Bf&fd&d' Z Z S))BltLocalDecoderConfigzB Configuration class for the Blt Local Decoder component. blt_local_decoderrrTr r r r N rrrrrrFrrrrrrrrrrrrr r!r" pad_token_id bos_token_id eos_token_idr$c s||_||_||_||_||_||_|p||_|||_|p%td|d|_ ||_ | |_ | |_ | |_ | |_||_||_||_||_d|_| |_tjdi|dSNr#rFr%)rrrrrrrr&r'r!rrrrr r"r?r@rAr$rr)r*)r+rrrrrrrrrrrrr r!r"r?r@rAr$r,r-r%r/r*Qs*  zBltLocalDecoderConfig.__init__)rTr r r r Nr>rrrNrrrNNNFr0r%r%r-r/r<Is|     r<cseZdZdZdZdZ       ddedBdedBdedBdedBdedBdedBdedBdee e efBdBde dBdedBdedBde dBffdd Z Z S)BltGlobalTransformerConfigzG Configuration class for the Blt Global Transformer component. blt_global_transformerrr r NrrrrFrrrrrrrrr r!r"r$c  s|||_||_|p ||_|||_| ptd|d|_||_||_||_||_ | |_ | |_ d|_ ||_ tjdi| dSrB)rrrr&r'r!rrrrr r"r$rr)r*)r+rrrrrrrrr r!r"r$r,r-r%r/r*s  z#BltGlobalTransformerConfig.__init__) r r NrErrrFNrrGrF)r1r2r3r4r5r6r'r8rr9r:r7r*r;r%r%r-r/rCsR     rCcseZdZdZdZ        ddedBdedBdedBdedBdedBdedBdedBdedBdedBdeee efBdBdedBde dBffdd Z Z S)BltPatcherConfiga Configuration class for the Blt Patcher/Entropy model component. Args: vocab_size (`int`, *optional*, defaults to 260): Vocabulary size of the Blt patcher model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling the patcher model. hidden_size (`int`, *optional*, defaults to 768): Dimension of the hidden representations. num_hidden_layers (`int`, *optional*, defaults to 14): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. intermediate_size (`int`, *optional*, defaults to 2048): Dimension of the MLP representations. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. blt_patcherr N rrr rFrrrrrrrrr!rr"r$c  s||_||_||_||_|||_|dur|n||_||_||_||_d|_ | p0t d|jd|_ | |_ | |_ d|_tjdi| dS)Nrr#rFr%)rrrrr&rrrrr r'r!r"rr$r)r*)r+rrrrrrrrr!rr"r$r,r-r%r/r*s zBltPatcherConfig.__init__) rrJrKrLNrMrrr NrF) r1r2r3r4r5r'r8rr9r:r7r*r;r%r%r-r/rHsP%     rHc0s(eZdZdZdZdgZdZeee e dZ        d*de d Bde d Bde d Bde d Bded Bded Bde d Bde d Bde d Bde d Bde d Bde d Bded Bded Bd ed Bd!ed Bd"e d Bd#e d Bd$e d Bd%e d Bd&ed Bd'eeeefBd Bf,fd(d) ZZS)+ BltConfiga< This is the configuration class to store the configuration of a [`BltModel`]. It is used to instantiate a Blt model according to the specified arguments, defining the model architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 260): Vocabulary size of the Blt model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`BltModel`]. max_position_embeddings (`int`, *optional*, defaults to 4096): The maximum sequence length that this model might ever be used with. patch_in_forward (`bool`, *optional*, defaults to `True`): Whether to perform patching during the forward pass. patch_size (`int`, *optional*, defaults to 4): Size of the patches used in the patching mechanism. patching_mode (`str`, *optional*, defaults to `"entropy"`): The mode used for patching, such as entropy-based patching. patching_threshold (`float`, *optional*, defaults to 1.34): Threshold value used for determining when to apply patches. patching_batch_size (`int`, *optional*, defaults to 1): Batch size used during the patching process. max_patch_length (`int`, *optional*): Maximum length of patches that can be generated. cross_attn_k (`int`, *optional*, defaults to 2): Number of cross-attention heads used in the model. encoder_hash_byte_group_size (`list`, *optional*): List of byte group sizes used in the encoder hash function. encoder_hash_byte_group_vocab (`int`, *optional*, defaults to 500002): Vocabulary size for the encoder hash byte groups. encoder_hash_byte_group_nb_functions (`int`, *optional*, defaults to 1): Number of hash functions used in the encoder byte grouping. patcher_config (`BltPatcherConfig`, *optional*): Configuration for the patcher component of the model. encoder_config (`BltLocalEncoderConfig`, *optional*): Configuration for the local encoder component of the model. decoder_config (`BltLocalDecoderConfig`, *optional*): Configuration for the local decoder component of the model. global_config (`BltGlobalTransformerConfig`, *optional*): Configuration for the global transformer component of the model. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. ```python >>> from transformers import BltModel, BltConfig >>> # Initializing a Blt configuration >>> configuration = BltConfig() >>> # Initializing a model from the configuration >>> model = BltModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` Checkpoint: [facebook/blt](https://huggingface.co/facebook/blt) bltpast_key_valuesr)patcher_configencoder_configdecoder_config global_configrrFTentropy]?r Nr "Frrrpatch_in_forward patch_size patching_modepatching_thresholdpatching_batch_sizemax_patch_lengthrencoder_hash_byte_group_sizeencoder_hash_byte_group_vocab$encoder_hash_byte_group_nb_functionsrQrRrSrTr$r?r@rAr"rc sH||_||_||_||_||_||_||_||_||_| dd|_ | dd|_ | d|_ | dd|_ | |_| p>gd|_| |_| |_| durVt|d |_td nt| trj| d |tdi| |_nt| trr| |_|durt|d |_td nt|tr|d |tdi||_nt|tr||_|durt|d |_td nt|tr|d |tdi||_nt|tr||_|durt|d |_tdnt|tr|d |tdi||_nt|tr||_|jj|j}||jjkr|nd|j_ ||_!||_"||_#||_$||_%t&j'di|dS)Npatching_devicecudarealtime_patchingTpatching_threshold_add monotonicityF)rrUr#)r"z8patcher_config is None, using default Blt patcher configr"z8encoder_config is None, using default Blt encoder configz8decoder_config is None, using default Blt decoder configz6global_config is None, using default Blt global configr%)(rrr"rYrZr[r\r]r^getrbrdrerfrr_r`rarHrQloggerinfo isinstancer9 setdefaultrrRr<rSrCrTrencoder_cross_output_sizer?r@rAr$rr)r*)r+rrrYrZr[r\r]r^rr_r`rarQrRrSrTr$r?r@rAr"rr,ror-r%r/r*@st                     zBltConfig.__init__)rrFTrUrVrWr Nr NrXr NNNNFNNNrN)r1r2r3r4r5keys_to_ignore_at_inferencer6rHrr<rC sub_configsr'r7r:r8r9rr*r;r%r%r-r/rNsB      rN)rNrHrr<rCN)r4configuration_utilsrmodeling_rope_utilsrutilsr get_loggerr1rkrr<rCrHrN__all__r%r%r%r/s    17)J 5