o ei9@sXddlmZddlmZmZddlmZddlmZe e Z GdddeZ dgZ dS) )Literal)PreTrainedConfiglayer_type_validation)RopeParameters)loggingcFseZdZdZdZdgZdddZfddZ                             dJdedBdedBdedBd edBd!edBd"e dBd#edBd$e dBd%e dBd&e dBd'e dBd(edBd)edBd*edBd+edBd,edBd-e dBd.e dBd/e e dBd0e ed1efdBd2edBd3e dBd4e dBd5e dBd6e dBd7ed8d9e dBd:e dBd;e dBdedBd?e dBd@e dBfDfdAdB ZdKdCdDZfdEdFZedGdHZejdIdHZZS)LModernBertConfigaO This is the configuration class to store the configuration of a [`ModernBertModel`]. It is used to instantiate an ModernBert model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the ModernBERT-base. e.g. [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 50368): Vocabulary size of the ModernBert model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`ModernBertModel`] hidden_size (`int`, *optional*, defaults to 768): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 1152): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 22): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer decoder. hidden_activation (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the decoder. Will default to `"gelu"` if not specified. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_cutoff_factor (`float`, *optional*, defaults to 2.0): The cutoff factor for the truncated_normal_initializer for initializing all weight matrices. norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. norm_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the normalization layers. pad_token_id (`int`, *optional*, defaults to 50283): Padding token id. eos_token_id (`int`, *optional*, defaults to 50282): End of stream token id. bos_token_id (`int`, *optional*, defaults to 50281): Beginning of stream token id. cls_token_id (`int`, *optional*, defaults to 50281): Classification token id. sep_token_id (`int`, *optional*, defaults to 50282): Separation token id. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. layer_types (`list`, *optional*): Attention pattern for each layer. rope_parameters (`dict`, *optional*): Dictionary mapping attention patterns (`"full_attention"`, `"sliding_attention"`) to `RopeParameters`. Each value should be a dictionary containing `rope_type` and optional scaling parameters. local_attention (`int`, *optional*, defaults to 128): The window size for local attention. embedding_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the embeddings. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the MLP layers. mlp_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the MLP layers. decoder_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the decoder layers. classifier_pooling (`str`, *optional*, defaults to `"cls"`): The pooling method for the classifier. Should be either `"cls"` or `"mean"`. In local attention layers, the CLS token doesn't attend to all tokens on long sequences. classifier_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the classifier. classifier_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the classifier. classifier_activation (`str`, *optional*, defaults to `"gelu"`): The activation function for the classifier. deterministic_flash_attn (`bool`, *optional*, defaults to `False`): Whether to use deterministic flash attention. If `False`, inference will be faster but not deterministic. sparse_prediction (`bool`, *optional*, defaults to `False`): Whether to use sparse prediction for the masked language model instead of returning the full dense logits. sparse_pred_ignore_index (`int`, *optional*, defaults to -100): The index to ignore for the sparse prediction. reference_compile (`bool`, *optional*): Whether to compile the layers of the model which were compiled during pretraining. If `None`, then parts of the model will be compiled if 1) `triton` is installed, 2) the model is not on MPS, 3) the model is not shared between devices, and 4) the model is not resized after initialization. If `True`, then the model may be faster in some scenarios. This argument is deprecated and will be removed in a future version. tie_word_embeddings (`bool`, *optional*, defaults to `True`): Whether to tie weight embeddings Examples: ```python >>> from transformers import ModernBertModel, ModernBertConfig >>> # Initializing a ModernBert style configuration >>> configuration = ModernBertConfig() >>> # Initializing a model from the modernbert-base style configuration >>> model = ModernBertModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` modernbertpast_key_valuesgAg@)globallocalcs0|dkr|durtdd}t||dS)Nreference_compilezThe `reference_compile` argument is deprecated and will be removed in `transformers v5.2.0`Use `torch.compile()` directly on the model instead.)logger warning_oncesuper __setattr__)selfnamevalue __class__u/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/modernbert/configuration_modernbert.pyrs zModernBertConfig.__setattr__ gelu {Gz?@h㈵>FkjiNTcls vocab_size hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headshidden_activationmax_position_embeddingsinitializer_rangeinitializer_cutoff_factornorm_eps norm_bias pad_token_id eos_token_id bos_token_id cls_token_id sep_token_idattention_biasattention_dropout layer_typesrope_parameters)full_attentionsliding_attentionlocal_attentionembedding_dropoutmlp_bias mlp_dropout decoder_biasclassifier_poolingr(meanclassifier_dropoutclassifier_biasclassifier_activationdeterministic_flash_attnsparse_predictionsparse_pred_ignore_indexr tie_word_embeddingsc#$ s>| _|_| _|_|_|"_|_|_|_|_ |_ |_ |_ | _ | _| _|_|_|_|_|_|_|_|_|_|_|_|_|_|_| _|!_jdvrnt djd|_!|#"dd_#j!durfddt$j D_!t%j!j |_&t'j(di|#dS) NrFzQInvalid value for `classifier_pooling`, should be either "cls" or "mean", but is .global_attn_every_n_layersrcs"g|] }t|jr dndqSr?r>)boolrP).0irrr sz-ModernBertConfig.__init__..r))r5r7r6r8r9rNr*r0r+r,r-r.r1r2r3r4r:r;r/r@rArBrCrDrErHrIrJrKrLrMr ValueErrorr<getrPrangerr=r__init__)$rr*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r@rArBrCrDrErHrIrJrKrLrMr rNkwargsrrUrrZsZ&    zModernBertConfig.__init__cKs|dd}ddiddid}|jdur|jn||_|dur.|jd||jd||jddur=ddi|jd<|jdd|d|jd |jddur\ddi|jd<|jdd|d |jd ||j|d |S) N rope_scaling rope_typedefaultrQr>r? rope_thetaglobal_rope_thetar local_rope_thetar ) ignore_keys)popr=updaterX setdefault default_thetastandardize_rope_params validate_rope)rignore_keys_at_rope_validationr[r\default_rope_paramsrrrconvert_rope_params_to_dicts*    z,ModernBertConfig.convert_rope_params_to_dictcst}|dd|S)Nr )rto_dictrc)routputrrrrl s  zModernBertConfig.to_dictcCs |jdS)zKHalf-window size: `local_attention` is the total window, so we divide by 2.r@rUrrrsliding_windows zModernBertConfig.sliding_windowcCs|d|_dS)zs     ~