o ei@sxddlZddlmZmZddlZddlmZddlmZmZm Z ddl m Z ddl mZddlmZmZdd lmZmZdd lmZmZdd lmZdd lmZmZmZmZmZm Z dd l!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5m6Z6e,7e8Z9GdddeZ:Gdddej;ZeddedBd?e dBd@e dBfDfdAdB ZdKdCdDZfdEdFZedGdHZejdIdHZZS)LModernBertConfigaO This is the configuration class to store the configuration of a [`ModernBertModel`]. It is used to instantiate an ModernBert model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the ModernBERT-base. e.g. [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 50368): Vocabulary size of the ModernBert model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`ModernBertModel`] hidden_size (`int`, *optional*, defaults to 768): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 1152): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 22): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer decoder. hidden_activation (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the decoder. Will default to `"gelu"` if not specified. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_cutoff_factor (`float`, *optional*, defaults to 2.0): The cutoff factor for the truncated_normal_initializer for initializing all weight matrices. norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. norm_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the normalization layers. pad_token_id (`int`, *optional*, defaults to 50283): Padding token id. eos_token_id (`int`, *optional*, defaults to 50282): End of stream token id. bos_token_id (`int`, *optional*, defaults to 50281): Beginning of stream token id. cls_token_id (`int`, *optional*, defaults to 50281): Classification token id. sep_token_id (`int`, *optional*, defaults to 50282): Separation token id. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. layer_types (`list`, *optional*): Attention pattern for each layer. rope_parameters (`dict`, *optional*): Dictionary mapping attention patterns (`"full_attention"`, `"sliding_attention"`) to `RopeParameters`. Each value should be a dictionary containing `rope_type` and optional scaling parameters. local_attention (`int`, *optional*, defaults to 128): The window size for local attention. embedding_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the embeddings. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the MLP layers. mlp_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the MLP layers. decoder_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the decoder layers. classifier_pooling (`str`, *optional*, defaults to `"cls"`): The pooling method for the classifier. Should be either `"cls"` or `"mean"`. In local attention layers, the CLS token doesn't attend to all tokens on long sequences. classifier_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the classifier. classifier_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the classifier. classifier_activation (`str`, *optional*, defaults to `"gelu"`): The activation function for the classifier. deterministic_flash_attn (`bool`, *optional*, defaults to `False`): Whether to use deterministic flash attention. If `False`, inference will be faster but not deterministic. sparse_prediction (`bool`, *optional*, defaults to `False`): Whether to use sparse prediction for the masked language model instead of returning the full dense logits. sparse_pred_ignore_index (`int`, *optional*, defaults to -100): The index to ignore for the sparse prediction. reference_compile (`bool`, *optional*): Whether to compile the layers of the model which were compiled during pretraining. If `None`, then parts of the model will be compiled if 1) `triton` is installed, 2) the model is not on MPS, 3) the model is not shared between devices, and 4) the model is not resized after initialization. If `True`, then the model may be faster in some scenarios. This argument is deprecated and will be removed in a future version. tie_word_embeddings (`bool`, *optional*, defaults to `True`): Whether to tie weight embeddings Examples: ```python >>> from transformers import ModernBertModel, ModernBertConfig >>> # Initializing a ModernBert style configuration >>> configuration = ModernBertConfig() >>> # Initializing a model from the modernbert-base style configuration >>> model = ModernBertModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` modernbertpast_key_valuesgAg@)globallocalcs0|dkr|durtdd}t||dS)Nreference_compilezThe `reference_compile` argument is deprecated and will be removed in `transformers v5.2.0`Use `torch.compile()` directly on the model instead.)logger warning_oncesuper __setattr__)selfnamevalue __class__o/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/modernbert/modular_modernbert.pyr0s zModernBertConfig.__setattr__ gelu {Gz?@h㈵>FkjiNTcls vocab_size hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headshidden_activationmax_position_embeddingsinitializer_rangeinitializer_cutoff_factornorm_eps norm_bias pad_token_id eos_token_id bos_token_id cls_token_id sep_token_idattention_biasattention_dropout layer_typesrope_parametersfull_attentionsliding_attentionlocal_attentionembedding_dropoutmlp_bias mlp_dropout decoder_biasclassifier_poolingrGmeanclassifier_dropoutclassifier_biasclassifier_activationdeterministic_flash_attnsparse_predictionsparse_pred_ignore_indexr,tie_word_embeddingsc#$ s>| _|_| _|_|_|"_|_|_|_|_ |_ |_ |_ | _ | _| _|_|_|_|_|_|_|_|_|_|_|_|_|_|_| _|!_jdvrnt djd|_!|#"dd_#j!durfddt$j D_!t%j!j |_&t'j(di|#dS) NrfzQInvalid value for `classifier_pooling`, should be either "cls" or "mean", but is .global_attn_every_n_layersrcs"g|] }t|jr dndqSr_r^)boolrp).0ir1r6r7 sz-ModernBertConfig.__init__..r6))rTrVrUrWrXrnrIrOrJrKrLrMrPrQrRrSrYrZrNr`rarbrcrdrerhrirjrkrlrmr, ValueErrorr[getrpranger r\r/__init__)$r1rIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\r`rarbrcrdrerhrirjrkrlrmr,rnkwargsr4rur7rzsZ&    zModernBertConfig.__init__cKs|dd}ddiddid}|jdur|jn||_|dur.|jd||jd||jddur=ddi|jd<|jdd|d|jd |jddur\ddi|jd<|jdd|d |jd ||j|d |S) N rope_scaling rope_typedefaultrqr^r_ rope_thetaglobal_rope_thetar*local_rope_thetar+) ignore_keys)popr\updaterx setdefault default_thetastandardize_rope_params validate_rope)r1ignore_keys_at_rope_validationr{r|default_rope_paramsr6r6r7convert_rope_params_to_dicts*    z,ModernBertConfig.convert_rope_params_to_dictcst}|dd|S)Nr,)r/to_dictr)r1outputr4r6r7rs  zModernBertConfig.to_dictcCs |jdS)zKHalf-window size: `local_attention` is the total window, so we divide by 2.r#r`rur6r6r7sliding_window$s zModernBertConfig.sliding_windowcCs|d|_dS)zr?r@rAFrBrCrDrDrCFrENNrFrEFrETrGrEFr=FFrHNTN)__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencerr0intstrfloatrrlistdictrrrzrrpropertyrsetter __classcell__r6r6r4r7r'2se          !"# [  r'csNeZdZdZdeffdd Z d dejdBdejdBdejfd d Z Z S) ModernBertEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. configcsRt||_tj|j|j|jd|_tj |j|j |j d|_ t |j|_dS)N) padding_idxepsbias)r/rzrr EmbeddingrIrJrTtok_embeddings LayerNormrRrSnormDropoutradropr1rr4r6r7rz4s zModernBertEmbeddings.__init__N input_ids inputs_embedsreturncCs6|dur|||}|S||||}|Sr)rrr)r1rr hidden_statesr6r6r7forward;s zModernBertEmbeddings.forwardNN) rrrrr'rztorch LongTensorTensorrrr6r6r4r7r/srcs<eZdZdZdeffdd ZdejdejfddZZ S) ModernBertMLPa6Applies the GLU at the end of each ModernBERT layer. Compared to the default BERT architecture, this block replaces :class:`~transformers.model.bert.modeling_bert.BertIntermediate` and :class:`~transformers.model.bert.modeling_bert.SelfOutput` with a single module that has similar functionality. rcsft||_tj|jt|jd|jd|_ t |j |_ t |j|_tj|j|j|jd|_dS)Nr#r)r/rzrrLinearrJrrKrbWir rNactrrcrWorr4r6r7rzLs   zModernBertMLP.__init__rrcCs2||jddd\}}|||||S)Nr#dim)rchunkrrr)r1rinputgater6r6r7rTszModernBertMLP.forward) rrrrr'rzrrrrr6r6r4r7rEsrcsleZdZd deffdd Ze    ddedBdeddedBdedBd e d e ff fd d Z Z S)ModernBertRotaryEmbeddingNrcst||dSr)r/rz)r1rdevicer4r6r7rzZsz"ModernBertRotaryEmbedding.__init__rz torch.deviceseq_len layer_typerz torch.Tensorcst||||Sr)r/compute_default_rope_parameters)rrrrr4r6r7r]sz9ModernBertRotaryEmbedding.compute_default_rope_parametersrNNNN) rrrr'rz staticmethodrrrtuplerrrr6r6r4r7rYs$ rrotary_pos_embcCsf|j}||}||}||t||}||t||}||||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )dtype unsqueezerr&to)qkcossin unsqueeze_dimoriginal_dtypeq_embedk_embedr6r6r7apply_rotary_pos_embgs   rcseZdZdZddededBffdd Z  ddejde ejejfdBd ejdBd e e d e ejejdBff d d Z Z S)ModernBertAttentionaPerforms multi-headed self attention on a batch of unpadded sequences. If Flash Attention 2 is installed, this module uses Flash Attention to improve throughput. If Flash Attention 2 is not installed, the implementation will use PyTorch's SDPA kernel, which requires padding and unpadding inputs, adding some overhead. See `forward` method for additional details. Nr layer_idxcst||_||_|j|jdkr td|jd|jd|j|_|j|_|j|j|_ t j |jd|j |j|j d|_ |j|dkrN|jd|_nd|_d |_t j |j|j|j d|_|jd krnt |j|_dSt |_dS) NrzThe hidden size (z6) is not a multiple of the number of attention heads ()rrr_rFrE)r/rzrrrJrMrwrZrkhead_dimrrrYWqkvr[r is_causalrrIdentityout_dropr1rrr4r6r7rzs& *zModernBertAttention.__init__rposition_embeddingsattention_maskr{rcKs|jdd}||}|jg|dd|jR}|jdd\}}} |dd}|dd}| dd} |\} } t||| | dd\}}t} |jj dkrSt |jj } | |||| |f|j r`|j nd |jd |j |jd |\} }| jg|dR} ||| } | |fS) Nrrrrr#)reagerrE)dropoutscalingr deterministic)shaperviewrunbind transposerr$r_attn_implementationrtrainingrZrrkreshape contiguousrr)r1rrrr{ input_shapeqkv query_states key_states value_statesrrattention_interface attn_output attn_weightsr6r6r7rs:        zModernBertAttention.forwardrr)rrrrr'rrzrrrrrrrr6r6r4r7rs   rc sdeZdZd dededBffdd Z  ddejdejdBdejdBd ee d ejf d d Z Z S)ModernBertEncoderLayerNrrcst||_||_|dkrt|_n tj|j|j |j d|_t ||d|_ tj|j|j |j d|_ t||_|j||_dS)Nrr)rr)r/rzrrrr attn_normrrJrRrSrattnmlp_normrmlpr[attention_typerr4r6r7rzs   zModernBertEncoderLayer.__init__rrrr{rcKsB|j||f||d|\}}||}||||}|S)N)rr)rrrr)r1rrrr{r_r6r6r7rs zModernBertEncoderLayer.forwardrr) rrrr'rrzrrrrrrr6r6r4r7rsrc@sXeZdZUeed<dZdZddgZdZdZ dZ dZ e e dZedejfdd Zd S) ModernBertPreTrainedModelrmodelTrr)r attentionsmodulecs|jjdur ddtjdtffdd }|jj|jjtd|jj|jj|jj dd}t |t r?||j |d dSt |t rV||j|d ||j|d dSt |trm||j|d ||j|d dSt |tr|||j|d dSt |tr||j|d dSt |ttttfr||j|d dSt |tjrt|j|jdurt |jdSdSt |t!r|j"D]6}|j#}|j$|d krt%|j$|}||j|d\}}t&t'||d|t&t'||d|qdSdS)NrrstdcsNtj|jd| ||dt|tjr#|jdur%t|jdSdSdS)NrE)rgrab)init trunc_normal_weight isinstancerrrzeros_)rr cutoff_factorr6r7 init_weight s  z.init_weightr@r)inout embedding final_outrrrrr~)r _inv_freq_original_inv_freq)(rrQrModulerrPmathsqrtrLrJr rrrrrrrModernBertPredictionHeaddenseModernBertForMaskedLMdecoder#ModernBertForSequenceClassificationModernBertForMultipleChoice ModernBertForTokenClassificationModernBertForQuestionAnswering classifierrrones_r rr rr[rr}rcopy_getattr)r1rrstdsr rope_init_fn curr_inv_freqrr6r r7 _init_weightssZ           z'ModernBertPreTrainedModel._init_weightsN)rrrr'__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrr_can_record_outputsrno_gradrrr'r6r6r6r7rs rcseZdZdeffdd ZddZddZeee    dd e j dBd e j dBd e j dBd e j dBd e edef ddZZS)ModernBertModelrcsrt|_t|_tfddtjD|_ tj j j j d|_td|_d|_|dS)Ncsg|]}t|qSr6)r)rsrrr6r7rvIsz,ModernBertModel.__init__..rr3F)r/rzrr embeddingsr ModuleListryrLlayersrrJrRrS final_normr rotary_embgradient_checkpointing post_initrr4r3r7rzDs    zModernBertModel.__init__cCs|jjSrr4rrur6r6r7get_input_embeddingsPsz$ModernBertModel.get_input_embeddingscCs ||j_dSrr;rr6r6r7set_input_embeddingsSs z$ModernBertModel.set_input_embeddingsNrr position_idsrr{rcKs|du|duAr td|dur|jdn|jd}|dur!|jn|j}|dur2tj||dd}|j||d}t|} tsV|j ||d} t d i| t d i| d} i} |j j D] } | ||| | | <q\|jD]} | |f| | j| | jd|}qk||}t|d S) Nz:You must specify exactly one of input_ids or inputs_embedsrrr)rr)rrrr])rr)last_hidden_stater6)rwrrrarangerr4r rrrrr[r8r6rr7r)r1rrr>rr{rrrattention_mask_mapping mask_kwargsrr encoder_layerr6r6r7rVs:       zModernBertModel.forwardr)rrrr'rzr<r=r!r"rrrrrrrrrr6r6r4r7r2Bs0 r2cs8eZdZdeffdd ZdejdejfddZZS)rrcsNt||_t|j|j|j|_t|j |_ tj |j|j |j d|_dS)Nr)r/rzrrrrJrirr rjrrrRrSrrr4r6r7rzs  z!ModernBertPredictionHead.__init__rrcCs||||Sr)rrr)r1rr6r6r7rsz ModernBertPredictionHead.forward) rrrr'rzrrrrr6r6r4r7rsrzd The ModernBert Model with a decoder head on top that is used for masked language modeling. ) custom_introcseZdZddiZdeffdd ZddZdejfd d Z e e dd e j d Bd e jd Bde jd Bde jd Bde jd Bdeedee jeBfddZZS)rzdecoder.weightz&model.embeddings.tok_embeddings.weightrcs^t|||_t||_t||_tj|j |j |j d|_ |jj |_ |jj|_|dS)Nr)r/rzrr2rrheadrrrJrIrdrrlrmr:rr4r6r7rzs      zModernBertForMaskedLM.__init__cCs|jSrrrur6r6r7get_output_embeddingssz+ModernBertForMaskedLM.get_output_embeddingsnew_embeddingscCs ||_dSrrG)r1rIr6r6r7set_output_embeddingss z+ModernBertForMaskedLM.set_output_embeddingsNrrr>rlabelsr{rc Ks|jd||||d|}|d}|jr3|dur3|d}||jdd}||jk} || }|| }|||} d} |durO|j| |fd|jj i|} t | | |j |j dS)Nrrr>rrrrIlosslogitsrrr6) rrlrrrmrrF loss_functionrrIrrr) r1rrr>rrKr{outputsr@ mask_tokensrOrNr6r6r7rs2   zModernBertForMaskedLM.forwardNNNNN)rrr_tied_weights_keysr'rzrHrrrJr rrrrrrrrrrr6r6r4r7rs6  rz` The ModernBert Model with a sequence classification head on top that performs pooling. ceZdZdeffdd Zee     ddejdBdej dBdej dBdej dBd ej dBd e e d e ej e Bfd d ZZS)rrcs\t||j|_||_t||_t||_tj |j |_ t |j|j|_|dSr)r/rz num_labelsrr2rrrFrrrrhrrrJr r:rr4r6r7rzs    z,ModernBertForSequenceClassification.__init__Nrrr>rrKr{rc Ks|jd||||d|}|d}|jjdkr |dddf}n+|jjdkrK|dur9tj|jdd|jtjd}||dj d d |j d d d }| |} | | } | | } d} |dur|jj dur|jd krpd |j_ n|jd kr|jtjks|jtjkrd|j_ nd|j_ |jj d krt} |jd kr| | |} n+| | |} n%|jj dkrt} | | d|j|d} n|jj dkrt} | | |} t| | |j|jdS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rLrrGNrgr#)rrrrrTrkeepdim regressionsingle_label_classificationmulti_label_classificationrMr6)rrreronesrrrrrsumrFrr  problem_typerVrlongrrsqueezerrrrrr) r1rrr>rrKr{rQr@ pooled_outputrOrNloss_fctr6r6r7rs^        "       z+ModernBertForSequenceClassification.forwardrS)rrrr'rzr rrrrrrrrrrr6r6r4r7rs0  rzv The ModernBert Model with a token classification head on top, e.g. for Named Entity Recognition (NER) tasks. crU)rrcVt||j|_t||_t||_tj |j |_ t |j |j|_|dSrr/rzrVr2rrrFrrrrhrrrJr r:rr4r6r7rz<s    z)ModernBertForTokenClassification.__init__Nrrr>rrKr{rc Ks|jd||||d|}|d}||}||}||} d} |dur6t} | | d|j|d} t| | |j|j dS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. rLrNrrMr6) rrFrr rrrVrrr) r1rrr>rrKr{rQr@rOrNrbr6r6r7rHs,   z(ModernBertForTokenClassification.forwardrS)rrrr'rzr rrrrrrrrrrr6r6r4r7r60  rcseZdZdeffdd Zee     ddejdBdejdBdejdBdejdBd ejdBd e e d e eje Bfd d Z ZS)rrcrcrrdrr4r6r7rzss    z'ModernBertForQuestionAnswering.__init__Nrrr>start_positions end_positionsr{rc Ks|j|f||d|}|d}||}||}||} | jddd\} } | d} | d} d} |durL|durL|j| | ||fi|} t| | | |j |j dS)N)rr>rrrr)rN start_logits end_logitsrr) rrFrr splitr`rrPrrr) r1rrr>rfrgr{rQr@rOrhrirNr6r6r7r~s2    z&ModernBertForQuestionAnswering.forwardrS)rrrr'rzr rrrrrrrrrr6r6r4r7rqs0  rz The ModernBert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. crU)rrcsRt|||_t||_t||_tj |j |_ t |j d|_|dS)Nr)r/rzrr2rrrFrrrrhrrrJr r:rr4r6r7rzs    z$ModernBertForMultipleChoice.__init__Nrrr>rrKr{rcKs|dur |jdn|jd}|dur|d|dnd}|dur*|d|dnd}|dur9|d|dnd}|durL|d|d|dnd}|jd||||d|}|d} |jjdkrtj| jd| jd} |dur|j dd  | j} n tj dtj | jd } | | | f} n|jjd kr|j dd d } | |dj dd | } || } || } || }|d|}d}|durt}|||}t|||j|jdS)a& labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. NrrrLrrGr?r)rrrgTrWrMr6)rrsizerrrerrArargmaxrtensorr_r]rrFrr rrrrr)r1rrr>rrKr{ num_choicesrQr@ indices_0cls_masknum_non_pad_tokensrarOreshaped_logitsrNrbr6r6r7rsP       z#ModernBertForMultipleChoice.forwardrS)rrrr'rzr rrrrrrrrrrr6r6r4r7rrer)r'r2rrrrrr)r)Krtypingrrrrtorch.nnrrrr r activationsr configuration_utilsr r integrationsr r masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericr r!utils.output_capturingr"align.modeling_alignr$gemma3.modeling_gemma3r%r& get_loggerrr-r'rrrrrrrrr2rrrrrr__all__r6r6r6r7sn         ~ Q ME BV64 U