o wi@sddlZddlmZddlmZmZmZddlZddlm m Z ddl Zddlm Z ddl mZmZmZddlmZddlmZdd lmZdd lmZdd lmZmZmZmZmZdd lm Z dd l!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)e#rddl*m+Z+ddl,m-Z-ddl.m/Z/ne0Z-e$1e2Z3GdddeZ4  dXdej5dej5deej5deej5de6ej5ej5ej5e7eej5eej5ff ddZ8dej5dej5de7de7dej5f d d!Z9Gd"d#d#ej:j;Z<  dXd$eej5d%ee7fd&d'Z=Gd(d)d)e-Z>Gd*d+d+e j?Z@Gd,d-d-e j?ZAGd.d/d/e(ZB 0dYd1d2d3ej5dej5d4ej5deejCd5e6e7e7fd6e7d7e7d8eeDdee6ej5ej5fe6ej5ffd9d:ZEejFfd1d2d3ej5d;e>d$ej5d%e7d5e6e7e7fd6e7d7e7dZHd1d2d3ej5dej5d4ej5deejCd5e6e7e7fd6e7d7e7de6ej5fd?d@ZIeHeEeIdAZJGdBd2d2e j?ZKGdCdDdDeZLe"GdEdFdFe ZMe"GdGdHdHeMZNGdIdJdJe j?ZOe"dKdLGdMdNdNeMZPe"dOdLGdPdQdQeMZQe"dRdLGdSdTdTeMZRe"GdUdVdVeMZSgdWZTdS)ZN) nullcontext)LiteralOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)PretrainedConfig)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)auto_docstringis_flash_attn_2_availablelogging)is_triton_available)GemmaRotaryEmbeddingapply_rotary_pos_emb) flash_attn_varlen_qkvpacked_func)RotaryEmbedding) apply_rotarycseZdZdZdZdgZ                             d!dedffdd Zfdd ZZ S)"ModernBertConfiga This is the configuration class to store the configuration of a [`ModernBertModel`]. It is used to instantiate an ModernBert model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the ModernBERT-base. e.g. [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 50368): Vocabulary size of the ModernBert model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`ModernBertModel`] hidden_size (`int`, *optional*, defaults to 768): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 1152): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 22): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer decoder. hidden_activation (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the decoder. Will default to `"gelu"` if not specified. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_cutoff_factor (`float`, *optional*, defaults to 2.0): The cutoff factor for the truncated_normal_initializer for initializing all weight matrices. norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. norm_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the normalization layers. pad_token_id (`int`, *optional*, defaults to 50283): Padding token id. eos_token_id (`int`, *optional*, defaults to 50282): End of stream token id. bos_token_id (`int`, *optional*, defaults to 50281): Beginning of stream token id. cls_token_id (`int`, *optional*, defaults to 50281): Classification token id. sep_token_id (`int`, *optional*, defaults to 50282): Separation token id. global_rope_theta (`float`, *optional*, defaults to 160000.0): The base period of the global RoPE embeddings. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. global_attn_every_n_layers (`int`, *optional*, defaults to 3): The number of layers between global attention layers. local_attention (`int`, *optional*, defaults to 128): The window size for local attention. local_rope_theta (`float`, *optional*, defaults to 10000.0): The base period of the local RoPE embeddings. embedding_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the embeddings. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the MLP layers. mlp_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the MLP layers. decoder_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the decoder layers. classifier_pooling (`str`, *optional*, defaults to `"cls"`): The pooling method for the classifier. Should be either `"cls"` or `"mean"`. In local attention layers, the CLS token doesn't attend to all tokens on long sequences. classifier_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the classifier. classifier_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the classifier. classifier_activation (`str`, *optional*, defaults to `"gelu"`): The activation function for the classifier. deterministic_flash_attn (`bool`, *optional*, defaults to `False`): Whether to use deterministic flash attention. If `False`, inference will be faster but not deterministic. sparse_prediction (`bool`, *optional*, defaults to `False`): Whether to use sparse prediction for the masked language model instead of returning the full dense logits. sparse_pred_ignore_index (`int`, *optional*, defaults to -100): The index to ignore for the sparse prediction. reference_compile (`bool`, *optional*): Whether to compile the layers of the model which were compiled during pretraining. If `None`, then parts of the model will be compiled if 1) `triton` is installed, 2) the model is not on MPS, 3) the model is not shared between devices, and 4) the model is not resized after initialization. If `True`, then the model may be faster in some scenarios. repad_logits_with_grad (`bool`, *optional*, defaults to `False`): When True, ModernBertForMaskedLM keeps track of the logits' gradient when repadding for output. This only applies when using Flash Attention 2 with passed labels. Otherwise output logits always have a gradient. Examples: ```python >>> from transformers import ModernBertModel, ModernBertConfig >>> # Initializing a ModernBert style configuration >>> configuration = ModernBertConfig() >>> # Initializing a model from the modernbert-base style configuration >>> model = ModernBertModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` modernbertpast_key_values gelu {Gz?@h㈵>FkjiAr @TclsNclassifier_poolingr3meanc$% stjd| || ||d|$||_||_||_||_||_||_||_| |_ | |_ | |_ ||_ ||_ ||_||_||_||_||_||_||_||_||_||_||_||_||_||_| |_|!|_|"|_|#|_|jdvrwt d|jddS)N) pad_token_id bos_token_id eos_token_id cls_token_id sep_token_idr6zQInvalid value for `classifier_pooling`, should be either "cls" or "mean", but is .)!super__init__ vocab_sizemax_position_embeddings hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headsinitializer_rangeinitializer_cutoff_factornorm_eps norm_biasglobal_rope_thetaattention_biasattention_dropouthidden_activationglobal_attn_every_n_layerslocal_attentionlocal_rope_thetaembedding_dropoutmlp_bias mlp_dropout decoder_biasr5classifier_dropoutclassifier_biasclassifier_activationdeterministic_flash_attnsparse_predictionsparse_pred_ignore_indexreference_compilerepad_logits_with_grad ValueError)%selfrArCrDrErFrNrBrGrHrIrJr8r:r9r;r<rKrLrMrOrPrQrRrSrTrUr5rVrWrXrYrZr[r\r]kwargs __class__r>n/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/modernbert/modular_modernbert.pyr@sX'  zModernBertConfig.__init__cst}|dd|S)Nr\)r?to_dictpopr_outputrar>rcrds  zModernBertConfig.to_dict)#r"r#r$r%r&r'r(r)r*r+Fr,r-r.r.r-r/Fr0r r1r2r0Fr0Tr3r0Fr'FFr4NF) __name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencerr@rd __classcell__r>r>rarcr6sVgSrinputsattention_mask position_idslabelsreturncCs|jdtjd}tj|dd}t|}tjj tj |dtjdd}| dkr7||}n|j ^} } } | | } |j| g| R|}|durV||nd} |durb||nd}||||| |fS) a Remove padding from input sequences. Args: inputs: (batch, seqlen, ...) or (batch, seqlen) attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid. position_ids: (batch, seqlen), int, position ids labels: (batch, seqlen), int, labels Returns: unpadded_inputs: (total_nnz, ...), where total_nnz = number of tokens selected in attention_mask. indices: (total_nnz) cu_seqlens: (batch + 1), the cumulative sequence lengths max_seqlen_in_batch: int unpadded_position_ids: (total_nnz) or None unpadded_labels: (total_nnz) or None dimdtypeF)as_tupler)rrN)sumtorchint32nonzeroflattenintmaxitemr functionalpadcumsumrvshapeview)rorprqrrseqlens_in_batchindicesmax_seqlen_in_batch cu_seqlensunpadded_inputsbatchseqlenrestrunpadded_position_idsunpadded_labelsr>r>rc_unpad_modernbert_inputs  rrrrcCs|dkrtj|||j|jd}|||<|||}|S|j^}}tj||g|R|j|jd}|||<|j||g|R}|S)aQ Add padding to sequences. Args: inputs: (total_nnz, ...) or (total_nnz,), where total_nnz = number of tokens selected in attention_mask. indices: (total_nnz) batch: int, batch size seqlen: int, max sequence length Returns: padded_inputs: (batch, seqlen, ...) or (batch, seqlen) ry)rwdevice)rvr{zerosrwrrr)rorrrrg padded_inputs_rr>r>rc_pad_modernbert_output#s   "rc@s>eZdZe  ddeejdeefddZeddZ dS) ApplyRotaryEmbUnpadNr max_seqlenc Csd|}|j\}}}} |ddddf|d| } t| ||d||ddd||||||_|S)NrrtrFT)seqlen_offsetsrr interleavedinplace) contiguousrrrsave_for_backwardr) ctxqkvcossinrr total_nnz_three_nheadsheaddimqkr>r>rcforwardCs   zApplyRotaryEmbUnpad.forwardc Csn|j\}}}|}|j\}}}}|ddddf|d|} t| ||d||jdddd |ddddddfS)NrrtrFT)rrrrr conjugate) saved_tensorsrrrrr) rdorrrrrrrdqkr>r>rcbackwardbs  zApplyRotaryEmbUnpad.backwardNN) rhrirj staticmethodrr{Tensorrrrr>r>r>rcrBs rrrcCst|||||S)a Arguments: qkv: (total_nnz, 3, nheads, headdim) - input tensor for packed QKV. cos, sin: (seqlen_rotary, rotary_dim / 2) interleaved: if True, rotate pairs of even and odd dimensions (GPT-J style) instead of 1st half and 2nd half (GPT-NeoX style). inplace: if True, apply rotary embedding in-place. seqlen_offsets: (batch_size,) or int. Each sequence in x is shifted by this amount. Most commonly used in inference when we have KV cache. cu_seqlens: (batch + 1,) or None max_seqlen: int Return: out: (total_nnz, dim) rotary_dim must be <= headdim Apply rotary embedding to the first rotary_dim of x. )rapply)rrrrrr>r>rcapply_rotary_unpaddedysrc seZdZdZ    ddededeedeejdeej f fd d Z dd ej d ej deed e ej e ej ej fffddZd efddZZS)!ModernBertUnpaddedRotaryEmbeddingzP The rotary position embeddings applied directly to unpadded sequences. r2NrvbaserrrwcsRtj|||dd||_|dur#|dur%|dur'|j|||ddSdSdSdS)a max_seqlen: if max_seqlen, device, and dtype are provided, we precompute the cos_sin_cache up to max_seqlen. If the max_seqlen, device, or dtype during training/inference differ, the cos_sin_cache will be recomputed during the forward pass. F)rvrrrNrrw)r?r@r_update_cos_sin_cache)r_rvrrrrwrar>rcr@s   z*ModernBertUnpaddedRotaryEmbedding.__init__rrrscCs6|dur|j||j|jdt||j|j||d}|S)z Apply rotary embedding *inplace* to qkv. qkv: (total_nnz, 3, nheads, headdim) cu_seqlens: (batch + 1,) cumulative sequence lengths max_seqlen: int max seq length in the batch Nrrr)rrrwr _cos_cached _sin_cached)r_rrrr>r>rcrs z)ModernBertUnpaddedRotaryEmbedding.forwardcCsd|jd|jd|jS)Nzdim=z, base=z , scale_base=)rvr scale_baser_r>r>rc extra_reprsz,ModernBertUnpaddedRotaryEmbedding.extra_repr)r2NNNN)rhrirjrkrfloatrr{rrwr@rrtuplerstrrrnr>r>rarcrs8 rcspeZdZdZdeffdd Zejdddejdej fd d Z dde ejd e ej dej fd dZ Z S)ModernBertEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. configcsRt||_tj|j|j|jd|_tj |j|j |j d|_ t |j|_dS)N) padding_idxepsbias)r?r@rr EmbeddingrArCr8tok_embeddings LayerNormrIrJnormDropoutrRdropr_rrar>rcr@s zModernBertEmbeddings.__init__Tdynamic input_idsrscC||||Sr)rrr)r_rr>r>rccompiled_embeddingssz(ModernBertEmbeddings.compiled_embeddingsN inputs_embedscCsH|dur|||}|S|jjr||n ||||}|Sr)rrrr\rr)r_rr hidden_statesr>r>rcrs zModernBertEmbeddings.forwardr)rhrirjrkrr@r{compile LongTensorrrrrrnr>r>rarcrs rcs<eZdZdZdeffdd ZdejdejfddZZ S) ModernBertMLPa6Applies the GLU at the end of each ModernBERT layer. Compared to the default BERT architecture, this block replaces :class:`~transformers.model.bert.modeling_bert.BertIntermediate` and :class:`~transformers.model.bert.modeling_bert.SelfOutput` with a single module that has similar functionality. rcsft||_tj|jt|jd|jd|_ t |j |_ t |j|_tj|j|j|jd|_dS)Nrr)r?r@rrLinearrCrrDrSWir rNactrrTrWorrar>rcr@s   zModernBertMLP.__init__rrscCs2||jddd\}}|||||S)Nrrtrv)rchunkrrr)r_rinputgater>r>rcrszModernBertMLP.forward) rhrirjrkrr@r{rrrnr>r>rarcrsrc s6eZdZddedededeejffdd Z Z S) ModernBertRotaryEmbeddingNrrvrrcs.tj|||d|jd|||d\}|_dS)N)rr)rvr)r?r@ rope_init_fnattention_scaling)r_rrvrrinv_freqrar>rcr@sz"ModernBertRotaryEmbedding.__init__r) rhrirjrrrrr{rr@rnr>r>rarcrs.rFmoduleModernBertAttentionrsliding_window_maskrPbsrvoutput_attentionsc Ks|j||d\} } |ddjdd\} } }t| | | | \} } |jd}t| | dd|}|dkr6|}||}tjj |dtj d  | j }tjj ||j|jd }t||}|dd}||d|}|rn||fS|fS) Nrqr ryrr࿩rtrtrtru)ptraining) rotary_emb transposeunbindrhead_dimr{matmulrrsoftmaxfloat32torwdropoutrMrrr)rrrprrqrPrrvr_kwargsrrquerykeyvaluescale attn_weights attn_outputr>r>rceager_attention_forwards    rr target_dtypec Ks||||d}|jtjtjfv} | r1|j} ||}t||||jr$|jnd|j|d} | | } nt||||jr;|jnd|j|d} | ||fS)Nrr0)rr dropout_p deterministic window_size) rwr{float16bfloat16rrrrMrYr) rrrrrrPrrvrr convert_dtype orig_dtypeattnr>r>rcflash_attention_forward%s.   rcKs|j||d\} } |ddjdd\} } } t| | | | \} } |dkr%|}tj| | | |jr0|jnd|ddd}| |d |}|fS) Nrr ryrrrr0)r attn_maskrt) rrrrFscaled_dot_product_attentionrrMrr)rrrprrqrPrrvrrrrrrrr>r>rcsdpa_attention_forwardPs"  r )flash_attention_2eagersdpacsReZdZdZd dedeeffdd Z ddej d ee d ej fd d Z Z S)raPerforms multi-headed self attention on a batch of unpadded sequences. If Flash Attention 2 is installed, this module uses Flash Attention to improve throughput. If Flash Attention 2 is not installed, the implementation will use PyTorch's SDPA kernel, which requires padding and unpadding inputs, adding some overhead. See `forward` method for additional details. Nrlayer_idcs\t||_||_|j|jdkr td|jd|jd|j|_|j|_|j|_ |j|j|_ |j |j |_ t j |jd|j |jd|_||jdkr[|jd|jdf|_nd|_|j}|j}|jdkrt|jdurq|j}|j}|jd krt|j ||d |_n t||j |d |_t j |j|j|jd|_|jd krt |jnt |_t|_dS) NrzThe hidden size (z6) is not a multiple of the number of attention heads ()r rrrr )rvrr)rrvrr0)r?r@rrrCrFr^rMrY num_headsr all_head_sizerrrLWqkvrOrPrKrBrQ_attn_implementationrrrrrIdentityout_dropset pruned_heads)r_rr rope_thetarBrar>rcr@s<       zModernBertAttention.__init__Frrrsc Ks||}|jd}|jjdkr|dd|j|j}n ||dd|j|j}t|jj|f||j|j ||j |d|}|d}| | |}|f|ddS)Nrr rtr )rrrPrrvrry) rrrrrrrMODERNBERT_ATTENTION_FUNCTIONrrPrrr)r_rrr`rr attn_outputsr>r>rcrs(     zModernBertAttention.forwardrF) rhrirjrkrrrr@r{rboolrrnr>r>rarcrzs +cseZdZddedeeffdd Zejdddej d ej fd d Z      ddej d eej deej deej deej deedee d ej fddZ ZS)ModernBertEncoderLayerNrrcspt||_|dkrt|_n tj|j|j|j d|_t ||d|_ tj|j|j|j d|_ t ||_dS)Nrr)rr)r?r@rrr attn_normrrCrIrJrrmlp_normrmlp)r_rrrar>rcr@s  zModernBertEncoderLayer.__init__TrrrscC|||Sr)r"r!r_rr>r>rc compiled_mlpz#ModernBertEncoderLayer.compiled_mlpFrprrqrrrc Csf|j||||||||d}||d}|jjr||n|||} || }|f|ddS)Nrprrqrrrrry)rr rr\r%r"r!) r_rrprrqrrrr mlp_outputr>r>rcrs    zModernBertEncoderLayer.forwardr)NNNNNF)rhrirjrrrr@r{rrr%rrrrnr>r>rarcrs6  rc seZdZeZdZdZddgZdZdZ dZ de j fddZ e dd eejd eeeeeeffd effd d ZddZfddZZS)ModernBertPreTrainedModelmodelTrrFrcsv|jjdur ddtjdtffdd }|jj|jjtd|jj|jj|jj dd}t |t r?||j |d dSt |t rV||j|d ||j|d dSt |trm||j|d ||j|d dSt |tr|||j|d dSt |tr||j|d dSt |tttfr||j|d dSt |tjr|jjd |jdur|jjdSdSdS)Nr rstdcsRtjj|jd| ||dt|tjr%|jdur'tj|jdSdSdS)Nr0)r7r+ab)rinit trunc_normal_weight isinstancerrzeros_)rr+ cutoff_factorr>rc init_weights  z.init_weightr*r)inout embedding final_outr8r6r7r9g?) rrHrModulerrGmathsqrtrErCr1rrrrrrrModernBertPredictionHeaddenseModernBertForMaskedLMdecoder#ModernBertForSequenceClassification ModernBertForTokenClassificationModernBertForQuestionAnswering classifierrr0datafill_rzero_)r_rr5stdsr>r3rc _init_weightss@        z'ModernBertPreTrainedModel._init_weightsN torch_dtype device_mapcheck_device_mapc sZ|jdur#d|_z |j|tj|d|dWSttfy"d|_Ynwtj||||dS)Nr F)rJrKhard_check_onlyrL)rJrKrL)_attn_implementation_internal_check_and_enable_flash_attn_2r{rr^ ImportErrorr?_autoset_attn_implementation)r3rrJrKrLrar>rcrQ1s&  z6ModernBertPreTrainedModel._autoset_attn_implementationcCs|jjdurdSt|dr!t|jdkr!|jjrtdd|j_|jjdkr4|jjr0tdd|j_|jjdkrG|jjrCtdd|j_|jjdurTt |j_dSdS) NF hf_device_mapryzqIf `accelerate` split the model across devices, `torch.compile` will not work. Falling back to non-compiled mode.mpsz|Compiling the model with `torch.compile` and using a `torch.mps` device is not supported. Falling back to non-compiled mode.cpuz|Compiling the model with `torch.compile` and using a `torch.cpu` device is not supported. Falling back to non-compiled mode.) rr\hasattrlenrRlogger warning_oncertyperrr>r>rc_maybe_set_compilePs.    z,ModernBertPreTrainedModel._maybe_set_compilecs<tj|i|}|jjdvr|jjrtdd|j_|S)N>NTzcResizing token embeddings with `torch.compile` is not supported. Falling back to non-compiled mode.F)r?resize_token_embeddingsrr\rWrX)r_argsr` model_embedsrar>rcr[os z1ModernBertPreTrainedModel.resize_token_embeddings)NNT)rhrirjr config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_2_supports_sdpa_supports_flex_attnrr:rI classmethodrr{rwrrdictrrrQrZr[rnr>r>rarcr)s,/r)c!seZdZdeffdd ZddZddZe             dd ee j d ee j d ee j d ee j d ee j dee j dee j dee dee dee dee dee dee deee j dfeffddZd e j de de j fddZZS)ModernBertModelrcsft|_t|_tfddtjD|_ tj j j j d|_d|_|dS)Ncsg|]}t|qSr>)r).0rrr>rc sz,ModernBertModel.__init__..rF)r?r@rr embeddingsr ModuleListrangerElayersrrCrIrJ final_normgradient_checkpointing post_initrrarircr@~s   zModernBertModel.__init__cCs|jjSrrkrrr>r>rcget_input_embeddingssz$ModernBertModel.get_input_embeddingscCs ||j_dSrrr)r_rr>r>rcset_input_embeddingss z$ModernBertModel.set_input_embeddingsNrrprrqrrrr batch_sizeseq_lenroutput_hidden_states return_dictrs.c s| dur| n|jj} | dur| n|jj} | dur| n|jj} |du|duAr*td| r.dnd}| r4dnd}||durD|||durcdurc|durZ|jdd\n |jdd\|durj|jn|j}|dur|t j f|t j d}d}|jj dkrˆdur|dur|durd}|durt t||d ^}}}}Wdn1swYn#t||d ^}}}}n|durt j|d d }|j|| d \}}|j||d }|jD])}| r||f}|||||||| d}|d }| r t|dkr ||df}q| r||f}||}|r7t|d}|dur7tfdd|D}| sFtdd|||fDSt|||dS) sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nz:You must specify exactly one of input_ids or inputs_embedsr>rrFr T)rorp)rr)r)rrr'ryrorrrc3s |] }t|dVqdS)rzN)r)rhhsrurrvr>rc s  z*ModernBertModel.forward..css|] }|dur|VqdSrr>)rhvr>r>rcr}s)last_hidden_stater attentions)rrrwuse_return_dictr^rZ%warn_if_padding_and_no_attention_maskrrr{onesrrno_gradrarange unsqueeze_update_attention_maskrkrnrVrorrr)r_rrprrqrrrrrurvrrwrxall_hidden_statesall_self_attentionsrrepadrr encoder_layer layer_outputsr>r|rcrs!            zModernBertModel.forwardcCs|r#|jjdkrtdd|j_n|jjdkr#td|jjdt||j}t|jd d}t ||j }||jj dk d d |j}||t|jj}||fS)NrzOutputting attentions is only supported with the 'eager' attention implementation, not with "sdpa". Falling back to `attn_implementation="eager"`.rzZOutputting attentions is only supported with the eager attention implementation, not with zT. Consider setting `attn_implementation="eager"`. Setting `output_attentions=False`.rr)rrrWrXr rwr{rrrabsTrPrr masked_fill logical_notfinfomin)r_rprglobal_attention_maskrowsdistance window_maskrr>r>rcr s&    "z&ModernBertModel._update_attention_mask NNNNNNNNNNNNN)rhrirjrr@rsrtrrr{rrrrrrrrrrnr>r>rarcrg|s`       "zrgcs8eZdZdeffdd ZdejdejfddZZS)r=rcsNt||_t|j|j|j|_t|j |_ tj |j|j |j d|_dS)Nr)r?r@rrrrCrWr>r rXrrrIrJrrrar>rcr@+s  z!ModernBertPredictionHead.__init__rrscCrr)rrr>r$r>r>rcr2sz ModernBertPredictionHead.forward) rhrirjrr@r{rrrnr>r>rarcr=*sr=zd The ModernBert Model with a decoder head on top that is used for masked language modeling. ) custom_introc"s eZdZdgZdeffdd ZddZdejfdd Z e j d d d e j d e j fddZ e              d!dee jdee j dee j dee j dee j dee j dee j dee j deedeedeedeedeedeed eee j effdd ZZS)"r?zdecoder.weightrcs^t|||_t||_t||_tj|j |j |j d|_ |jj |_ |jj|_|dS)Nr)r?r@rrgr*r=headrrrCrArUr@rZr[rqrrar>rcr@>s      zModernBertForMaskedLM.__init__cCs|jSrr@rr>r>rcget_output_embeddingsKsz+ModernBertForMaskedLM.get_output_embeddingsnew_embeddingscCs ||_dSrr)r_rr>r>rcset_output_embeddingsNs z+ModernBertForMaskedLM.set_output_embeddingsTrrgrscCr#r)r@rrfr>r>rc compiled_headQr&z#ModernBertForMaskedLM.compiled_headNrrprrqrrrrrrrurvrrwrxcKsj|dur|n|jj}||jjdkr|dur|dur| dur| dur?| dur?|dur6|jdd\} } n |jdd\} } |durF|jn|j}|durXtj| | f|tjd}|durt t ||||d\}}}} }}Wdn1sywYnt ||||d\}}}} }}|j |||||||| | | | | |d }|d}|j r|dur| d}| |jdd}||jk}||}||}|jjr||n|||}d}|dur|j||fd |jji|}|jjdkr|jjs|durtnt t||| | d }Wdn 1swY|s+|f}|dur)|f|S|St|||j|jd S) ryNr rr)rorprqrr rrprrqrrrrrurvrrwrxrrtrArzlosslogitsrr)rrrZrrrr{rrrrr*rZrr[r\rr@r loss_functionrAr]rrrrr)r_rrprrqrrrrrrrurvrrwrxr`routputsr mask_tokensrrrgr>r>rcrUs~#      zModernBertForMaskedLM.forwardNNNNNNNNNNNNNN)rhrirj_tied_weights_keysrr@rrrrr{rrrrrrrrrrrrrnr>r>rarcr?6sj       r?z` The ModernBert Model with a sequence classification head on top that performs pooling. c"eZdZdeffdd Ze              ddeejdeej deej deej d eej d eej d eej d eej d ee dee dee dee dee dee de e ej effddZZS)rArcs\t||j|_||_t||_t||_tj |j |_ t |j|j|_|dSr)r?r@ num_labelsrrgr*r=rr{rrrVrrrCrDrqrrar>rcr@s    z,ModernBertForSequenceClassification.__init__NrrprrqrrrrrrrurvrrwrxrscKs|dur|n|jj}||j|||||||| | | | | |d }|d}|jjdkr3|dddf}n|jjdkrK||djdd|jdd d }||}||}| |}d}|dur|jj dur|j dkrpd |j_ n|j dkr|j t jks|j t jkrd |j_ nd |j_ |jj d krt}|j dkr|||}n+|||}n%|jj d krt}||d|j |d}n|jj d krt}|||}|s|f}|dur|f|S|St|||j|jdS)aB sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nrrr3r7rtryrT)rvkeepdim regressionsingle_label_classificationmulti_label_classificationr)rrrZr*r5rrzrrrD problem_typerrwr{longrr squeezerrrrrr)r_rrprrqrrrrrrrurvrrwrxr`rr pooled_outputrrloss_fctrgr>r>rcrsn'        "       z+ModernBertForSequenceClassification.forwardr)rhrirjrr@rrr{rrrrrrrrrnr>r>rarcrAs`      rAzv The ModernBert Model with a token classification head on top, e.g. for Named Entity Recognition (NER) tasks. c"r)rBrcVt||j|_t||_t||_tj |j |_ t |j |j|_|dSrr?r@rrgr*r=rr{rrrVrrrCrDrqrrar>rcr@Hs    z)ModernBertForTokenClassification.__init__NrrprrqrrrrrrrurvrrwrxrscCs|dur|n|jj}||j|||||||| | | | | |d }|d}||}||}||}d}|durIt}||d|j |d}|s_|f|dd}|dur]|f|S|St |||j |j dS)a sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nrrrtryr) rrrZr*rrrDrrrrrr)r_rrprrqrrrrrrrurvrrwrxrrrrrrgr>r>rcrTsD$   z(ModernBertForTokenClassification.forwardr)rhrirjrr@rrr{rrrrrrrrrnr>r>rarcrBBs`      rBc"seZdZdeffdd Ze             ddeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee dee dee dee de e eje ffddZZS)rCrcrrrrrar>rcr@s    z'ModernBertForQuestionAnswering.__init__Nrrprrqstart_positions end_positionsrrrrurvrrwrxrscKs|dur|n|jj}||j||||||| | | | | |d }|d}||}||}||}|jddd\}}|d }|d }d}|dur_|dur_|j ||||fi|}|sv||f|dd}|durt|f|S|St ||||j |j dS)ryN) rprrqrrrrurvrrwrxrryrtr)r start_logits end_logitsrr)rrrZr*rrrDsplitrrrrrr)r_rrprrqrrrrrrurvrrwrxr`rrrrrrrgr>r>rcrsH#   z&ModernBertForQuestionAnswering.forwardr)rhrirjrr@rrr{rrrrrrrrnr>r>rarcrCs^      rC)rrgr)r?rArBrCrr)Ur; contextlibrtypingrrrr{torch.nn.functionalrrr torch.utils.checkpointtorch.nnrrr activationsr configuration_utilsr modeling_attn_mask_utilsr modeling_layersrmodeling_outputsrrrrrmodeling_utilsrutilsrrrutils.import_utilsrgemma.modeling_gemmarrflash_attn.flash_attn_interfacerflash_attn.layers.rotaryrflash_attn.ops.triton.rotaryrobject get_loggerrhrWrrrrrrautogradFunctionrrrr:rrrrrrrrwrr rrrr)rgr=r?rArBrC__all__r>r>r>rcs2           H$ ) ; 5   .   +  $P..  wZ [