o wi@sddlZddlmZddlmZmZddlZddlmm Z ddlmZddl m Z m Z mZddlmZddlmZdd lmZdd lmZmZmZmZmZdd lmZmZdd lmZdd l m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'e"rddl(m)Z)ddl*m+Z+ddl,m-Z-ne.Z+e#/e0Z1Gdddej2j3Z4  dZdeej5dee6fddZ7Gddde+Z8Gdddej9Z:Gdddej9Z;Gd d!d!ej9Z &d\d'd(d)ej5d*ej5d+ej5d,eej?d-e@e6e6fd.e6d/e6d0eeAd1ee@ej5ej5fe@ej5ffd2d3ZBejCfd'd(d)ej5d4e8dej5de6d-e@e6e6fd.e6d/e6d5ejDd1e@ej5fd6d7ZEd'd(d)ej5d*ej5d+ej5d,eej?d-e@e6e6fd.e6d/e6d1e@ej5fd8d9ZFeEeBeFd:ZGGd;d(d(ej9ZHGdd?d?eZJ  dZd@ej5d*ej5d,eej5dAeej5d1e@ej5ej5ej5e6eej5eej5ff dBdCZKd@ej5dDej5dEe6dFe6d1ej5f dGdHZLe!GdIdJdJeJZMGdKdLdLej9ZNe!dMdNGdOdPdPeJZOe!dQdNGdRdSdSeJZPe!dTdNGdUdVdVeJZQe!GdWdXdXeJZRgdYZSdS)]N) nullcontext)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)PreTrainedModel)auto_docstringis_flash_attn_2_availablelogging)is_triton_available)ModernBertConfig) flash_attn_varlen_qkvpacked_func)RotaryEmbedding) apply_rotaryc@s>eZdZe  ddeejdeefddZeddZ dS) ApplyRotaryEmbUnpadN cu_seqlens max_seqlenc Csd|}|j\}}}} |ddddf|d| } t| ||d||ddd||||||_|S)NrFT)seqlen_offsetsrr interleavedinplace) contiguousshapeviewrsave_for_backwardr ) ctxqkvcossinrr total_nnz_three_nheadsheaddimqkr3o/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/modernbert/modeling_modernbert.pyforward<s   zApplyRotaryEmbUnpad.forwardc Csn|j\}}}|}|j\}}}}|ddddf|d|} t| ||d||jdddd |ddddddfS)Nr!r"rFT)r#rr r$r% conjugate) saved_tensorsr&r'r(rr ) r*dor,r-rr.r/r0r1dqkr3r3r4backward[s  zApplyRotaryEmbUnpad.backwardNN) __name__ __module__ __qualname__ staticmethodrtorchTensorintr5r:r3r3r3r4r;s rrr cCst|||||S)a Arguments: qkv: (total_nnz, 3, nheads, headdim) - input tensor for packed QKV. cos, sin: (seqlen_rotary, rotary_dim / 2) interleaved: if True, rotate pairs of even and odd dimensions (GPT-J style) instead of 1st half and 2nd half (GPT-NeoX style). inplace: if True, apply rotary embedding in-place. seqlen_offsets: (batch_size,) or int. Each sequence in x is shifted by this amount. Most commonly used in inference when we have KV cache. cu_seqlens: (batch + 1,) or None max_seqlen: int Return: out: (total_nnz, dim) rotary_dim must be <= headdim Apply rotary embedding to the first rotary_dim of x. )rapply)r+r,r-rr r3r3r4apply_rotary_unpaddedrsrDc seZdZdZ    ddededeedeejdeej f fd d Z dd ej d ej deed e ej e ej ej fffddZd efddZZS)!ModernBertUnpaddedRotaryEmbeddingzP The rotary position embeddings applied directly to unpadded sequences. @Ndimbaser devicedtypecsRtj|||dd||_|dur#|dur%|dur'|j|||ddSdSdSdS)a max_seqlen: if max_seqlen, device, and dtype are provided, we precompute the cos_sin_cache up to max_seqlen. If the max_seqlen, device, or dtype during training/inference differ, the cos_sin_cache will be recomputed during the forward pass. F)rGrHrIr$NrIrJ)super__init__r _update_cos_sin_cache)selfrGrHr rIrJ __class__r3r4rMs   z*ModernBertUnpaddedRotaryEmbedding.__init__r+rreturncCs6|dur|j||j|jdt||j|j||d}|S)z Apply rotary embedding *inplace* to qkv. qkv: (total_nnz, 3, nheads, headdim) cu_seqlens: (batch + 1,) cumulative sequence lengths max_seqlen: int max seq length in the batch NrKrr )rNrIrJrD _cos_cached _sin_cached)rOr+rr r3r3r4r5s z)ModernBertUnpaddedRotaryEmbedding.forwardcCsd|jd|jd|jS)Nzdim=z, base=z , scale_base=)rGrH scale_baserOr3r3r4 extra_reprsz,ModernBertUnpaddedRotaryEmbedding.extra_repr)rFNNNN)r<r=r>__doc__rBfloatrr@rIrJrMrArtupler5strrX __classcell__r3r3rPr4rEs8 rEcspeZdZdZdeffdd Zejdddejdej fd d Z dde ejd e ej dej fd dZ Z S)ModernBertEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. configcsRt||_tj|j|j|jd|_tj |j|j |j d|_ t |j|_dS)N) padding_idxepsbias)rLrMr`r Embedding vocab_size hidden_size pad_token_idtok_embeddings LayerNormnorm_eps norm_biasnormDropoutembedding_dropoutdroprOr`rPr3r4rMs zModernBertEmbeddings.__init__Tdynamic input_idsrRcC||||SrY)rprmri)rOrtr3r3r4compiled_embeddingssz(ModernBertEmbeddings.compiled_embeddingsN inputs_embedscCsH|dur|||}|S|jjr||n ||||}|SrY)rprmr`reference_compilervri)rOrtrw hidden_statesr3r3r4r5s zModernBertEmbeddings.forwardr;)r<r=r>rZrrMr@compile LongTensorrArvrr5r^r3r3rPr4r_s r_cs<eZdZdZdeffdd ZdejdejfddZZ S) ModernBertMLPa6Applies the GLU at the end of each ModernBERT layer. Compared to the default BERT architecture, this block replaces :class:`~transformers.model.bert.modeling_bert.BertIntermediate` and :class:`~transformers.model.bert.modeling_bert.SelfOutput` with a single module that has similar functionality. r`csft||_tj|jt|jd|jd|_ t |j |_ t |j|_tj|j|j|jd|_dS)Nr!rd)rLrMr`rLinearrgrBintermediate_sizemlp_biasWir hidden_activationactrn mlp_dropoutrpWorqrPr3r4rMs   zModernBertMLP.__init__ryrRcCs2||jddd\}}|||||S)Nr!r"rG)rchunkrrpr)rOryinputgater3r3r4r5szModernBertMLP.forward) r<r=r>rZrrMr@rAr5r^r3r3rPr4r|sr|c sJeZdZd dedededeejffdd Z e e dd Z Z S) ModernBertRotaryEmbeddingNr`rGrHrIcstt|dr|jdur|jd|jd|_nd|_|j|_|j|_||_ t |j|_ |j d|||d\}|_ |j d|dd|j|_dS) N rope_scaling rope_typetypedefault)rGrHinv_freqF) persistent)rLrMhasattrrgetrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr`r rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)rOr`rGrHrIrrPr3r4rMs   z"ModernBertRotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t j |dd+|| dd}t j ||fdd }| |j}||j} Wdn1smwY|j|jd | j|jd fS) Nrr"rmpscpuF) device_typeenabledr!r)rJ)rr[expandr'torI isinstancerr]r@autocast transposecatr,rr-rJ) rOx position_idsinv_freq_expandedposition_ids_expandedrfreqsembr,r-r3r3r4r5s0&z!ModernBertRotaryEmbedding.forwardrY)r<r=r>rrBr[rr@rIrMno_gradrr5r^r3r3rPr4rs &rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr"r!r)r'r@r)rx1x2r3r3r4 rotate_halfsrcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkr,r-r unsqueeze_dimq_embedk_embedr3r3r4apply_rotary_pos_embs  rFmoduleModernBertAttentionr+attention_masksliding_window_maskrlocal_attentionbsrGoutput_attentionsrRc Ks|j||d\} } |ddjdd\} } }t| | | | \} } |jd}t| | dd|}|dkr6|}||}tjj |dtj d  | j }tjj ||j|jd }t||}|dd}||d|}|rn||fS|fS) Nrr rr!r࿩r"r"r"rGrJ)ptraining) rotary_embrunbindrhead_dimr@matmulr functionalsoftmaxfloat32rrJdropoutattention_dropoutrr&r()rr+rrrrrrGr_kwargsr,r-querykeyvaluescale attn_weights attn_outputr3r3r4eager_attention_forward6s    rr target_dtypec Ks||||d}|jtjtjfv} | r1|j} ||}t||||jr$|jnd|j|d} | | } nt||||jr;|jnd|j|d} | ||fS)NrS)rr dropout_p deterministic window_size) rJr@float16bfloat16rrrrdeterministic_flash_attnr() rr+rrr rrrGrr convert_dtype orig_dtypeattnr3r3r4flash_attention_forward[s.   rcKs|j||d\} } |ddjdd\} } } t| | | | \} } |dkr%|}tj| | | |jr0|jnd|ddd}| |d |}|fS) Nrr rr!rrr)r attn_maskr") rrrrFscaled_dot_product_attentionrrr&r()rr+rrrrrrGrr,r-rrrrr3r3r4sdpa_attention_forwards"  r)flash_attention_2eagersdpacsReZdZdZd dedeeffdd Z ddej d ee d ej fd d Z Z S)raPerforms multi-headed self attention on a batch of unpadded sequences. If Flash Attention 2 is installed, this module uses Flash Attention to improve throughput. If Flash Attention 2 is not installed, the implementation will use PyTorch's SDPA kernel, which requires padding and unpadding inputs, adding some overhead. See `forward` method for additional details. Nr`layer_idcs\t||_||_|j|jdkr td|jd|jd|j|_|j|_|j|_ |j|j|_ |j |j |_ t j |jd|j |jd|_||jdkr[|jd|jdf|_nd|_|j}|j}|jdkrt|jdurq|j}|j}|jd krt|j ||d |_n t||j |d |_t j |j|j|jd|_|jd krt |jnt |_t|_dS) NrzThe hidden size (z6) is not a multiple of the number of attention heads ()r r}r!rr)rGr rH)r`rGrHr)rLrMr`rrgnum_attention_heads ValueErrorrr num_headsr all_head_sizerr~attention_biasWqkvglobal_attn_every_n_layersrglobal_rope_thetarlocal_rope_theta_attn_implementationrErrrrnIdentityout_dropset pruned_heads)rOr`r rope_thetarrPr3r4rMs<       zModernBertAttention.__init__FryrrRc Ks||}|jd}|jjdkr|dd|j|j}n ||dd|j|j}t|jj|f||j|j ||j |d|}|d}| | |}|f|ddS)Nrrr"r )r+rrrrGrr) rr'r`rr(rrMODERNBERT_ATTENTION_FUNCTIONrrrrr)rOryrkwargsr+r attn_outputsr3r3r4r5s(     zModernBertAttention.forwardrYF) r<r=r>rZrrrBrMr@rAboolr5r^r3r3rPr4rs +cseZdZddedeeffdd Zejdddej d ej fd d Z      ddej d eej deej deej deej deedee d ej fddZ ZS)ModernBertEncoderLayerNr`rcspt||_|dkrt|_n tj|j|j|j d|_t ||d|_ tj|j|j|j d|_ t ||_dS)Nrrb)r`r)rLrMr`rr attn_normrjrgrkrlrrmlp_normr|mlp)rOr`rrPr3r4rMs  zModernBertEncoderLayer.__init__TrrryrRcC|||SrY)rrrOryr3r3r4 compiled_mlp z#ModernBertEncoderLayer.compiled_mlpFrrrrr rc Csf|j||||||||d}||d}|jjr||n|||} || }|f|ddS)Nrrrrr rrr)rrr`rxrrr) rOryrrrrr rr mlp_outputr3r3r4r5s    zModernBertEncoderLayer.forwardrY)NNNNNF)r<r=r>rrrBrMr@rzrArr{rr5r^r3r3rPr4rs6  rc seZdZeZdZdZddgZdZdZ dZ de j fddZ e dd eejd eeeeeeffd effd d ZddZfddZZS)ModernBertPreTrainedModelmodelTr_rFrcsv|jjdur ddtjdtffdd }|jj|jjtd|jj|jj|jj dd}t |t r?||j |d dSt |t rV||j|d ||j|d dSt |trm||j|d ||j|d dSt |tr|||j|d dSt |tr||j|d dSt |tttfr||j|d dSt |tjr|jjd |jdur|jjdSdSdS)Nr rstdcsRtjj|jd| ||dt|tjr%|jdur'tj|jdSdSdS)Nr)meanrab)rinit trunc_normal_weightrr~rdzeros_)rr cutoff_factorr3r4 init_weight=s  z.init_weightg@r)inout embedding final_outrrrrg?) r`initializer_cutoff_factorrModuler[initializer_rangemathsqrtnum_hidden_layersrgrr_rir|rrrrModernBertPredictionHeaddenseModernBertForMaskedLMdecoder#ModernBertForSequenceClassification ModernBertForTokenClassificationModernBertForQuestionAnswering classifierrjrdatafill_rdzero_)rOrrstdsr3rr4 _init_weights8s@        z'ModernBertPreTrainedModel._init_weightsN torch_dtype device_mapcheck_device_mapc sZ|jdur#d|_z |j|tj|d|dWSttfy"d|_Ynwtj||||dS)NrF)r*r+hard_check_onlyr,)r*r+r,)_attn_implementation_internal_check_and_enable_flash_attn_2r@rr ImportErrorrL_autoset_attn_implementation)clsr`r*r+r,rPr3r4r1gs&  z6ModernBertPreTrainedModel._autoset_attn_implementationcCs|jjdurdSt|dr!t|jdkr!|jjrtdd|j_|jjdkr4|jjr0tdd|j_|jjdkrG|jjrCtdd|j_|jjdurTt |j_dSdS) NF hf_device_maprzqIf `accelerate` split the model across devices, `torch.compile` will not work. Falling back to non-compiled mode.rz|Compiling the model with `torch.compile` and using a `torch.mps` device is not supported. Falling back to non-compiled mode.rz|Compiling the model with `torch.compile` and using a `torch.cpu` device is not supported. Falling back to non-compiled mode.) r`rxrlenr3logger warning_oncerIrrrWr3r3r4_maybe_set_compiles.    z,ModernBertPreTrainedModel._maybe_set_compilecs<tj|i|}|jjdvr|jjrtdd|j_|S)N>NTzcResizing token embeddings with `torch.compile` is not supported. Falling back to non-compiled mode.F)rLresize_token_embeddingsr`rxr5r6)rOargsr model_embedsrPr3r4r8s z1ModernBertPreTrainedModel.resize_token_embeddings)NNT)r<r=r>r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_2_supports_sdpa_supports_flex_attnrrr) classmethodrr@rJrr]dictrBrr1r7r8r^r3r3rPr4r.s,/rinputslabelscCs|jdtjd}tj|dd}t|}tjj tj |dtjdd}| dkr7||}n|j ^} } } | | } |j| g| R|}|durV||nd} |durb||nd}||||| |fS) a Remove padding from input sequences. Args: inputs: (batch, seqlen, ...) or (batch, seqlen) attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid. position_ids: (batch, seqlen), int, position ids labels: (batch, seqlen), int, labels Returns: unpadded_inputs: (total_nnz, ...), where total_nnz = number of tokens selected in attention_mask. indices: (total_nnz) cu_seqlens: (batch + 1), the cumulative sequence lengths max_seqlen_in_batch: int unpadded_position_ids: (total_nnz) or None unpadded_labels: (total_nnz) or None r"rF)as_tupler)rrr!N)sumr@int32nonzeroflattenrBmaxitemrrpadcumsumrGr'r()rDrrrEseqlens_in_batchindicesmax_seqlen_in_batchrunpadded_inputsbatchseqlenrestr'unpadded_position_idsunpadded_labelsr3r3r4_unpad_modernbert_inputs  rXrPrSrTcCs|dkrtj|||j|jd}|||<|||}|S|j^}}tj||g|R|j|jd}|||<|j||g|R}|S)aQ Add padding to sequences. Args: inputs: (total_nnz, ...) or (total_nnz,), where total_nnz = number of tokens selected in attention_mask. indices: (total_nnz) batch: int, batch size seqlen: int, max sequence length Returns: padded_inputs: (batch, seqlen, ...) or (batch, seqlen) r)rJrI)rGr@zerosrJrIr(r')rDrPrSrToutput padded_inputs_rUr3r3r4_pad_modernbert_outputs   "r]c!seZdZdeffdd ZddZddZe             dd ee j d ee j d ee j d ee j d ee j dee j dee j dee dee dee dee dee dee deee j dfeffddZd e j de de j fddZZS)ModernBertModelr`csft|_t|_tfddtjD|_ tj j j j d|_d|_|dS)Ncsg|]}t|qSr3)r).0rr`r3r4 sz,ModernBertModel.__init__..rbF)rLrMr`r_ embeddingsr ModuleListrangerlayersrjrgrkrl final_normgradient_checkpointing post_initrqrPr`r4rMs   zModernBertModel.__init__cCs|jjSrYrbrirWr3r3r4get_input_embeddingssz$ModernBertModel.get_input_embeddingscCs ||j_dSrYri)rOrr3r3r4set_input_embeddings s z$ModernBertModel.set_input_embeddingsNrtrrrrwrPrr batch_sizeseq_lenroutput_hidden_states return_dictrR.c s| dur| n|jj} | dur| n|jj} | dur| n|jj} |du|duAr*td| r.dnd}| r4dnd}||durD|||durcdurc|durZ|jdd\n |jdd\|durj|jn|j}|dur|t j f|t j d}d}|jj dkrˆdur|dur|durd}|durt t||d ^}}}}Wdn1swYn#t||d ^}}}}n|durt j|d d }|j|| d \}}|j||d }|jD])}| r||f}|||||||| d}|d }| r t|dkr ||df}q| r||f}||}|r7t|d}|dur7tfdd|D}| sFtdd|||fDSt|||dS) sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nz:You must specify exactly one of input_ids or inputs_embedsr3r!rKFrT)rDr)rIr)r)rtrwrrrDrPrSrTc3s |] }t|dVqdS)rqN)r])r_hsrlrPrmr3r4 {s  z*ModernBertModel.forward..css|] }|dur|VqdSrYr3)r_vr3r3r4rts)last_hidden_statery attentions)r`rrnuse_return_dictrr7%warn_if_padding_and_no_attention_maskr'rIr@onesrrrrXaranger_update_attention_maskrbrer4rfr]r\r )rOrtrrrrwrPrr rlrmrrnroall_hidden_statesall_self_attentionsrIrepadr\ry encoder_layer layer_outputsr3rsr4r5 s!            zModernBertModel.forwardcCs|r#|jjdkrtdd|j_n|jjdkr#td|jjdt||j}t|jd d}t ||j }||jj dk d d |j}||t|jj}||fS)NrzOutputting attentions is only supported with the 'eager' attention implementation, not with "sdpa". Falling back to `attn_implementation="eager"`.rzZOutputting attentions is only supported with the eager attention implementation, not with zT. Consider setting `attn_implementation="eager"`. Setting `output_attentions=False`.r!r)r`rr5r6r rJr@r{r'rabsTrrrI masked_fill logical_notfinfomin)rOrrglobal_attention_maskrowsdistance window_maskrr3r3r4r|s&    "z&ModernBertModel._update_attention_mask NNNNNNNNNNNNN)r<r=r>rrMrjrkrrr@r{rArBrrr\r r5r|r^r3r3rPr4r^s`       "zr^cs8eZdZdeffdd ZdejdejfddZZS)rr`csNt||_t|j|j|j|_t|j |_ tj |j|j |j d|_dS)Nrb)rLrMr`rr~rgclassifier_biasrr classifier_activationrrjrkrlrmrqrPr3r4rMs  z!ModernBertPredictionHead.__init__ryrRcCrurY)rmrrrr3r3r4r5sz ModernBertPredictionHead.forward) r<r=r>rrMr@rAr5r^r3r3rPr4rsrzd The ModernBert Model with a decoder head on top that is used for masked language modeling. ) custom_introc"s eZdZdgZdeffdd ZddZdejfdd Z e j d d d e j d e j fddZ e              d!dee jdee j dee j dee j dee j dee j dee j dee j deedeedeedeedeedeed eee j effdd ZZS)"rzdecoder.weightr`cs^t|||_t||_t||_tj|j |j |j d|_ |jj |_ |jj|_|dS)Nr})rLrMr`r^rrheadrr~rgrf decoder_biasr sparse_predictionsparse_pred_ignore_indexrhrqrPr3r4rMs      zModernBertForMaskedLM.__init__cCs|jSrYr rWr3r3r4get_output_embeddingssz+ModernBertForMaskedLM.get_output_embeddingsnew_embeddingscCs ||_dSrYr)rOrr3r3r4set_output_embeddingss z+ModernBertForMaskedLM.set_output_embeddingsTrrrZrRcCrrY)r r)rOrZr3r3r4 compiled_headrz#ModernBertForMaskedLM.compiled_headNrtrrrrwrErPrr rlrmrrnrocKsj|dur|n|jj}||jjdkr|dur|dur| dur| dur?| dur?|dur6|jdd\} } n |jdd\} } |durF|jn|j}|durXtj| | f|tjd}|durt t ||||d\}}}} }}Wdn1sywYnt ||||d\}}}} }}|j |||||||| | | | | |d }|d}|j r|dur| d}| |jdd}||jk}||}||}|jjr||n|||}d}|dur|j||fd |jji|}|jjdkr|jjs|durtnt t||| | d }Wdn 1swY|s+|f}|dur)|f|S|St|||j|jd S) rpNrr!rK)rDrrrE rtrrrrwrPrr rlrmrrnrorr"rfrqlosslogitsryrw)r`rxr7rr'rIr@rzrrrXrrr(rrxrr r loss_functionrfrepad_logits_with_gradrr]rryrw)rOrtrrrrwrErPrr rlrmrrnrorrIoutputsrv mask_tokensrrrZr3r3r4r5s~#      zModernBertForMaskedLM.forwardNNNNNNNNNNNNNN)r<r=r>_tied_weights_keysrrMrrr~rr@rzrArrrr{rBrrr\rr5r^r3r3rPr4rsj       rz` The ModernBert Model with a sequence classification head on top that performs pooling. c"eZdZdeffdd Ze              ddeejdeej deej deej d eej d eej d eej d eej d ee dee dee dee dee dee de e ej effddZZS)r!r`cs\t||j|_||_t||_t||_tj |j |_ t |j|j|_|dSrY)rLrM num_labelsr`r^rrrr@rrnclassifier_dropoutrpr~rgr$rhrqrPr3r4rMJs    z,ModernBertForSequenceClassification.__init__NrtrrrrwrErPrr rlrmrrnrorRcKs|dur|n|jj}||j|||||||| | | | | |d }|d}|jjdkr3|dddf}n|jjdkrK||djdd|jdd d }||}||}| |}d}|dur|jj dur|j dkrpd |j_ n|j dkr|j t jks|j t jkrd |j_ nd |j_ |jj d krt}|j dkr|||}n+|||}n%|jj d krt}||d|j |d}n|jj d krt}|||}|s|f}|dur|f|S|St|||j|jdS)aB sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nrrr2r r"rrT)rGkeepdim regressionsingle_label_classificationmulti_label_classificationr)r`rxr7rclassifier_poolingrrGrrpr$ problem_typerrJr@longrBrsqueezerr(rrryrw)rOrtrrrrwrErPrr rlrmrrnrorrrv pooled_outputrrloss_fctrZr3r3r4r5Wsn'        "       z+ModernBertForSequenceClassification.forwardr)r<r=r>rrMrrr@r{rArBrrr\rr5r^r3r3rPr4r!Ds`      r!zv The ModernBert Model with a token classification head on top, e.g. for Named Entity Recognition (NER) tasks. c"r)r"r`cVt||j|_t||_t||_tj |j |_ t |j |j|_|dSrYrLrMrr^rrrr@rrnrrpr~rgr$rhrqrPr3r4rMs    z)ModernBertForTokenClassification.__init__NrtrrrrwrErPrr rlrmrrnrorRcCs|dur|n|jj}||j|||||||| | | | | |d }|d}||}||}||}d}|durIt}||d|j |d}|s_|f|dd}|dur]|f|S|St |||j |j dS)a sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nrrr"rr) r`rxr7rrrpr$rr(rrryrw)rOrtrrrrwrErPrr rlrmrrnrorrvrrrrZr3r3r4r5sD$   z(ModernBertForTokenClassification.forwardr)r<r=r>rrMrrr@r{rArBrrr\rr5r^r3r3rPr4r"s`      r"c"seZdZdeffdd Ze             ddeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee dee dee dee de e eje ffddZZS)r#r`crrYrrqrPr3r4rM!s    z'ModernBertForQuestionAnswering.__init__Nrtrrrstart_positions end_positionsrPrr rlrmrrnrorRcKs|dur|n|jj}||j||||||| | | | | |d }|d}||}||}||}|jddd\}}|d }|d }d}|dur_|dur_|j ||||fi|}|sv||f|dd}|durt|f|S|St ||||j |j dS)rpN) rrrrPrr rlrmrrnrorrr"r)r start_logits end_logitsryrw)r`rxr7rrrpr$splitrr&rrryrw)rOrtrrrrrrPrr rlrmrrnrorrrvrrrrrZr3r3r4r5,sH#   z&ModernBertForQuestionAnswering.forwardr)r<r=r>rrMrrr@rArBrrr\rr5r^r3r3rPr4r#s^      r#)r^rrr!r"r#r;)Nrr)Tr contextlibrtypingrrr@torch.nn.functionalrrrtorch.nnrrr activationsr modeling_attn_mask_utilsr modeling_layersr modeling_outputsr rrrrmodeling_rope_utilsrrmodeling_utilsrutilsrrrutils.import_utilsrconfiguration_modernbertrflash_attn.flash_attn_interfacerflash_attn.layers.rotaryrflash_attn.ops.triton.rotaryrobject get_loggerr<r5autogradFunctionrrArBrDrErr_r|rrrr{r\rrrrJrrrrrrrXr]r^rrr!r"r#__all__r3r3r3r4s0           ; 5! $   .   +  $P.$ ) .  wZ [