o ۷i@s ddlZddlZddlmZddlmZmZddlZddlm m Z ddlm Z ddl m Z mZmZddlmZddlmZdd lmZdd lmZmZmZmZmZmZdd lmZmZdd l m!Z!dd l"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)e$rddl*m+Z+ddl,m-Z-ddl.m/Z/ne0Z-e%1e2Z3Gdddej4j5Z6  d]deej7dee8fddZ9Gddde-Z:Gddde j;Zd"d#Z?d^d$d%Z@ &d_d'd(d)ej7d*ej7d+ej7d,eejAd-eBe8e8fd.e8d/e8d0eeCd1eeBej7ej7feBej7ffd2d3ZDejEfd'd(d)ej7d4e:dej7de8d-eBe8e8fd.e8d/e8d5ejFd1eBej7fd6d7ZGd'd(d)ej7d*ej7d+ej7d,eejAd-eBe8e8fd.e8d/e8d1eBej7fd8d9ZHeGeDeHd:ZIGd;d(d(e j;ZJGdd?d?e!ZL  d]d@ej7d*ej7d,eej7dAeej7d1eBej7ej7ej7e8eej7eej7ff dBdCZMd@ej7dDej7dEe8dFe8d1ej7f dGdHZNe#GdIdJdJeLZOGdKdLdLe j;ZPe#dMdNGdOdPdPeLZQe#dQdNGdRdSdSeLZRe#dTdNGdUdVdVeLZSe#GdWdXdXeLZTe#dYdNGdZd[d[eLZUgd\ZVdS)`N) nullcontext)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)PreTrainedModel)auto_docstringis_flash_attn_2_availablelogging)is_triton_available)ModernBertConfig) flash_attn_varlen_qkvpacked_func)RotaryEmbedding) apply_rotaryc@s>eZdZe  ddeejdeefddZeddZ dS) ApplyRotaryEmbUnpadN cu_seqlens max_seqlenc Csd|}|j\}}}} |ddddf|d| } t| ||d||ddd||||||_|S)NrFT)seqlen_offsetsr r! interleavedinplace) contiguousshapeviewrsave_for_backwardr!) ctxqkvcossinr r! total_nnz_three_nheadsheaddimqkr4h/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/modernbert/modeling_modernbert.pyforward>s   zApplyRotaryEmbUnpad.forwardc Csn|j\}}}|}|j\}}}}|ddddf|d|} t| ||d||jdddd |ddddddfS)Nr"r#rFT)r$r r!r%r& conjugate) saved_tensorsr'r(r)rr!) r+dor-r.r r/r0r1r2dqkr4r4r5backward]s  zApplyRotaryEmbUnpad.backwardNN) __name__ __module__ __qualname__ staticmethodrtorchTensorintr6r;r4r4r4r5r=s rr r!cCst|||||S)a Arguments: qkv: (total_nnz, 3, nheads, headdim) - input tensor for packed QKV. cos, sin: (seqlen_rotary, rotary_dim / 2) interleaved: if True, rotate pairs of even and odd dimensions (GPT-J style) instead of 1st half and 2nd half (GPT-NeoX style). inplace: if True, apply rotary embedding in-place. seqlen_offsets: (batch_size,) or int. Each sequence in x is shifted by this amount. Most commonly used in inference when we have KV cache. cu_seqlens: (batch + 1,) or None max_seqlen: int Return: out: (total_nnz, dim) rotary_dim must be <= headdim Apply rotary embedding to the first rotary_dim of x. )rapply)r,r-r.r r!r4r4r5apply_rotary_unpaddedtsrEc seZdZdZ    ddededeedeejdeej f fd d Z dd ej d ej deed e ej e ej ej fffddZd efddZZS)!ModernBertUnpaddedRotaryEmbeddingzP The rotary position embeddings applied directly to unpadded sequences. @Ndimbaser!devicedtypecsRtj|||dd||_|dur#|dur%|dur'|j|||ddSdSdSdS)a max_seqlen: if max_seqlen, device, and dtype are provided, we precompute the cos_sin_cache up to max_seqlen. If the max_seqlen, device, or dtype during training/inference differ, the cos_sin_cache will be recomputed during the forward pass. F)rHrIrJr%NrJrK)super__init__r!_update_cos_sin_cache)selfrHrIr!rJrK __class__r4r5rNs   z*ModernBertUnpaddedRotaryEmbedding.__init__r,r returncCs6|dur|j||j|jdt||j|j||d}|S)z Apply rotary embedding *inplace* to qkv. qkv: (total_nnz, 3, nheads, headdim) cu_seqlens: (batch + 1,) cumulative sequence lengths max_seqlen: int max seq length in the batch NrLr r!)rOrJrKrE _cos_cached _sin_cached)rPr,r r!r4r4r5r6s z)ModernBertUnpaddedRotaryEmbedding.forwardcCsd|jd|jd|jS)Nzdim=z, base=z , scale_base=)rHrI scale_baserPr4r4r5 extra_reprsz,ModernBertUnpaddedRotaryEmbedding.extra_repr)rGNNNN)r=r>r?__doc__rCfloatrrArJrKrNrBrtupler6strrY __classcell__r4r4rQr5rFs8 rFcspeZdZdZdeffdd Zejdddejdej fd d Z dde ejd e ej dej fd dZ Z S)ModernBertEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. configcsRt||_tj|j|j|jd|_tj |j|j |j d|_ t |j|_dS)N) padding_idxepsbias)rMrNrar Embedding vocab_size hidden_size pad_token_idtok_embeddings LayerNormnorm_eps norm_biasnormDropoutembedding_dropoutdroprPrarQr4r5rNs zModernBertEmbeddings.__init__Tdynamic input_idsrScC||||SrZ)rqrnrj)rPrur4r4r5compiled_embeddingssz(ModernBertEmbeddings.compiled_embeddingsN inputs_embedscCsH|dur|||}|S|jjr||n ||||}|SrZ)rqrnrareference_compilerwrj)rPrurx hidden_statesr4r4r5r6s zModernBertEmbeddings.forwardr<)r=r>r?r[rrNrAcompile LongTensorrBrwrr6r_r4r4rQr5r`s r`cs<eZdZdZdeffdd ZdejdejfddZZ S) ModernBertMLPa6Applies the GLU at the end of each ModernBERT layer. Compared to the default BERT architecture, this block replaces :class:`~transformers.model.bert.modeling_bert.BertIntermediate` and :class:`~transformers.model.bert.modeling_bert.SelfOutput` with a single module that has similar functionality. racsft||_tj|jt|jd|jd|_ t |j |_ t |j|_tj|j|j|jd|_dS)Nr"re)rMrNrarLinearrhrCintermediate_sizemlp_biasWir hidden_activationactro mlp_dropoutrqWorrrQr4r5rNs   zModernBertMLP.__init__rzrScCs2||jddd\}}|||||S)Nr"r#rH)rchunkrrqr)rPrzinputgater4r4r5r6szModernBertMLP.forward) r=r>r?r[rrNrArBr6r_r4r4rQr5r}sr}csDeZdZUejed<ddeffdd Zee ddZ Z S) ModernBertRotaryEmbeddinginv_freqNracstt|drt|jtr|jd|jd|_nd|_|j|_ |j|_ ||_ t |j|_ | |j |\}|_|jd|dd|j|_dS)N rope_scaling rope_typetypedefaultrF) persistent)rMrNhasattr isinstancerdictgetrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrar rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)rPrarJrrQr4r5rNs   z"ModernBertRotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t j |dd+|| dd}t j ||fdd }| |j}||j} Wdn1smwY|j|jd | j|jd fS) Nrr#rmpscpuF) device_typeenabledr"r)rK)rr\expandr(torJrrr^rAautocast transposecatr-rr.rK) rPx position_idsinv_freq_expandedposition_ids_expandedrfreqsembr-r.r4r4r5r6 s0&z!ModernBertRotaryEmbedding.forwardrZ) r=r>r?rArB__annotations__rrNno_gradrr6r_r4r4rQr5rs rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr#r"r)r(rAr)rx1x2r4r4r5 rotate_halfsrcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkr-r.r unsqueeze_dimq_embedk_embedr4r4r5apply_rotary_pos_emb s  rFmoduleModernBertAttentionr,attention_masksliding_window_maskrlocal_attentionbsrHoutput_attentionsrSc Ks|j||d\} } |ddjdd\} } }t| | | | \} } |jd}t| | dd|}|dkr6|}||}tjj |dtj d  | j }tjj ||j|jd }t||}|dd}||d|}|rn||fS|fS) Nrr rr"r࿩r#r#r#rHrK)ptraining) rotary_embrunbindrhead_dimrAmatmulr functionalsoftmaxfloat32rrKdropoutattention_dropoutrr'r))rr,rrrrrrHr_kwargsr-r.querykeyvaluescale attn_weights attn_outputr4r4r5eager_attention_forward;s    rr target_dtypec Ks||||d}|jtjtjfv} | r1|j} ||}t||||jr$|jnd|j|d} | | } nt||||jr;|jnd|j|d} | ||fS)NrT)r r! dropout_p deterministic window_size) rKrAfloat16bfloat16rrrrdeterministic_flash_attnr)) rr,rr r!rrrHrr convert_dtype orig_dtypeattnr4r4r5flash_attention_forward`s.   rcKs|j||d\} } |ddjdd\} } } t| | | | \} } |dkr%|}tj| | | |jr0|jnd|ddd}| |d |}|fS) Nrr rr"rrr)r attn_maskr#) rrrrFscaled_dot_product_attentionrrr'r))rr,rrrrrrHrr-r.rrrrr4r4r5sdpa_attention_forwards"  r)flash_attention_2eagersdpacsReZdZdZd dedeeffdd Z ddej d ee d ej fd d Z Z S)raPerforms multi-headed self attention on a batch of unpadded sequences. If Flash Attention 2 is installed, this module uses Flash Attention to improve throughput. If Flash Attention 2 is not installed, the implementation will use PyTorch's SDPA kernel, which requires padding and unpadding inputs, adding some overhead. See `forward` method for additional details. Nralayer_idcsbt||_||_|j|jdkr td|jd|jd|j|_|j|_|j|_ |j|j|_ |j |j |_ t j |jd|j |jd|_||jdkri|jd|jdf|_|jdurb|jn|j}|j}n d|_|j}|j}|jd krt|j ||d |_nt|}||_t|d |_t j |j|j|jd|_|jd krt |jnt |_t |_!dS) NrzThe hidden size (z6) is not a multiple of the number of attention heads ()r r~r"rr)rHr!rIrar)"rMrNrarrhnum_attention_heads ValueErrorrr num_headsr all_head_sizerrattention_biasWqkvglobal_attn_every_n_layersrlocal_rope_thetaglobal_rope_thetar_attn_implementationrFrcopydeepcopy rope_thetarrroIdentityout_dropset pruned_heads)rPrarrr config_copyrQr4r5rNs<      zModernBertAttention.__init__FrzrrSc Ks||}|jd}|jjdkr|dd|j|j}n ||dd|j|j}t|jj|f||j|j ||j |d|}|d}| | |}|f|ddS)Nrrr#r )r,rrrrHrr) rr(rarr)rrMODERNBERT_ATTENTION_FUNCTIONrrrrr)rPrzrkwargsr,r attn_outputsr4r4r5r6s(     zModernBertAttention.forwardrZF) r=r>r?r[rrrCrNrArBboolr6r_r4r4rQr5rs *cseZdZddedeeffdd Zejdddej d ej fd d Z      ddej d eej deej deej deej deedee d ej fddZ ZS)ModernBertEncoderLayerNrarcspt||_|dkrt|_n tj|j|j|j d|_t ||d|_ tj|j|j|j d|_ t ||_dS)Nrrc)rar)rMrNrarr attn_normrkrhrlrmrrmlp_normr}mlp)rPrarrQr4r5rNs  zModernBertEncoderLayer.__init__TrsrzrScC|||SrZ)rrrPrzr4r4r5 compiled_mlpz#ModernBertEncoderLayer.compiled_mlpFrrrr r!rc Csf|j||||||||d}||d}|jjr||n|||} || }|f|ddS)Nrrrr r!rrr)rrraryr rr) rPrzrrrr r!rr mlp_outputr4r4r5r6s    zModernBertEncoderLayer.forwardrZ)NNNNNF)r=r>r?rrrCrNrAr{rBr r|rr6r_r4r4rQr5rs6  rcs|eZdZUeed<dZdZddgZdZdZ dZ de j fdd Z dd eed ed effd d ZddZfddZZS)ModernBertPreTrainedModelramodelTr`rFrcsx|jjdur ddtjdtffdd }|jj|jjtd|jj|jj|jj dd}t |t r?||j |d dSt |t rV||j|d ||j|d dSt |trm||j|d ||j|d dSt |tr|||j|d dSt |tr||j|d dSt |ttttfr||j|d dSt |tjr|jjd |jdur|jj dSdSdS)Nr rstdcsRtjj|jd| ||dt|tjr%|jdur'tj|jdSdSdS)Nr)meanrab)rinit trunc_normal_weightrrrezeros_)rr cutoff_factorr4r5 init_weightAs  z.init_weightg@r)inout embedding final_outrrrrg?)!rainitializer_cutoff_factorrModuler\initializer_rangemathsqrtnum_hidden_layersrhrr`rjr}rrrrModernBertPredictionHeaddenseModernBertForMaskedLMdecoder#ModernBertForSequenceClassificationModernBertForMultipleChoice ModernBertForTokenClassificationModernBertForQuestionAnswering classifierrkrdatafill_rezero_)rPrrstdsr4rr5 _init_weights<sH        z'ModernBertPreTrainedModel._init_weightsattn_implementation is_init_checkrSc sDz|dur |r dn|}Wn ttfyYnwtj||dS)zR Checks and dispatches to hhe requested attention implementation. Nr)r2r3)_flash_attn_2_can_dispatchr ImportErrorrM%_check_and_adjust_attn_implementation)rPr2r3rQr4r5r6ps z?ModernBertPreTrainedModel._check_and_adjust_attn_implementationcCs|jjdurdSt|dr!t|jdkr!|jjrtdd|j_|jjdkr4|jjr0tdd|j_|jjdkrG|jjrCtdd|j_|jjdurTt |j_dSdS) NF hf_device_maprzqIf `accelerate` split the model across devices, `torch.compile` will not work. Falling back to non-compiled mode.rz|Compiling the model with `torch.compile` and using a `torch.mps` device is not supported. Falling back to non-compiled mode.rz|Compiling the model with `torch.compile` and using a `torch.cpu` device is not supported. Falling back to non-compiled mode.) raryrlenr7logger warning_oncerJrrrXr4r4r5_maybe_set_compiles.    z,ModernBertPreTrainedModel._maybe_set_compilecs<tj|i|}|jjdvr|jjrtdd|j_|S)N>NTzcResizing token embeddings with `torch.compile` is not supported. Falling back to non-compiled mode.F)rMresize_token_embeddingsraryr9r:)rPargsr model_embedsrQr4r5r<s z1ModernBertPreTrainedModel.resize_token_embeddingsr)r=r>r?rrbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_supports_flex_attnrrr1rr^rr6r;r<r_r4r4rQr5r 2s& 5r inputslabelscCs|jdtjd}tj|dd}t|}tjj tj |dtjdd}| dkr7||}n|j ^} } } | | } |j| g| R|}|durV||nd} |durb||nd}||||| |fS) a Remove padding from input sequences. Args: inputs: (batch, seqlen, ...) or (batch, seqlen) attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid. position_ids: (batch, seqlen), int, position ids labels: (batch, seqlen), int, labels Returns: unpadded_inputs: (total_nnz, ...), where total_nnz = number of tokens selected in attention_mask. indices: (total_nnz) cu_seqlens: (batch + 1), the cumulative sequence lengths max_seqlen_in_batch: int unpadded_position_ids: (total_nnz) or None unpadded_labels: (total_nnz) or None r#rF)as_tupler)rrr"N)sumrAint32nonzeroflattenrCmaxitemrrpadcumsumrHr(r))rErrrFseqlens_in_batchindicesmax_seqlen_in_batchr unpadded_inputsbatchseqlenrestr(unpadded_position_idsunpadded_labelsr4r4r5_unpad_modernbert_inputs  rYrQrTrUcCs|dkrtj|||j|jd}|||<|||}|S|j^}}tj||g|R|j|jd}|||<|j||g|R}|S)aQ Add padding to sequences. Args: inputs: (total_nnz, ...) or (total_nnz,), where total_nnz = number of tokens selected in attention_mask. indices: (total_nnz) batch: int, batch size seqlen: int, max sequence length Returns: padded_inputs: (batch, seqlen, ...) or (batch, seqlen) rrKrJ)rHrAzerosrKrJr)r()rErQrTrUoutput padded_inputs_rVr4r4r5_pad_modernbert_outputs   "r_c!seZdZdeffdd ZddZddZe             dd ee j d ee j d ee j d ee j d ee j dee j dee j dee dee dee dee dee dee deee j dfeffddZd e j de de j fddZZS)ModernBertModelracsft|_t|_tfddtjD|_ tj j j j d|_d|_|dS)Ncsg|]}t|qSr4)r).0rrr4r5 sz,ModernBertModel.__init__..rcF)rMrNrar` embeddingsr ModuleListranger#layersrkrhrlrm final_normgradient_checkpointing post_initrrrQrr5rNs   zModernBertModel.__init__cCs|jjSrZrcrjrXr4r4r5get_input_embeddingssz$ModernBertModel.get_input_embeddingscCs ||j_dSrZrj)rPrr4r4r5set_input_embeddings s z$ModernBertModel.set_input_embeddingsNrurrrrxrQr r! batch_sizeseq_lenroutput_hidden_states return_dictrS.c s| dur| n|jj} | dur| n|jj} | dur| n|jj} |du|duAr*td| r.dnd}| r4dnd}||durD|||durcdurc|durZ|jdd\n |jdd\|durj|jn|j}|dur|t j f|t j d}d}|jj dkrˆdur|dur|durd}|durt t||d ^}}}}Wdn1swYn#t||d ^}}}}n|durt j|d d }|j|| d \}}|j||d }|jD])}| r||f}|||||||| d}|d }| r t|dkr ||df}q| r||f}||}|r8t|d}|dur7tfdd|D}n#|jj dkr[|dur[|ddkr[|d }tdd|D}| sjtdd|||fDSt|||dS) sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nz:You must specify exactly one of input_ids or inputs_embedsr4r"rLFrT)rErrJr)r)rurxr rrErQrTrUc3s |] }t|dVqdS)rsN)r_rahsrmrQrnr4r5 |s  z*ModernBertModel.forward..r#css|]}|dVqdS)rN)rrtr4r4r5rwscss|] }|dur|VqdSrZr4)ravr4r4r5rws)last_hidden_staterz attentions)rarrouse_return_dictrr;%warn_if_padding_and_no_attention_maskr(rJrAonesrrrrYaranger_update_attention_maskrcrfr8rgr_r]rHr )rPrurrrrxrQr r!rmrnrrorpall_hidden_statesall_self_attentionsrJrepadr^rz encoder_layer layer_outputsr4rvr5r6s!              zModernBertModel.forwardcCs|r#|jjdkrtdd|j_n|jjdkr#td|jjdt||j}t|jd d}t ||j }||jj dk d d |j}||t|jj}||fS)NrzOutputting attentions is only supported with the 'eager' attention implementation, not with "sdpa". Falling back to `attn_implementation="eager"`.rzZOutputting attentions is only supported with the eager attention implementation, not with zT. Consider setting `attn_implementation="eager"`. Setting `output_attentions=False`.r"r)rarr9r:r rKrAr~r(rabsTrrrJ masked_fill logical_notfinfomin)rPrrglobal_attention_maskrowsdistance window_maskrr4r4r5rs&    "z&ModernBertModel._update_attention_mask NNNNNNNNNNNNN)r=r>r?rrNrkrlrrrAr|rBrCrrr]r r6rr_r4r4rQr5r`sb       "r`cs8eZdZdeffdd ZdejdejfddZZS)r$racsNt||_t|j|j|j|_t|j |_ tj |j|j |j d|_dS)Nrc)rMrNrarrrhclassifier_biasr%r classifier_activationrrkrlrmrnrrrQr4r5rNs  z!ModernBertPredictionHead.__init__rzrScCrvrZ)rnrr%rr4r4r5r6sz ModernBertPredictionHead.forward) r=r>r?rrNrArBr6r_r4r4rQr5r$sr$zd The ModernBert Model with a decoder head on top that is used for masked language modeling. ) custom_introc"s eZdZdgZdeffdd ZddZdejfdd Z e j d d d e j d e j fddZ e              d!dee jdee j dee j dee j dee j dee j dee j dee j deedeedeedeedeedeed eee j effdd ZZS)"r&zdecoder.weightracs^t|||_t||_t||_tj|j |j |j d|_ |jj |_ |jj|_|dS)Nr~)rMrNrar`rr$headrrrhrg decoder_biasr'sparse_predictionsparse_pred_ignore_indexrirrrQr4r5rNs      zModernBertForMaskedLM.__init__cCs|jSrZr'rXr4r4r5get_output_embeddingssz+ModernBertForMaskedLM.get_output_embeddingsnew_embeddingscCs ||_dSrZr)rPrr4r4r5set_output_embeddingss z+ModernBertForMaskedLM.set_output_embeddingsTrsr\rScCrrZ)r'r)rPr\r4r4r5 compiled_headr z#ModernBertForMaskedLM.compiled_headNrurrrrxrFrQr r!rmrnrrorpcKs|dur|n|jj}||jjdkr|dur|dur| dur| dur?| dur?|dur6|jdd\} } n |jdd\} } |durF|jn|j}|durXtj| | f|tjd}|durt t ||||d\}}}} }}Wdn1sywYnt ||||d\}}}} }}|j |||||||| | | | | |d }|d}|j r|dur| d}| |jdd}||jk}||}||}|jjr||n|||}d}|dur|j||fd |jji|}|jjdkrO|jjs|durtnt t||| | d }Wdn 1swYt|d ddurOg}|jD]"}|d kr=|jdd kr=|d}|t||| | d q't||_|sa|f}|dur_|f|S|St|||j|jdS)rqNrr"rL)rErrrF rurrrrxrQr r!rmrnrrorprr#rgrsrzr rlosslogitsrzrz) rar{r;rr(rJrAr}rrrYrrr)rryrr'r loss_functionrgrepad_logits_with_gradrr_getattrrzrHsqueezeappendr]rrz)rPrurrrrxrFrQr r!rmrnrrorprrJoutputsry mask_tokensrrpadded_hidden_statesrur\r4r4r5r6s#         zModernBertForMaskedLM.forwardNNNNNNNNNNNNNN)r=r>r?_tied_weights_keysrrNrrrrrAr{rBrrrr|rCrrr]rr6r_r4r4rQr5r&sj       r&z` The ModernBert Model with a sequence classification head on top that performs pooling. c"eZdZdeffdd Ze              ddeejdeej deej deej d eej d eej d eej d eej d ee dee dee dee dee dee de e ej effddZZS)r(racs\t||j|_||_t||_t||_tj |j |_ t |j|j|_|dSrZ)rMrN num_labelsrar`rr$rrArroclassifier_dropoutrqrrhr,rirrrQr4r5rN_s    z,ModernBertForSequenceClassification.__init__NrurrrrxrFrQr r!rmrnrrorprScKs\|dur|n|jj}||dur|||| dur7| dur7|dur.|jdd\} } n |jdd\} } |dur>|jn|j}|durPtj| | f|tjd}|j |||||||| | | | | |d }|d}|jj dkru|dddf}n|jj dkr|| dj d d |j d d d }| |}||}||}d}|dur|jjdur|jd krd |j_n|jd kr|jtjks|jtjkrd|j_nd|j_|jjd krt}|jd kr|||}n-|||}n'|jjdkrt}||d|j|d}n|jjdkrt}|||}|s$|f}|dur"|f|S|St|||j|jdS)aB sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nr"rLrrclsrr#rrTrHkeepdim regressionsingle_label_classificationmulti_label_classificationr)rar{r;r|r(rJrAr}rrclassifier_poolingrrHrrqr, problem_typerrKlongrCrrrr)rrrzrz)rPrurrrrxrFrQr r!rmrnrrorprrJrry pooled_outputrrloss_fctr\r4r4r5r6ls'          "     z+ModernBertForSequenceClassification.forwardr)r=r>r?rrNrrrAr|rBrCrrr]rr6r_r4r4rQr5r(Ys`      r(zv The ModernBert Model with a token classification head on top, e.g. for Named Entity Recognition (NER) tasks. c"r)r*racVt||j|_t||_t||_tj |j |_ t |j |j|_|dSrZrMrNrr`rr$rrArrorrqrrhr,rirrrQr4r5rNs    z)ModernBertForTokenClassification.__init__NrurrrrxrFrQr r!rmrnrrorprScCs|dur|n|jj}||j|||||||| | | | | |d }|d}||}||}||}d}|durIt}||d|j |d}|s_|f|dd}|dur]|f|S|St |||j |j dS)a sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nrrr#rr) rar{r;rrrqr,rr)rrrzrz)rPrurrrrxrFrQr r!rmrnrrorprryrrrr\r4r4r5r6sD$   z(ModernBertForTokenClassification.forwardr)r=r>r?rrNrrrAr|rBrCrrr]rr6r_r4r4rQr5r*s`      r*c"seZdZdeffdd Ze             ddeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee dee dee dee de e eje ffddZZS)r+racrrZrrrrQr4r5rNCs    z'ModernBertForQuestionAnswering.__init__Nrurrrstart_positions end_positionsrQr r!rmrnrrorprScKs|dur|n|jj}||j||||||| | | | | |d }|d}||}||}||}|jddd\}}|d }|d }d}|dur_|dur_|j ||||fi|}|sv||f|dd}|durt|f|S|St ||||j |j dS)rqN) rrrrQr r!rmrnrrorprrr#r)r start_logits end_logitsrzrz)rar{r;rrrqr,splitrr'rrrzrz)rPrurrrrrrQr r!rmrnrrorprrryrrrrr\r4r4r5r6NsH#   z&ModernBertForQuestionAnswering.forwardr)r=r>r?rrNrrrArBrCrrr]rr6r_r4r4rQr5r+As^      r+z The ModernBert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. c"r)r)racsRt|||_t||_t||_tj |j |_ t |j d|_|dSNr)rMrNrar`rr$rrArrorrqrrhr,rirrrQr4r5rNs    z$ModernBertForMultipleChoice.__init__NrurrrrxrFrQr r!rmrnrrorprScKs|dur|n|jj}|dur|jdn|jd}|dur%|d|dnd}|dur4|d|dnd}|durC|d|dnd}|durV|d|d|dnd}||j|||||||| | | | | |d }|d}|jjdkrtj |jd|j d}|dur|j dd  |j }n tj dtj|j d }|||f}n|jjd kr|jdd d }||djdd |}||}||}||}|d|}d}|durt}|||}|s|f|dd}|dur|f|S|St|||j|jdS)a sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nrr#rrrrrrrZrTrr)rar{r(r)sizer;rrrAr~rJargmaxrtensorrrHrrrqr,rrrrzrz)rPrurrrrxrFrQr r!rmrnrrorpr num_choicesrry indices_0cls_masknum_non_pad_tokensrrreshaped_logitsrrr\r4r4r5r6sh&       z#ModernBertForMultipleChoice.forwardr)r=r>r?rrNrrrAr|rBrCrrr]rr6r_r4r4rQr5r)s`      r))r`r r&r(r*r+r)r<rr)Wrr! contextlibrtypingrrrAtorch.nn.functionalrrrtorch.nnrrr activationsr modeling_attn_mask_utilsr modeling_layersr modeling_outputsr rrrrrmodeling_rope_utilsrrmodeling_utilsrutilsrrrutils.import_utilsrconfiguration_modernbertrflash_attn.flash_attn_interfacerflash_attn.layers.rotaryrflash_attn.ops.triton.rotaryrobject get_loggerr=r9autogradFunctionrrBrCrErFrr`r}rrrr|r]rrrrKrrrrrr rYr_r`r$r&r(r*r+r)__all__r4r4r4r5s<            ; 5$ $   .   +  $O.$ ) 7 Z[ z