o ۷i @sdZddlZddlmZmZddlZddlmZddlmZm Z m Z ddl m Z m Z ddlmZmZmZmZmZmZmZdd lmZdd lmZmZdd lmZmZd d lmZe e!Z"eGdddeZ#Gdddej$Z%Gdddej$Z&Gdddej$Z'Gdddej$Z(Gdddej$Z)Gdddej$Z*Gdddej$Z+Gdddej$Z,eGd d!d!e#Z-Gd"d#d#e#Z.Gd$d%d%ej$Z/ed&d'Gd(d)d)e#Z0eGd*d+d+e#Z1eGd,d-d-e#Z2Gd.d/d/ej$Z3eGd0d1d1e#Z4d2d3Z5gd4Z6dS)5zPyTorch MPNet model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNgelu)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) MPNetConfigc@s"eZdZUeed<dZddZdS)MPNetPreTrainedModelconfigmpnetcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSt|tre|jj dSdS)zInitialize the weightsg)meanstdNg?) isinstancerLinearweightdatanormal_rinitializer_rangebiaszero_ Embedding padding_idx LayerNormfill_ MPNetLMHead)selfmoduler-^/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/mpnet/modeling_mpnet.py _init_weights1s       z"MPNetPreTrainedModel._init_weightsN)__name__ __module__ __qualname__r__annotations__base_model_prefixr/r-r-r-r.r,s  rcs.eZdZfddZdddZddZZS) MPNetEmbeddingscstd|_tj|j|j|jd|_tj|j|j|jd|_ tj |j|j d|_ t |j |_|jdt|jddddS)Nr)r'eps position_ids)rF) persistent)super__init__r'rr& vocab_size hidden_sizeword_embeddingsmax_position_embeddingsposition_embeddingsr(layer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandr+r __class__r-r.r<Es   zMPNetEmbeddings.__init__Nc Ks|dur|durt||j}n||}|dur|}n|dd}|d}|dur8|jddd|f}|durA||}||}||}||}||}|S)Nr9r) "create_position_ids_from_input_idsr'&create_position_ids_from_inputs_embedssizer8r?rAr(rE) r+ input_idsr8 inputs_embedskwargs input_shape seq_lengthrA embeddingsr-r-r.forwardSs"      zMPNetEmbeddings.forwardcCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr9r)dtypedevicer)rOrGrHr'longrX unsqueezerI)r+rQrSsequence_lengthr8r-r-r.rNms  z6MPNetEmbeddings.create_position_ids_from_inputs_embeds)NNN)r0r1r2r<rVrN __classcell__r-r-rKr.r5Ds  r5c.eZdZfddZ    dddZZS)MPNetSelfAttentioncst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|j|_t |j|_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())r;r<r>num_attention_headshasattr ValueErrorintattention_head_size all_head_sizerrqkvorCattention_probs_dropout_probrErJrKr-r.r<s  zMPNetSelfAttention.__init__NFcKsF|j\}}} |||d|j|jdd} |||d|j|jdd} |||d|j|jdd} t | | dd} | t |j} |durS| |7} |dur[| |} t j j| dd}||}|durp||}t || }|dddd}|dd|jf}|j|}||}|r||f}|S|f}|S)Nr9rdimrr)shapergviewrare transposerhrirGmatmulmathsqrtr functionalsoftmaxrEpermute contiguousrOrfrj)r+ hidden_statesattention_mask head_mask position_biasoutput_attentionsrR batch_sizerT_rgrhriattention_scoresattention_probsc new_c_shaperjoutputsr-r-r.rVs@      zMPNetSelfAttention.forwardNNNFr0r1r2r<rVr\r-r-rKr.r^s r^cs6eZdZfddZddZ    d ddZZS) MPNetAttentioncsBtt||_tj|j|jd|_t|j |_ t |_ dSNr6) r;r<r^attnrr(r>rBrCrDrEset pruned_headsrJrKr-r.r<s   zMPNetAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|jj |dd|j_ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrrn) lenrrrarerrrgrhrirjrfunion)r+headsindexr-r-r. prune_headss zMPNetAttention.prune_headsNFc KsB|j|||||d}|||d|}|f|dd} | S)N)r~rr)rr(rE) r+rzr{r|r}r~rR self_outputsattention_outputrr-r-r.rVs zMPNetAttention.forwardr)r0r1r2r<rrVr\r-r-rKr.rs rc2eZdZfddZdejdejfddZZS)MPNetIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSN) r;r<rrr>intermediate_sizedenser hidden_actstrr intermediate_act_fnrJrKr-r.r<s   zMPNetIntermediate.__init__rzreturncCs||}||}|Sr)rr)r+rzr-r-r.rVs  zMPNetIntermediate.forwardr0r1r2r<rGTensorrVr\r-r-rKr.rs rcs8eZdZfddZdejdejdejfddZZS) MPNetOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r;r<rrrr>rr(rBrCrDrErJrKr-r.r< s zMPNetOutput.__init__rz input_tensorrcCs&||}||}|||}|Sr)rrEr()r+rzrr-r-r.rVs  zMPNetOutput.forwardrr-r-rKr.r s $rcr]) MPNetLayercs,tt||_t||_t||_dSr)r;r<r attentionr intermediateroutputrJrKr-r.r<s   zMPNetLayer.__init__NFc KsL|j|||||d}|d}|dd} ||} || |} | f| } | S)N)r}r~rr)rrr) r+rzr{r|r}r~rRself_attention_outputsrrintermediate_output layer_outputr-r-r.rV s     zMPNetLayer.forwardrrr-r-rKr.rs  rc speZdZfddZ     ddejdeejdeejded ed ef d d ZdddZ e dddZ Z S) MPNetEncodercsNt|_j|_tfddtjD|_ t j |j|_ dS)Ncsg|]}tqSr-)r).0rrr-r. >sz)MPNetEncoder.__init__..) r;r<rran_headsr ModuleListrangenum_hidden_layerslayerr&relative_attention_num_bucketsrelative_attention_biasrJrKrr.r<:s  zMPNetEncoder.__init__NFrzr{r|r~output_hidden_states return_dictcKs||}|r dnd} |rdnd} t|jD]'\} } |r!| |f} | |||| |fd|i|} | d}|r=| | df} q|rE| |f} |sStdd|| | fDSt|| | dS)Nr-r~rrcss|] }|dur|VqdSrr-)rrir-r-r. dsz'MPNetEncoder.forward..)last_hidden_staterz attentions)compute_position_bias enumeratertupler )r+rzr{r|r~rrrRr}all_hidden_statesall_attentionsi layer_module layer_outputsr-r-r.rVAs:    zMPNetEncoder.forward c Cs|d|d|d}}}|dur,|dddddf}|dddddf}ntj|tjddddf}tj|tjddddf}||} |j| |d} | |j} || } | gd d} | |d||f } | S)Nrr)rW) num_buckets)rlrrr9) rOrGrHrYrelative_position_buckettorXrrxrZrIry) r+xr8rbszqlenklencontext_positionmemory_positionrelative_position rp_bucketvaluesr-r-r.rks"  z"MPNetEncoder.compute_position_biascCsd}| }|d}||dktj|7}t|}|d}||k}|t||t||||tj}t|t||d}|t |||7}|S)Nrrlr) rrGrYabslogfloatrtmin full_likewhere)rr max_distanceretn max_exactis_small val_if_larger-r-r.r}s &z%MPNetEncoder.relative_position_bucket)NNFFF)Nr)rr) r0r1r2r<rGrrboolrVr staticmethodrr\r-r-rKr.r9s.   *rcr) MPNetPoolercs*tt|j|j|_t|_dSr)r;r<rrr>rTanh activationrJrKr-r.r<s zMPNetPooler.__init__rzrcCs(|dddf}||}||}|SNr)rr)r+rzfirst_token_tensor pooled_outputr-r-r.rVs  zMPNetPooler.forwardrr-r-rKr.rs rcseZdZdfdd ZddZddZdd Ze dd ee j d ee j d ee j dee j dee j dee dee dee de ee jeffddZZS) MPNetModelTcsDt|||_t||_t||_|rt|nd|_| dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r;r<rr5rUrencoderrpooler post_init)r+radd_pooling_layerrKr-r.r<s    zMPNetModel.__init__cC|jjSrrUr?r+r-r-r.get_input_embeddingszMPNetModel.get_input_embeddingscCs ||j_dSrr)r+valuer-r-r.set_input_embeddingss zMPNetModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r+heads_to_prunerrr-r-r. _prune_headsszMPNetModel._prune_headsNrPr{r8r|rQr~rrrc KsL|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur9||||} n|durF|dd} ntd|durQ|jn|j} |dur_tj | | d}| || } | ||jj }|j |||d} |j| | ||||d}|d}|jdur||nd}|s||f|ddSt|||j|jd S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer9z5You have to specify either input_ids or inputs_embeds)rX)rPr8rQ)r{r|r~rrrr)r pooler_outputrzr)rr~ruse_return_dictrc%warn_if_padding_and_no_attention_maskrOrXrGonesget_extended_attention_mask get_head_maskrrUrrr rzr)r+rPr{r8r|rQr~rrrRrSrXextended_attention_maskembedding_outputencoder_outputssequence_outputrr-r-r.rVsH    zMPNetModel.forward)T)NNNNNNNN)r0r1r2r<rrrrrrG LongTensor FloatTensorrrrrr rVr\r-r-rKr.rsB  rcseZdZdgZfddZddZddZe         dd ee j d ee j d ee j d ee j d ee j dee j dee dee dee de ee jeffddZZS)MPNetForMaskedLMzlm_head.decodercs0t|t|dd|_t||_|dSNF)r)r;r<rrr*lm_headrrJrKr-r.r<s   zMPNetForMaskedLM.__init__cCrr)rdecoderrr-r-r.get_output_embeddingsrz&MPNetForMaskedLM.get_output_embeddingscCs||j_|j|j_dSr)rrr$)r+new_embeddingsr-r-r.set_output_embeddings sz&MPNetForMaskedLM.set_output_embeddingsNrPr{r8r|rQlabelsr~rrrc  Cs| dur| n|jj} |j|||||||| d} | d} || } d} |dur7t}|| d|jj|d} | sM| f| dd}| durK| f|S|St| | | j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nr{r8r|rQr~rrrr9rllosslogitsrzr) rrrrrrqr=r rzr)r+rPr{r8r|rQrr~rrrrprediction_scoresmasked_lm_lossloss_fctrr-r-r.rVs4 zMPNetForMaskedLM.forward NNNNNNNNN)r0r1r2_tied_weights_keysr<rrrrrGrrrrrrr rVr\r-r-rKr.rsH     rcs0eZdZdZfddZddZddZZS)r*z5MPNet Head for masked and permuted language modeling.cshtt|j|j|_tj|j|jd|_tj|j|j dd|_ t t |j |_|j|j _dS)Nr6F)r$)r;r<rrr>rr(rB layer_normr=r ParameterrGzerosr$rJrKr-r.r<Fs zMPNetLMHead.__init__cCs|j|j_dSr)r$rrr-r-r. _tie_weightsQszMPNetLMHead._tie_weightscKs*||}t|}||}||}|Sr)rr rrr+featuresrRrr-r-r.rVTs   zMPNetLMHead.forward)r0r1r2__doc__r<rrVr\r-r-rKr.r*Cs   r*z MPNet Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introceZdZfddZe         ddeejdeejdeejdeejdeejd eejd ee d ee d ee d e e ej e ffddZZS)MPNetForSequenceClassificationcs8t||j|_t|dd|_t||_|dSr)r;r< num_labelsrrMPNetClassificationHead classifierrrJrKr-r.r<fs   z'MPNetForSequenceClassification.__init__NrPr{r8r|rQrr~rrrc  Csf| dur| n|jj} |j|||||||| d} | d} || } d} |dur|jjdurP|jdkr6d|j_n|jdkrL|jtjksG|jtj krLd|j_nd|j_|jjdkrnt }|jdkrh|| | } n+|| |} n%|jjdkrt }|| d|j| d} n|jjdkrt}|| |} | s| f| d d}| dur| f|S|St| | | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrr regressionsingle_label_classificationmulti_label_classificationr9rlr)rrrr problem_typerrWrGrYrdrsqueezerrqrrrzrr+rPr{r8r|rQrr~rrrrrrr rr-r-r.rVpsR    "       z&MPNetForSequenceClassification.forwardr )r0r1r2r<rrrGrrrrrrrrVr\r-r-rKr.r_sB     rcr)MPNetForMultipleChoicecs@t|t||_t|j|_t|j d|_ | dS)Nr) r;r<rrrrCrDrErr>rrrJrKr-r.r<s   zMPNetForMultipleChoice.__init__NrPr{r8r|rQrr~rrrc  CsN| dur| n|jj} |dur|jdn|jd} |dur%|d|dnd} |dur4|d|dnd} |durC|d|dnd} |durV|d|d|dnd}|j| | | ||||| d}|d}||}||}|d| }d}|durt}|||}| s|f|dd}|dur|f|S|St |||j |j dS)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr9rm)r8r{r|rQr~rrrlr) rrrprqrOrrErrrrzr)r+rPr{r8r|rQrr~rr num_choicesflat_input_idsflat_position_idsflat_attention_maskflat_inputs_embedsrrrreshaped_logitsrr rr-r-r.rVsH$    zMPNetForMultipleChoice.forwardr )r0r1r2r<rrrGrrrrrrrrVr\r-r-rKr.r!sB     r!cr)MPNetForTokenClassificationcsNt||j|_t|dd|_t|j|_t |j |j|_ | dSr) r;r<rrrrrCrDrErr>rrrJrKr-r.r<s  z$MPNetForTokenClassification.__init__NrPr{r8r|rQrr~rrrc  Cs| dur| n|jj} |j|||||||| d} | d} || } || } d} |dur;t}|| d|j|d} | sQ| f| dd}| durO| f|S|St| | | j | j dS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr9rlr) rrrrErrrqrrrzrr r-r-r.rVs6  z#MPNetForTokenClassification.forwardr )r0r1r2r<rrrGrrrrrrrrVr\r-r-rKr.r(sB     r(cs(eZdZdZfddZddZZS)rz-Head for sentence-level classification tasks.cs@tt|j|j|_t|j|_t|j|j |_ dSr) r;r<rrr>rrCrDrErout_projrJrKr-r.r<Ws z MPNetClassificationHead.__init__cKsL|dddddf}||}||}t|}||}||}|Sr)rErrGtanhr)rr-r-r.rV]s     zMPNetClassificationHead.forward)r0r1r2rr<rVr\r-r-rKr.rTs rcseZdZfddZe          ddeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e ej e ffddZZS)MPNetForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSr) r;r<rrrrrr> qa_outputsrrJrKr-r.r<is  z"MPNetForQuestionAnswering.__init__NrPr{r8r|rQstart_positions end_positionsr~rrrc  CsF| dur| n|jj} |j||||||| | d} | d} || } | jddd\}}|d}|d}d}|dur|durt|dkrN|d}t|dkr[|d}|d}| d|}| d|}t |d}|||}|||}||d}| s||f| dd}|dur|f|S|St |||| j | j dS) Nrrrr9rn) ignore_indexrl)r start_logits end_logitsrzr)rrrr,splitrryrrOclamprrrzr)r+rPr{r8r|rQr-r.r~rrrrrr0r1 total_loss ignored_indexr  start_lossend_lossrr-r-r.rVssN          z!MPNetForQuestionAnswering.forward) NNNNNNNNNN)r0r1r2r<rrrGrrrrrrrrVr\r-r-rKr.r+gsH      r+cCs2||}tj|dd||}||S)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. :param torch.Tensor x: :return torch.Tensor: rrn)nerdrGcumsumtype_asrY)rPr'maskincremental_indicesr-r-r.rMs rM)rr!r+rr(rrr)7rrttypingrrrGrtorch.nnrrr activationsr r modeling_outputsr r r rrrrmodeling_utilsr pytorch_utilsrrutilsrrconfiguration_mpnetr get_loggerr0loggerrModuler5r^rrrrrrrrr*rr!r(rr+rM__all__r-r-r-r.sP $   ;M/ ZYGP\AK