o ei@sdZddlZddlZddlmZddlmZmZmZmZddlm Z ddl m Z m Z ddlmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZdd lmZmZddlm Z e!e"Z#d)ddZ$Gdddej%Z&Gdddej%Z'GdddeZ(eGdddeZ)eGddde)Z*eddGdd d e)eZ+ed!dGd"d#d#e)Z,eGd$d%d%e)Z-eGd&d'd'e)Z.gd(Z/dS)*zPyTorch MPT model.N)nn)BCEWithLogitsLossCrossEntropyLoss LayerNormMSELoss) functional)Cache DynamicCache)GenerationMixin)create_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) MptConfigcCstjd|dtj|dddd|}dtt|}tjd|dtj|d}|||}dt d|}|d|dd}||krjtj |ddddddf|ddddddfgddddd|df}||}| dS) a Link to paper: https://huggingface.co/papers/2108.12409 - Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation. This implementation has been copied from the alibi implementation of MPT source code that led to slightly different results than the Bloom alibi: https://huggingface.co/mosaicml/mpt-7b/blob/main/attention.py#L292 r)dtypedeviceg?N.dimr) torcharangeint32viewmathceillog2int64floatpowconcatsqueeze) num_headssequence_lengthalibi_bias_maxralibinum_heads_power_of_2baseslopesr1b/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/mpt/modeling_mpt.pybuild_mpt_alibi_tensor*s$ L r3c sjeZdZdZddededBffdd Z   ddejdejd e dBd ejdBd ejdBf d d Z Z S) MptAttentionzzMulti-head self attention. Using torch or triton attention implementation enables user to also use additive bias. Nconfig layer_idxcst|j|_|j|_|j|_|j|j|_|jj|_|jdur.dt |j|j|_|jj |_ |jj |_ tj|jd|jdd|_tj|j|jdd|_||_dS)NrrFbias)super__init__ hidden_sizen_heads max_seq_lenmax_seq_lengthhead_dim attn_config softmax_scaler"sqrt attn_pdropattn_dropout_pclip_qkvrLinearWqkvout_projr6)selfr5r6 __class__r1r2r:Fs      zMptAttention.__init__ hidden_states position_biaspast_key_valuesattention_maskcache_positioncCs|jdd\}}||}|jr|j|j |jd}|jddd\} } } | |||j|jdd} | |||j|jdd} | |||j|jdd} |durbd|i} | | | |j | \} } t | | dd|j } |duru|n||}|durt|jdkrtd t|j| jd}td |d|}td |d|}|dd|d|df}| |} |dur| |t | jj} tjj| dd| j}tjj||j|jd }t || }|d ddd !||d}|"|}||fS) Nr)minmaxrrrrPz6Expecting position_bias shape to be 3 dimensions, got rptraining)#shaperGrEclampchunkreshaper<r? transposeupdater6rmatmulrAget_seq_lengthlen ValueErrorrRsize masked_fillfinforrQrrsoftmaxr&todropoutrDrWpermute contiguousr!rH)rIrLrMrNrOrP batch_size seq_length mixed_qkv query_states key_states value_states cache_kwargsattention_scores query_length key_lengthposition_bias_query_indexposition_bias_key_index attn_weightscontext_states attn_outputr1r1r2forwardVs:    zMptAttention.forwardN)NNN) __name__ __module__ __qualname____doc__rintr:rTensorr ry __classcell__r1r1rJr2r4As"r4cs>eZdZdeffdd ZdejdejdejfddZZS) MptMLPr5csXt|j}tj|d|dd|_tjdd|_tjd||dd|_|j j |_ dS)NFr7none) approximate) r9r:r;rrFup_projGELUact down_projr@rChidden_dropout)rIr5r;rJr1r2r:s zMptMLP.__init__rLresidualreturncCs:|||}||}tj||j|jd}||}|S)NrU)rrrFrgrrW)rIrLrintermediate_outputoutputr1r1r2rys  zMptMLP.forward) r{r|r}rr:rrryrr1r1rJr2rs$ rcsleZdZddededBffdd Z    ddejdejd ejd edBd e d e d ejdBfddZ Z S)MptBlockNr5r6cszt|j}t||jd|_d|j_|j|_t |||_ t||jd|_ d|j _t ||_ |jj|_t|j|_dS)Neps)r9r:r;rlayer_norm_epsilonnorm_1r8r<r*r4attnnorm_2rffnr@rC dropout_raterDropoutresid_attn_dropout)rIr5r6r;rJr1r2r:s    zMptBlock.__init__FrLrMrO layer_past use_cacheoutput_attentionsrPc CsV||}|} |j|||||d\} } || | }||}|} ||| } | | fS)N)rMrOrNrP)rrrrr) rIrLrMrOrrrrPlayernorm_outputr attn_outputsrvrr1r1r2rys    zMptBlock.forwardrz)NFFN) r{r|r}rrr:rrr boolryrr1r1rJr2rs*rc@s$eZdZUeed<dZdZdgZdS)MptPreTrainedModelr5 transformerTrN)r{r|r}r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesr1r1r1r2rs  rcseZdZdeffdd ZddZddd Zd ejfd d Z e         dd ej dBde dBdejdBdej dBde dBde dBde dBde dBdejdBdeejdfeBfddZZS)MptModelr5csztj|_j|_tj|j|_t fddt j D|_ t |jjd|_d|j_d|_|dS)Ncsg|]}t|dqS))r6)r).0ir5r1r2 sz%MptModel.__init__..rF)r9r:r;r<r*r Embedding vocab_sizewte ModuleListrangen_layersblocksrrnorm_fr8gradient_checkpointing post_initrIr5rJrr2r:s   zMptModel.__init__cCs|jSrzr)rIr1r1r2get_input_embeddingsszMptModel.get_input_embeddingsrNcCst||||Srz)r3)rIr*r+r,rr1r1r2r3szMptModel.build_mpt_alibi_tensornew_embeddingscC ||_dSrzrrIrr1r1r2set_input_embeddings zMptModel.set_input_embeddings input_idsrNrO inputs_embedsrroutput_hidden_states return_dictrPr.c  Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur$|n|jj}|dur4|dur4td|dur>|j\} } n|durI|j\} } } ntd|jr\|jr\|r\t dd}|dure| |}|rq|durqt |jd}|}|rwdnd}|r}dnd}|j |j|jj|jd}|dur|nd }| durtj||| |jd} t|j||| |d tj}|jD]!}|r||f}|||||||| d }|d }|r||d f}q||}|r||f}|std d||||fDSt||||dS) input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frr1rr)r5rrOrPrN)rrOrrrMrPrcss|] }|dur|VqdSrzr1)rvr1r1r2 isz#MptModel.forward..)last_hidden_staterNrL attentions)r5rrruse_return_dictrarXrrWlogger warning_oncerr r3r*r=rr_rrr rfrrrtupler)rIrrNrOrrrrrrPkwargsrjrk_rLall_self_attentionsall_hidden_statesr-past_key_values_length causal_maskblockoutputsr1r1r2rys              zMptModel.forwardrN NNNNNNNNN)r{r|r}rr:rr3rrrr LongTensorr rrrryrr1r1rJr2rsH    rz The MPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introcseZdZddiZdeffdd ZdejfddZe dd ej d Bd e d Bd ejd Bdejd Bdejd Bde d Bde d Bde d Bde d Bdejd Bde ejBdeejeBfddZZS)MptForCausalLMzlm_head.weightztransformer.wte.weightr5cs8t|t||_tj|j|jdd|_| dSNFr7) r9r:rrrrFr;rlm_headrrrJr1r2r:~s   zMptForCausalLM.__init__rcCrrz)rrr1r1r2set_output_embeddingsrz$MptForCausalLM.set_output_embeddingsNrrrNrOrlabelsrrrrrPlogits_to_keeprc  Ks| dur| n|jj} |j|||||||| | d } | d}t| tr't| dn| }||dd|ddf}d}|durK|jd|||jjd| }| sa|f| dd}|dur_|f|S|St ||| j | j | j dS)a\ input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N)rNrOrrrrrrPr)logitsrrrlossrrNrLrr1) r5rr isinstancerslicer loss_functionrrrNrLr)rIrrNrOrrrrrrrPrrtransformer_outputsrL slice_indicesrrrr1r1r2rys8! zMptForCausalLM.forward) NNNNNNNNNNr)r{r|r}_tied_weights_keysrr:rrrrrr rrrrryrr1r1rJr2rusR     ra The MPT Model transformer with a sequence classification head on top (linear layer). [`MptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-1) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cseZdZdeffdd ZdejfddZe         ddej dBd e dBd ejdBd ejdBd ejdBd e dBde dBde dBde dBde eje BfddZZS)MptForSequenceClassificationr5cs@t||j|_t||_tj|j|jdd|_| dSr) r9r: num_labelsrrrrFr;scorerrrJr1r2r:s   z%MptForSequenceClassification.__init__rcCrrz)rrr1r1r2rrz2MptForSequenceClassification.set_output_embeddingsNrrNrOrrrrrrrc  Ks$| dur| n|jj} |j|||||||| d} | d} || } |dur*|jd}n|jd}|jjdur=|dkr=td|jjdurFd}n1|durk||jjk| jt j }t j |jd| jt j d}|| d}n d}t |jjd| t j || jd |f}d}|dur|jjdur|jdkrd |j_n|jdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jdkr|||}n#|||}n|jjd krt}|||}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jd S)6 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrNrOrrrrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.rS)rrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)r5rrrrX pad_token_idrarfrrr rargmaxrrrKr{ problem_typerrlongrrr)rrrrNrLr)rIrrNrOrrrrrrrrrLrrjlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrr1r1r2ryst        "        z$MptForSequenceClassification.forwardr)r{r|r}rr:rrrrrr rrrryrr1r1rJr2rsD    rcseZdZdeffdd Ze         ddejdBdedBdej dBdej dBd ej dBd e dBd e dBd e dBd e dBde ej e BfddZ ZS)MptForTokenClassificationr5cst||j|_t||_t|dr|jdur|j}nt|dr+|jdur+|j}nd}t ||_ t |j |j|_ |dS)Nclassifier_dropoutrg?)r9r:rrrhasattrrrrrrgrFr; classifierr)rIr5rrJr1r2r:Ts    z"MptForTokenClassification.__init__NrrNrOrrrrrrrc  Ks| dur| n|jj} |j|||||||| d} | d} || } || } d}|durJ|| j}|j\}}t}|| |||j | ||}| s`| f| dd}|dur^|f|S|St || | j | j dS)rNrrr)rrrLr)r5rrrgrrfrrXrr!rrrLr)rIrrNrOrrrrrrdeprecated_argumentsrrLrrrjrkrrr1r1r2ryes>    z!MptForTokenClassification.forwardr)r{r|r}rr:rrrr rrrrryrr1r1rJr2rRsB   rcseZdZfddZe        ddejdBdejdBdejdBdejdBdejdBd edBd edBd edBd e e Bfd dZ Z S)MptForQuestionAnsweringcs2t|t||_t|jd|_|dS)Nr) r9r:rrrrFr; qa_outputsrrrJr1r2r:s   z MptForQuestionAnswering.__init__NrrOrstart_positions end_positionsrrrrc KsB|dur|n|jj}|j||||||d} | d} || } | jddd\} }| d} |d}d}|dur|durt|dkrL|d}t|dkrY|d}| d}| d|}| d|}t |d}|| |}|||}||d}|s| |f| dd}|dur|f|S|St || || j | j d S) rN)rOrrrrrrrSr) ignore_indexr)r start_logits end_logitsrLr)r5rrrsplitr)rir`rbrYrrrLr)rIrrOrrrrrrrrsequence_outputrrr total_loss ignored_indexr start_lossend_lossrr1r1r2rysJ          zMptForQuestionAnswering.forward)NNNNNNNN) r{r|r}r:rrr FloatTensorrrrryrr1r1rJr2rs<   r)rrrrrrr)0r~r"rrtorch.nnrrrrrr cache_utilsr r generationr masking_utilsr modeling_layersr modeling_outputsrrrrrmodeling_utilsrutilsrrconfiguration_mptr get_loggerr{rr3Moduler4rrrrrrrr__all__r1r1r1r2sJ          I:SvX S