o wiG@sdZddlZddlmZmZddlZddlZddlmZddlm Z m Z m Z m Z ddlm ZddlmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZdd lmZmZddl m!Z!e"e#Z$d)ddZ%Gdddej&Z'Gdddej&Z(GdddeZ)eGdddeZ*eGddde*Z+eddGdd d e*eZ,ed!dGd"d#d#e*Z-eGd$d%d%e*Z.eGd&d'd'e*Z/gd(Z0dS)*zPyTorch MPT model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLoss LayerNormMSELoss) functional)GenerationMixin)!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) MptConfigcCstjd|dtj|dddd|}dtt|}tjd|dtj|d}|||}dt d|}|d|dd}||krjtj |ddddddf|ddddddfgddddd|df}||}| dS) a Link to paper: https://huggingface.co/papers/2108.12409 - Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation. This implementation has been copied from the alibi implementation of MPT source code that led to slightly different results than the Bloom alibi: https://huggingface.co/mosaicml/mpt-7b/blob/main/attention.py#L292 r)dtypedevice?N.dimr) torcharangeint32viewmathceillog2int64floatpowconcatsqueeze) num_headssequence_lengthalibi_bias_maxralibinum_heads_power_of_2baseslopesr2a/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/mpt/modeling_mpt.pybuild_mpt_alibi_tensor,s$ L r4c sZeZdZdZdeffdd Z  d dejdejdee ejd eejfd d Z Z S) MptAttentionzzMulti-head self attention. Using torch or triton attention implementation enables user to also use additive bias. configcst|j|_|j|_|j|_|j|j|_|jj|_|jdur.dt |j|j|_|jj |_ |jj |_ tj|jd|jdd|_tj|j|jdd|_dS)Nrr Fbias)super__init__ hidden_sizen_heads max_seq_lenmax_seq_lengthhead_dim attn_config softmax_scaler#sqrt attn_pdropattn_dropout_pclip_qkvrLinearWqkvout_projselfr6 __class__r2r3r:Hs     zMptAttention.__init__N hidden_states position_biaspast_key_valueattention_maskcCs4|jdd\}}||}|jr|j|j |jd}|jddd\}} } ||||j|jdd}| |||j|jdd} | |||j|jdd} |durtt |dkrot j |d| gdd} t j |d| gdd} | | f}n| | f}t || dd|j } |dur|n||djd} |durt |jdkrtd t |j| jd} td|d| }td|d| }|dd|d|df}| |} |dur| |t |jj} tjj| dd| j}tjj||j|jd }t || }|dddd||d}| |}|||fS) Nr)minmaxr rrrz6Expecting position_bias shape to be 3 dimensions, got ptraining)!shaperGrEclampchunkreshaper<r? transposelenrcatmatmulrA ValueErrorrRsize masked_fillfinforrQrr softmaxr'todropoutrDrWpermute contiguousr"rH)rJrMrNrOrP batch_size seq_length mixed_qkv query_states key_states value_statesattention_scores query_length key_lengthposition_bias_query_indexposition_bias_key_index attn_weightscontext_states attn_outputr2r2r3forwardWs@       zMptAttention.forward)NN) __name__ __module__ __qualname____doc__rr:rTensorrtuplerw __classcell__r2r2rKr3r5Cs r5cs>eZdZdeffdd ZdejdejdejfddZZS) MptMLPr6csXt|j}tj|d|dd|_tjdd|_tjd||dd|_|j j |_ dS)NFr7none) approximate) r9r:r;rrFup_projGELUact down_projr@rChidden_dropoutrJr6r;rKr2r3r:s zMptMLP.__init__rMresidualreturncCs:|||}||}tj||j|jd}||}|S)NrU)rrrFrfrrW)rJrMrintermediate_outputoutputr2r2r3rws  zMptMLP.forward) rxryrzrr:rr|rwr~r2r2rKr3rs$ rc sbeZdZdeffdd Z   ddejdejdejd eeejejfd e d e f d d Z Z S)MptBlockr6csxt|j}t||jd|_d|j_|j|_t ||_ t||jd|_ d|j _t ||_ |jj|_t|j|_dS)Neps)r9r:r;rlayer_norm_epsilonnorm_1r8r<r+r5attnnorm_2rffnr@rC dropout_raterDropoutresid_attn_dropoutrrKr2r3r:s    zMptBlock.__init__NFrMrNrP layer_past use_cacheoutput_attentionscCst||}|}|j||||d\} } } || |}||}|}|||} | f} |r1| | f7} |r8| | f7} | S)N)rNrPrO)rrrrr)rJrMrNrPrrrlayernorm_outputr attn_outputsrtrOroutputsr2r2r3rws$      zMptBlock.forward)NFF) rxryrzrr:rr|rr}boolrwr~r2r2rKr3rs$rcszeZdZeZdZdZdgZdgZfddZ de j fdd Z e d eeejejfd eeejejffd d ZZS)MptPreTrainedModel transformerTrz lm_head.*.cstj|i|dSN)r9r:)rJinputskwargsrKr2r3r:szMptPreTrainedModel.__init__modulecCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|t r\|jdurS|jj |jj ddSdS)zInitialize the weights.g)meanstdNr) isinstancerrFweightdatanormal_r6initializer_ranger8zero_ Embedding padding_idxrfill_)rJrr2r2r3 _init_weightss       z MptPreTrainedModel._init_weightsrOrcs8|ddj\}}||tfdd|DS)zw Converts the cache to the format expected by Mpt, i.e. to tuple(tuple([batch_size * num_heads, ...])) rc3s4|]}|d|dfVqdSrrN)r[.0rbatch_size_times_num_headsr?rjr2r3 s z;MptPreTrainedModel._convert_to_mpt_cache..)rXr})rOrir+r2rr3_convert_to_mpt_caches z(MptPreTrainedModel._convert_to_mpt_cache)rxryrzr config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_keys_to_ignore_on_load_missingr:rModuler staticmethodr}rr|rr~r2r2rKr3rs rcseZdZdeffdd ZddZddd Zd ejfd d Z e        dd e ej de e e ejejfdfde ejde ej de ede ede ede edee ejdfeffddZZS)MptModelr6csztj|_j|_tj|j|_t fddt j D|_ t |jjd|_d|j_d|_|dS)Ncsg|]}tqSr2)r)r_r6r2r3 sz%MptModel.__init__..rF)r9r:r;r<r+rr vocab_sizewte ModuleListrangen_layersblocksrrnorm_fr8gradient_checkpointing post_initrIrKrr3r:s   zMptModel.__init__cC|jSrrrJr2r2r3get_input_embeddings,zMptModel.get_input_embeddingsrNcCst||||Sr)r4)rJr+r,r-rr2r2r3r4/szMptModel.build_mpt_alibi_tensornew_embeddingscC ||_dSrrrJrr2r2r3set_input_embeddings2 zMptModel.set_input_embeddings input_idspast_key_values.rP inputs_embedsrroutput_hidden_states return_dictrc  KsT|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur$|n|jj}|dur4|dur4td|dur>|j\} } n|durI|j\} } } ntd|dur[tdgt|j }|durd| |}|} |rjdnd}|rpdnd}|rvdnd}|j r|j r|rt dd}| }d}|ddur|ddjd}||}|durtj| |f| jd }n|| j}|j|j|jj| jd }t|| | f||}|}t|j |D]2\}}|r|| f}|| |||||d }|d} |d ur||d f}|r|||rdnd f}q|| } |r|| f}|s"td d| |||fDSt| |||dS) input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0][0].shape[2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedsr2zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frrr)rrPrrrNTrcss|] }|dur|VqdSrr2)rvr2r2r3rsz#MptModel.forward..)last_hidden_staterrM attentions)r6rrruse_return_dictr`rXr}r]rrrrWlogger warning_onceronesrrer4r+r=r rziprr)rJrrrPrrrrrrrirjrrMpresentsall_self_attentionsall_hidden_statesseq_length_with_pastpast_key_values_lengthr. causal_maskblockrrr2r2r3rw5s             zMptModel.forwardrNNNNNNNNN)rxryrzrr:rr4rr|rrr LongTensorr}rrrrwr~r2r2rKr3rsB   rz The MPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introcseZdZdgZdeffdd ZddZdejfdd Z e dd e ej d e e e ejejfd fde ejde ejde ejde ede ede ede edee ejeffddZde e ejejfd fdej de e ejejfd ffddZZS)MptForCausalLMzlm_head.weightr6cs8t|t||_tj|j|jdd|_| dSNFr7) r9r:rrrrFr;rlm_headrrIrKr2r3r:s   zMptForCausalLM.__init__cCrrrrr2r2r3get_output_embeddingsrz$MptForCausalLM.get_output_embeddingsrcCrrrrr2r2r3set_output_embeddingsrz$MptForCausalLM.set_output_embeddingsNrr.rPrlabelsrrrrrc  Ks| dur| n|jj} |j|||||||| d} | d} || } d}|dur:|| j}|j| |fd|jji| }| sP| f| dd}|durN|f|S|St|| | j | j | j dS)aZ input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0][0].shape[2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` NrrPrrrrrrrrlosslogitsrrMr) r6rrrrer loss_functionrrrrMr)rJrrrPrrrrrrrtransformer_outputsrM lm_logitsrrr2r2r3rwsD  zMptForCausalLM.forwardpastbeam_idxcs,fdd|Dtfdd|D}|S)aL This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or [`~PreTrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct beam_idx at every generation step. Output shares the same memory storage as `past`. cs&i|]}|D] }|j|jqqSr2)rre)rr past_state)rr2r3 s z1MptForCausalLM._reorder_cache..c3sD|]}|dd|dj|dd|djfVqdSr) index_selectrr)device_to_beam_idxr2r3rs z0MptForCausalLM._reorder_cache..)r})rJrrreordered_pastr2)rrr3_reorder_caches  zMptForCausalLM._reorder_cache NNNNNNNNN)rxryrz_tied_weights_keysrr:rrr|rrrrr}rrrrwrr~r2r2rKr3rsV   Fra The MPT Model transformer with a sequence classification head on top (linear layer). [`MptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-1) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ceZdZdeffdd Ze         ddeejdee e ej ej fdfdeej d eej d eej d ee d ee d ee dee de e ej e ffddZZS)MptForSequenceClassificationr6cs@t||j|_t||_tj|j|jdd|_| dSr) r9r: num_labelsrrrrFr;scorerrIrKr2r3r:.s   z%MptForSequenceClassification.__init__Nrr.rPrrrrrrrc  Cs$| dur| n|jj} |j|||||||| d} | d} || } |dur*|jd} n|jd} |jjdur=| dkr=td|jjdurFd}n1|durk||jjk| jt j }t j |jd| jt j d}|| d}n d}t |jjd| t j | | jd |f}d}|dur|jjdur|jdkrd |j_n|jdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jdkr|||}n#|||}n|jjd krt}|||}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jd S)4 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0][0].shape[2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrrz=Cannot handle batch sizes > 1 if no padding token is defined.rS)rrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)r6rrr rX pad_token_idr`rerrr!r argmaxrrrLrx problem_typer rlongintrr*rrrrrMr)rJrrrPrrrrrrrrMrrilast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrr2r2r3rw7st        "        z$MptForSequenceClassification.forwardr)rxryrzrr:rrrrr}r|rrrrwr~r2r2rKr3r sB    r cr )MptForTokenClassificationr6cst||j|_t||_t|dr|jdur|j}nt|dr+|jdur+|j}nd}t ||_ t |j |j|_ |dS)Nclassifier_dropoutrg?)r9r:r rrhasattrrrrrrfrFr; classifierr)rJr6rrKr2r3r:s    z"MptForTokenClassification.__init__Nrr.rPrrrrrrrc  Ks| dur| n|jj} |j|||||||| d} | d} || } || } d}|durJ|| j}|j\}}t}|| |||j | ||}| s`| f| dd}|dur^|f|S|St || | j | j dS)r Nrrr)rrrMr)r6rrrfrrerrXrr"r rrMr)rJrrrPrrrrrrdeprecated_argumentsrrMrrrirjrrr2r2r3rws>    z!MptForTokenClassification.forwardr)rxryrzrr:rrrrr}r|rrrrwr~r2r2rKr3rsB   rcseZdZfddZe        ddeejdeejdeejdeejdeejd ee d ee d ee d e e e ffd dZ ZS)MptForQuestionAnsweringcs2t|t||_t|jd|_|dS)Nr) r9r:rrrrFr; qa_outputsrrIrKr2r3r:s   z MptForQuestionAnswering.__init__NrrPrstart_positions end_positionsrrrrc CsB|dur|n|jj}|j||||||d} | d} || } | jddd\} } | d} | d} d}|dur|durt|dkrL|d}t|dkrY|d}| d}| d|}| d|}t |d}|| |}|| |}||d}|s| | f| dd}|dur|f|S|St || | | j | j d S) rN)rPrrrrrrrSr) ignore_indexr)r start_logits end_logitsrMr)r6rrr!splitr*rhr]rarYrrrMr)rJrrPrr"r#rrrrsequence_outputrr%r& total_loss ignored_indexr start_lossend_lossrr2r2r3rwsJ          zMptForQuestionAnswering.forwardr)rxryrzr:rrrr FloatTensorrrr}rrwr~r2r2rKr3r s<   r )rrrr rr r)1r{r#typingrrrtorch.utils.checkpointrtorch.nnrrrrr r generationr modeling_attn_mask_utilsr modeling_layersr modeling_outputsrrrrrmodeling_utilsrutilsrrconfiguration_mptr get_loggerrxrr4rr5rrrrrr rr __all__r2r2r2r3sL         L@/prX R