o ib@sdZddlZddlmZmZddlZddlmZddlmZm Z m Z m Z ddlm Z ddlmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZddlm Z m!Z!ddl"m#Z#ddl$m%Z%e!&e'Z(d+ddZ)Gdddej*Z+Gdddej*Z,GdddeZ-e GdddeZ.e Gddde.Z/e dd Gd!d"d"e.eZ0e d#d Gd$d%d%e.Z1e Gd&d'd'e.Z2e Gd(d)d)e.Z3gd*Z4dS),zPyTorch MPT model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLoss LayerNormMSELoss) functional)Cache DynamicCache)GenerationMixin)!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)auto_docstringlogging)deprecate_kwarg) MptConfigcCstjd|dtj|dddd|}dtt|}tjd|dtj|d}|||}dt d|}|d|dd}||krjtj |ddddddf|ddddddfgddddd|df}||}| dS) a Link to paper: https://huggingface.co/papers/2108.12409 - Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation. This implementation has been copied from the alibi implementation of MPT source code that led to slightly different results than the Bloom alibi: https://huggingface.co/mosaicml/mpt-7b/blob/main/attention.py#L292 r)dtypedevice?N.dimr) torcharangeint32viewmathceillog2int64floatpowconcatsqueeze) num_headssequence_lengthalibi_bias_maxralibinum_heads_power_of_2baseslopesr5a/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/mpt/modeling_mpt.pybuild_mpt_alibi_tensor-s$ L r7c sxeZdZdZddedeeffdd Zeddd d    dd e j d e j dee d ee j dee j f ddZ Z S) MptAttentionzzMulti-head self attention. Using torch or triton attention implementation enables user to also use additive bias. Nconfig layer_idxcst|j|_|j|_|j|_|j|j|_|jj|_|jdur.dt |j|j|_|jj |_ |jj |_ tj|jd|jdd|_tj|j|jdd|_||_dS)Nrr Fbias)super__init__ hidden_sizen_heads max_seq_lenmax_seq_lengthhead_dim attn_config softmax_scaler&sqrt attn_pdropattn_dropout_pclip_qkvrLinearWqkvout_projr:)selfr9r: __class__r5r6r>Is      zMptAttention.__init__past_key_valuepast_key_values4.58new_nameversion hidden_states position_biasattention_maskcache_positioncCs|jdd\}}||}|jr|j|j |jd}|jddd\} } } | |||j|jdd} | |||j|jdd} | |||j|jdd} |durbd|i} | | | |j | \} } t | | dd|j } |duru|n||}|durt|jdkrtd t|j| jd}td |d|}td |d|}|dd|d|df}| |} |dur| |t | jj} tjj| dd| j}tjj||j|jd }t || }|d ddd !||d}|"|}||fS) Nr)minmaxr r rrYz6Expecting position_bias shape to be 3 dimensions, got rptraining)#shaperKrIclampchunkreshaper@rC transposeupdater:r"matmulrEget_seq_lengthlen ValueErrorr[size masked_fillfinforrZrr softmaxr*todropoutrHr`permute contiguousr%rL)rMrVrWrQrXrY batch_size seq_length mixed_qkv query_states key_states value_states cache_kwargsattention_scores query_length key_lengthposition_bias_query_indexposition_bias_key_index attn_weightscontext_states attn_outputr5r5r6forwardYs:    zMptAttention.forwardN)NNN)__name__ __module__ __qualname____doc__rrintr>rr"Tensorr r __classcell__r5r5rNr6r8Ds$ r8cs>eZdZdeffdd ZdejdejdejfddZZS) MptMLPr9csXt|j}tj|d|dd|_tjdd|_tjd||dd|_|j j |_ dS)NFr;none) approximate) r=r>r?rrJup_projGELUact down_projrDrGhidden_dropout)rMr9r?rNr5r6r>s zMptMLP.__init__rVresidualreturncCs:|||}||}tj||j|jd}||}|S)Nr^)rrrFrprr`)rMrVrintermediate_outputoutputr5r5r6rs  zMptMLP.forward) rrrrr>r"rrrr5r5rNr6rs$ rcsleZdZddedeeffdd Z    ddejdejd ejd ee d e d e d eejfddZ Z S)MptBlockNr9r:cszt|j}t||jd|_d|j_|j|_t |||_ t||jd|_ d|j _t ||_ |jj|_t|j|_dS)Neps)r=r>r?rlayer_norm_epsilonnorm_1r<r@r.r8attnnorm_2rffnrDrG dropout_raterDropoutresid_attn_dropout)rMr9r:r?rNr5r6r>s    zMptBlock.__init__FrVrWrX layer_past use_cacheoutput_attentionsrYc CsV||}|} |j|||||d\} } || | }||}|} ||| } | | fS)N)rWrXrQrY)rrrrr) rMrVrWrXrrrrYlayernorm_outputr attn_outputsrrr5r5r6rs    zMptBlock.forwardr)NFFN) rrrrrrr>r"rr boolrrr5r5rNr6rs*rc seZdZUeed<dZdZdgZdgZfddZ de j fd d Z e ed d d dd eeejejfdeeejejffddZZS)MptPreTrainedModelr9 transformerTrz lm_head.*.cstj|i|dSr)r=r>)rMinputskwargsrNr5r6r>szMptPreTrainedModel.__init__modulecCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|t r\|jdurS|jj |jj ddSdS)zInitialize the weights.g)meanstdNr) isinstancerrJweightdatanormal_r9initializer_ranger<zero_ Embedding padding_idxrfill_)rMrr5r5r6 _init_weightss       z MptPreTrainedModel._init_weightsrPrQrRrSrcs8|ddj\}}||tfdd|DS)zw Converts the cache to the format expected by Mpt, i.e. to tuple(tuple([batch_size * num_heads, ...])) rc3s4|]}|d|dfVqdS)rrN)rd).0rbatch_size_times_num_headsrCrtr5r6 s z;MptPreTrainedModel._convert_to_mpt_cache..)ratuple)rQrsr.r5rr6_convert_to_mpt_caches z(MptPreTrainedModel._convert_to_mpt_cache)rrrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_keys_to_ignore_on_load_missingr>rModuler staticmethodrrr"rrrr5r5rNr6rs   rcseZdZdeffdd ZddZddd Zd ejfd d Z e         dd e ej de e de ejde ej de ede ede ede ede ejdeeejdfeffddZZS)MptModelr9csztj|_j|_tj|j|_t fddt j D|_ t |jjd|_d|j_d|_|dS)Ncsg|]}t|dqS))r:)r)rir9r5r6 sz%MptModel.__init__..rF)r=r>r?r@r.rr vocab_sizewte ModuleListrangen_layersblocksrrnorm_fr<gradient_checkpointing post_initrMr9rNrr6r>s   zMptModel.__init__cCs|jSrr)rMr5r5r6get_input_embeddings&szMptModel.get_input_embeddingsrNcCst||||Sr)r7)rMr.r/r0rr5r5r6r7)szMptModel.build_mpt_alibi_tensornew_embeddingscC ||_dSrrrMrr5r5r6set_input_embeddings, zMptModel.set_input_embeddings input_idsrQrX inputs_embedsrroutput_hidden_states return_dictrYr.c  Ks*|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur$|n|jj}|dur4|dur4td|dur>|j\} } n|durI|j\} } } ntd|jr\|jr\|r\t dd}|dure| |}|rq|durqt |jd}|rt |trt dt |}|}|rdnd}|rdnd}|dur|nd }| |}|durtj| |f|jd }n||j}|j|j|jj|jd }t|| | f||}|}|jD]!}|r||f}|||||||| d }|d }|r||d f}q||}|r||f}|s td d||||fDSt||||dS) input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FrzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `DynamicCache` instead, e.g. `past_key_values=DynamicCache.from_legacy_cache(past_key_values)`.r5rr)rrXrrrWrYrcss|] }|dur|VqdSrr5)rvr5r5r6rsz#MptModel.forward..)last_hidden_staterQrV attentions)r9rrruse_return_dictrjrarr`logger warning_oncerr rrfrom_legacy_cacherhr"onesrror7r.rArrrrr)rMrrQrXrrrrrrYrrsrt_rVall_self_attentionsall_hidden_statespast_key_values_lengthseq_length_with_pastr1 causal_maskblockoutputsr5r5r6r/s                zMptModel.forwardrN NNNNNNNNN)rrrrr>rr7r"rrrr LongTensorr rrrrrrr5r5rNr6rsH    rz The MPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introcseZdZdgZdeffdd ZdejfddZe          dd e ej d e e d e ejd e ejd e ejde e de e de e de e de ejdeeejeffddZZS)MptForCausalLMzlm_head.weightr9cs8t|t||_tj|j|jdd|_| dSNFr;) r=r>rrrrJr?rlm_headrrrNr5r6r>s   zMptForCausalLM.__init__rcCrr)rrr5r5r6set_output_embeddingsrz$MptForCausalLM.set_output_embeddingsNrrQrXrlabelsrrrrrYrc  Ks| dur| n|jj} |j|||||||| | d } | d} || }d}|dur;||j}|j||fd|jji| }| sQ|f| dd}|durO|f|S|St||| j | j | j dS)a\ input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N)rQrXrrrrrrYrrrlosslogitsrQrVr) r9rrrror loss_functionrrrQrVr)rMrrQrXrrrrrrrYrtransformer_outputsrV lm_logitsrrr5r5r6rsF   zMptForCausalLM.forward) NNNNNNNNNN)rrr_tied_weights_keysrr>r"rrrrrr rrrrrrr5r5rNr6rsL    ra The MPT Model transformer with a sequence classification head on top (linear layer). [`MptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-1) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ceZdZdeffdd Ze         ddeejdee deej deej d eej d ee d ee d ee d ee de e ej effddZZS)MptForSequenceClassificationr9cs@t||j|_t||_tj|j|jdd|_| dSr) r=r> num_labelsrrrrJr?scorerrrNr5r6r>s   z%MptForSequenceClassification.__init__NrrQrXrrrrrrrc  Cs$| dur| n|jj} |j|||||||| d} | d} || } |dur*|jd} n|jd} |jjdur=| dkr=td|jjdurFd}n1|durk||jjk| jt j }t j |jd| jt j d}|| d}n d}t |jjd| t j | | jd |f}d}|dur|jjdur|jdkrd |j_n|jdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jdkr|||}n#|||}n|jjd krt}|||}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jd S)6 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrQrXrrrrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r\)rrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)r9rrrra pad_token_idrjrorr"r$r#argmaxrrrOr problem_typerrlongrrr-rrrrQrVr)rMrrQrXrrrrrrrrVrrslast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrr5r5r6rst        "        z$MptForSequenceClassification.forwardr)rrrrr>rrr"rr rrrrrrrr5r5rNr6rsB    rcr)MptForTokenClassificationr9cst||j|_t||_t|dr|jdur|j}nt|dr+|jdur+|j}nd}t ||_ t |j |j|_ |dS)Nclassifier_dropoutrg?)r=r>rrrhasattrrrrrrprJr? classifierr)rMr9rrNr5r6r>s    z"MptForTokenClassification.__init__NrrQrXrrrrrrrc  Ks| dur| n|jj} |j|||||||| d} | d} || } || } d}|durJ|| j}|j\}}t}|| |||j | ||}| s`| f| dd}|dur^|f|S|St || | j | j dS)rNrrr)rrrVr)r9rrrprrorrarr%rrrVr)rMrrQrXrrrrrrdeprecated_argumentsrrVrrrsrtrrr5r5r6rs>    z!MptForTokenClassification.forwardr)rrrrr>rrr"rr rrrrrrrr5r5rNr6rsB   rcseZdZfddZe        ddeejdeejdeejdeejdeejd ee d ee d ee d e e e ffd dZ ZS)MptForQuestionAnsweringcs2t|t||_t|jd|_|dS)Nr) r=r>rrrrJr? qa_outputsrrrNr5r6r>s   z MptForQuestionAnswering.__init__NrrXrstart_positions end_positionsrrrrc CsB|dur|n|jj}|j||||||d} | d} || } | jddd\} } | d} | d} d}|dur|durt|dkrL|d}t|dkrY|d}| d}| d|}| d|}t |d}|| |}|| |}||d}|s| | f| dd}|dur|f|S|St || | | j | j d S) rN)rXrrrrrrr\r ) ignore_indexr)r start_logits end_logitsrVr)r9rrrsplitr-rrrirkrbrrrVr)rMrrXrrrrrrrsequence_outputrrr  total_loss ignored_indexr start_lossend_lossrr5r5r6rsJ          zMptForQuestionAnswering.forward)NNNNNNNN)rrrr>rrr"r FloatTensorrrrrrrr5r5rNr6rs<   r)rrrrrrr)5rr&typingrrr"rtorch.nnrrrrr r cache_utilsr r generationr modeling_attn_mask_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_utilsrutilsrrutils.deprecationrconfiguration_mptr get_loggerrrr7rr8rrrrrrrr__all__r5r5r5r6sN          J:0XrX R