o iA@s.dZddlZddlmZmZddlZddlmZddlmZm Z m Z ddl m Z ddl mZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZmZdd l m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(e$)e*Z+dej,de-de-fddZ.Gdddej/Z0Gdddej1Z2GdddeZ3GdddeZ4Gdd d ej1Z5Gd!d"d"ej1Z6e#Gd#d$d$e!Z7Gd%d&d&e7Z8Gd'd(d(e7Z9e#Gd)d*d*e7Z:e#d+d,Gd-d.d.e7eZ;e#d/d,Gd0d1d1e7ZGd6d7d7e7eZ?gd8Z@dS)9zPyTorch MVP model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutput)PreTrainedModel)auto_docstringlogging)deprecate_kwarg) MvpConfig input_ids pad_token_iddecoder_start_token_idcCsh||j}|ddddf|ddddf<||dddf<|dur*td||dk||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)rrr shifted_input_idsr(a/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/mvp/modeling_mvp.pyshift_tokens_right3s (r*csPeZdZdZdedeffdd Z d dejd ed eejffd d Z Z S)MvpLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcsd|_t||j|dSN)offsetsuper__init__)selfr,r- __class__r(r)r2Isz&MvpLearnedPositionalEmbedding.__init__rNrpast_key_values_length position_idscs\|dur |jdd\}}tj|||tj|jjd|d}n|d}t ||j S)z3`input_ids' shape is expected to be [bsz x seqlen].Nr/)dtypedevicer!r) r#torcharangelongweightr9expand unsqueezer1forwardr0)r3rr6r7bszseq_lenr4r(r)r@Os z%MvpLearnedPositionalEmbedding.forward)rN) __name__ __module__ __qualname____doc__intr2r:Tensorrr@ __classcell__r(r(r4r)r+Dsr+cseZdZdZ    ddededeed eed eed eef fd d Ze dddd       dde j dee j dee dee j dee j dee j dedee j de e j ee j ee e j ffddZZS) MvpAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsdropout is_decoderbias layer_idxcst||_||_||_|||_|j||jkr'td|jd|d|jd|_||_||_ t j |||d|_ t j |||d|_ t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rP)r1r2rLrMrNhead_dimr%scalingrOrQrLineark_projv_projq_projout_proj)r3rLrMrNrOrPrQr4r(r)r2bs$    zMvpAttention.__init__past_key_valuepast_key_values4.58new_nameversion hidden_stateskey_value_statesattention_masklayer_head_mask attn_promptoutput_attentionscache_positionreturnc Cs|du} |\} } } |||j} d}|dur1t|tr/|j|j}| r+|j}n|j }n|}| r5|n|}| rN|durN|rN|j |jj }|j |jj }nJ| |}||}|| d|j|jdd}|| d|j|jdd}|dur| s||nd}||||jd|i\}}| rt|trd|j|j<|durtj|d| ddd|gdd }tj|d| ddd|gdd }|durt| d| |dd|j}tj||gdd }| |jd|jf}| | | |j|jdd} | j|} |j|}|j|}|d}t| |dd}|| |j| |fkr0td | |j| |fd ||dure|| d| |fkrPtd | d| |fd ||| |j| ||}|| |j| |}tjj|dd }|dur||jfkrtd |jfd ||dddd|| |j| |}|| |j| |}|r|| |j| |}|| |j| |}nd}tjj ||j |j!d}t||}|| |j| |jfkrtd| |j| |jfd ||| |j| |j}|dd}|| | |j"}|#|}||fS)z#Input shape: Batch x Time x ChannelNFr!rr/rfTrdimz$Attention weights should be of size z , but is z!Attention mask should be of size z/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )$sizerXrT isinstancer is_updatedgetrQcross_attention_cacheself_attention_cachelayerskeysvaluesrVrWviewrMrS transposeupdater:catr>zerostor9reshapebmmr%r functionalsoftmaxrNrlrLrY)r3r`rar[rbrcrdrerfis_cross_attentionrAtgt_len_ query_statesrocurr_past_key_valuecurrent_states key_states value_states prompt_mask proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputr(r(r)r@s      """       "   zMvpAttention.forward)rKFTN)NNNNNFN)rCrDrErFrGrfloatboolr2rr:rHr tupler@rIr(r(r4r)rJ_s\   rJcsdeZdZdeffdd Z d dejdejdejdejd eed e ejeejff d d Z Z S)MvpEncoderLayerconfigcst|j|_t|j|j|jd|_t |j|_ |j |_ t |j |_|j|_t|j|j|_t|j|j|_t |j|_dS)N)rLrMrN)r1r2d_modelrLrJencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrNr activation_function activation_fnactivation_dropoutrUencoder_ffn_dimfc1fc2final_layer_normr3rr4r(r)r2s  zMvpEncoderLayer.__init__Fr`rbrcself_attn_promptrergc Cs|}|j|||||d\}}tjj||j|jd}||}||}|}|||}tjj||j|jd}| |}tjj||j|jd}||}| |}|j t j krvt |sdt |rvt |j jd}t j|| |d}||fS)a@ Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, encoder_attention_heads, pro_len, head_dim)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r`rbrcrdrerji)minmax)rrr~rNrlrrrrrrr8r:float16isinfanyisnanfinforclamp) r3r`rbrcrreresidualr clamp_valuer(r(r)r@s4       zMvpEncoderLayer.forward)F) rCrDrErr2r: FloatTensorrrrr@rIr(r(r4r)rs rc!seZdZddeffdd Zedddd         dd ejd eejd eejdeejdeejdeejdeejdeejdee dee dee deejde ej ee ej ej fffddZ ZS)MvpDecoderLayerNrcst|j|_t|j|j|jd|d|_|j|_t |j |_ |j |_ t |j|_t|j|j|jd|d|_t |j|_t |j|j|_t |j|j|_t |j|_dS)NT)rLrMrNrOrQ)rNrOrQ)r1r2rrLrJdecoder_attention_headsrrrNr rrrrrr encoder_attnencoder_attn_layer_normrUdecoder_ffn_dimrrr)r3rrQr4r(r)r2Ds0  zMvpDecoderLayer.__init__rZr[r\r]FTr`rbencoder_hidden_statesencoder_attention_maskrccross_attn_layer_head_maskrcross_attn_promptre use_cacherfrgc  Cs|} |j|| |||| | d\}}tjj||j|jd}| |}||}d}|durN|} |j|||||| | d\}}tjj||j|jd}| |}||}|} || |}tjj||j |jd}| |}tjj||j|jd}| |}| |}|f}| r|||f7}|S)a1 Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. cross_attn_prompt (`torch.FloatTensor`): prompt of cross attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r`r[rbrcrdrerfrjN)r`rarbrcrdr[re) rrr~rNrlrrrrrrrr)r3r`rbrrrcrrrr[rerrfrself_attn_weightscross_attn_weightsoutputsr(r(r)r@`sN&         zMvpDecoderLayer.forwardN) NNNNNNNNFTN)rCrDrErr2rr:rHrr rrrr@rIr(r(r4r)rCsR      rcsHeZdZdZdedededeffdd Zdejd ejfd d Z Z S) MvpClassificationHeadz-Head for sentence-level classification tasks. input_dim inner_dim num_classespooler_dropoutcs8tt|||_tj|d|_t|||_dS)Nrk)r1r2rrUdenseDropoutrNrY)r3rrrrr4r(r)r2s zMvpClassificationHead.__init__r`rgcCs6||}||}t|}||}||}|Sr)rNrr:tanhrY)r3r`r(r(r)r@s     zMvpClassificationHead.forward) rCrDrErFrGrr2r:rHr@rIr(r(r4r)rs rcs:eZdZdZfddZdejdeejfddZZ S) MvpPromptz)Layer-wise prompt for encoder or decoder.c st|j|_||_||_|j||_tj|j d|_ t |j|j|_ t t |j|jtt |j|d|j|_dS)Nrr/)r1r2 prompt_length num_layersrMrrSrrrN Embeddingprompt_embedding SequentialrUprompt_mid_dimGELU prompt_trans)r3rrrMr4r(r)r2s   zMvpPrompt.__init__ prompt_idsrgcCsN|||}||j|jd|j|j}||}|gd d}|S)Nr/)rr/rr) rrrvrrrMrSrNpermutesplit)r3rpromptr(r(r)r@s  zMvpPrompt.forward) rCrDrErFr2r:rHrr@rIr(r(r4r)rs "rc@s2eZdZUeed<dZdZddZeddZ dS) MvpPreTrainedModelrmodelTcCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|tj rA|jjjd|d|j durC|jj|j  dSdSdS)NrK)meanstd) rinit_stdrnrrUr=datanormal_rPzero_r padding_idx)r3modulerr(r(r) _init_weightss    z MvpPreTrainedModel._init_weightscCs>|jj}tjgddddd|gg|jd}|||d}|S)N)r r/r r/r9)rbr)rrr:tensorr9ne)r3 pad_tokenr dummy_inputsr(r(r)rs "zMvpPreTrainedModel.dummy_inputsN) rCrDrEr__annotations__base_model_prefixsupports_gradient_checkpointingrpropertyrr(r(r(r)rs  rcseZdZdZ ddedeejdeeffdd Z       dd ee j d ee j d ee j d ee j d eedeedeedeeeffddZZS) MvpEncodera Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`MvpEncoderLayer`]. Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt NFr embed_tokens use_promptcstj|_j|_j}j|_j|_ j r!t |nd|_ |dur,||_n tj||j|_tj||_tfddtjD|_t||_||_|rej|_tjj|_d|_| dS)N?csg|]}tqSr()r).0rrr(r) (sz'MvpEncoder.__init__..F)!r1r2rNencoder_layerdrop layerdroprrrmax_position_embeddingsmax_source_positionsscale_embeddingmathsqrt embed_scalerrr vocab_sizer+embed_positions ModuleListrangeencoder_layersrsrlayernorm_embeddingrrrrrgradient_checkpointing post_init)r3rrrrLr4rr)r2s4    zMvpEncoder.__init__rrb head_mask inputs_embedsreoutput_hidden_states return_dictrgcCsb|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur<|}|j} |d| d}n|durT|dd} |dddddf}ntd|durd|||j }| |} || } | | } t j j| |j|jd} |jrt|j|j} || } |durt||j}|rdnd}|rdnd}|dur|dt|jkrtdt|jd |dd t|jD]G\}}|r|| f}d }|jrtg}||jkrd }|rd }n|| ||dur||nd|jr| |nd|d}|d} |r||df}q|r|| f}|s*tdd| ||fDSt| ||dS)a~ Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timer!z5You have to specify either input_ids or inputs_embedsrjr(rz&The head_mask should be specified for  layers, but it is for .FT)NN)rcrrercs|] }|dur|VqdSrr(rvr(r(r) sz%MvpEncoder.forward..last_hidden_stater` attentions) rreruse_return_dictr%r#rvrmrrrrrr~rNrlrr:r;rr{r9rrr8lenrs enumeraterandrrr)r3rrbrrrerrinput input_shape embed_posr`rrencoder_statesall_attentionsidx encoder_layerto_dropdropout_probability layer_outputsr(r(r)r@8s|.           zMvpEncoder.forwardNF)NNNNNNN)rCrDrErFrrrrrr2r: LongTensorrHrrrrr@rIr(r(r4r)rsD ( rcseZdZdZ ddedeejdeeffdd Z             dd ee j d ee j d ee j d ee j d ee j dee j deedee j deedeedeedeedee j deeeffddZZS) MvpDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`MvpDecoderLayer`] Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt NFrrrcstj|_j|_j|_j|_j rt j nd|_ |dur*||_n tjj |j|_tjj |_tfddtjD|_tj |_||_|roj|_tjj|_tjj|_d|_ |!dS)Nrcsg|]}t|dqS))rQ)r)rirr(r)rsz'MvpDecoder.__init__..F)"r1r2rNdecoder_layerdroprrrrmax_target_positionsrrrrrrrrrr+rrrdecoder_layersrsrrrrrrrrrr)r3rrrr4rr)r2s<   zMvpDecoder.__init__rrbrrrcross_attn_head_maskr[rrrerrrfrgcCs| dur| n|jj} | dur| n|jj} | dur| n|jj} | dur$| n|jj} |dur4|dur4td|durF|}|j}|d|d}n|dur^|dd}|dddddf}ntd|durn| ||j }|j r}|j r}| r}t dd} | r|dur|durtt|jdt|jdnt|jd}| rt|trt dt|}|dur|nd }t||||}|dur|durt||j|dd }|||}||}||}tjj||j|j d }|jrt|j !|j"}|#|}|$|}| rd nd}| r d nd}| r|durd nd}t%||gd dgD]+\}}|durL|d t&|j'krLtd|dt&|j'd|d dq"t(|j'D]j\}}| r_||f7}|j rpt)g}||j*krpqS||||||dur~||nd|dur||nd|jr||nd|jr||nd|| | | d }|d }| r||df7}|dur||df7}qS| r||f7}| stdd|||||fDSt+|||||dS)aE Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer!zEYou have to specify either decoder_input_ids or decoder_inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FrzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.r)rrjr(rr$zThe `z` should be specified for rr) rrcrrrr[rerrfrr/csrrr(r r(r(r)r sz%MvpDecoder.forward..)r r[r`rcross_attentions),rrerrrr%r#rvrmrrrrllogger warning_oncer r rnrfrom_legacy_cacheget_seq_lengthrrr8rrrr~rNrr:r;rr{r9rrziprrsrrrr)r3rrbrrrr$r[rrrerrrfrrr6 positionsr`rrrall_hidden_statesall_self_attnsall_cross_attentions attn_mask mask_namer decoder_layerrrr(r(r)r@sO                 zMvpDecoder.forwardr) NNNNNNNNNNNNN)rCrDrErFrrrrrr2r:rrHrr rrrr@rIr(r(r4r)rsh *      rc&s&eZdZdgZddgZdeffdd ZddZd d Zd d Z d dZ e                d#de e jde e jde e jde e jde e jde e jde e jde ee jde ede e jde e jde ede ede ede ede e jd eeeff"d!d"ZZS)$MvpModelfinal_logits_biasencoder.embed_tokens.weightdecoder.embed_tokens.weightrcsdt||j|j}}|j|_t||j||_t ||j|j|_ t ||j|j|_ | dSr)r1r2rrrrrrsharedrencoderrdecoderr)r3rrrr4r(r)r2s  zMvpModel.__init__cC|jSr)r6r3r(r(r)get_input_embeddingszMvpModel.get_input_embeddingscCs||_|j|j_|j|j_dSr)r6r7rr8r3valuer(r(r)set_input_embeddingss zMvpModel.set_input_embeddingscCr9r)r7r:r(r(r) get_encoderr<zMvpModel.get_encodercCsF|jsJd|d|jjd|jjd|jjddS)NzHIf you want to use lightweight tuning, make sure that `use_prompt=True`.FT)rrequires_grad_r7rr8rr:r(r(r)set_lightweight_tunings  zMvpModel.set_lightweight_tuningNrrbdecoder_input_idsdecoder_attention_maskrdecoder_head_maskr$encoder_outputsr[rdecoder_inputs_embedsrrerrrfrgcCsJ|dur| dur|durtdt||jj|jj}| dur | n|jj} |dur*|n|jj}| dur4| n|jj} |dur>|n|jj}|durS|j |||| | ||d}n$|rwt |t swt |dt |dkrh|dndt |dkrs|dndd}|j |||d|||| | | | |||d }|s||St|j|j|j|j|j|j|j|jd S) a* decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. NzIf no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.)rrbrrrerrrrr/r ) rrbrrrr$r[rrrerrrf)r r[decoder_hidden_statesdecoder_attentionsr%encoder_last_hidden_staterencoder_attentions)r%r*rrr rerrrr7rnrrr8rr r[r`rr%)r3rrbrCrDrrEr$rFr[rrGrrerrrfdecoder_outputsr(r(r)r@sp3 zMvpModel.forwardNNNNNNNNNNNNNNNN)rCrDrE"_keys_to_ignore_on_load_unexpected_tied_weights_keysrr2r;r?r@rBrrr:rrHlistrr rrrrr@rIr(r(r4r)r2sx       r2ze The MVP Model with a language modeling head. Can be used for various text generation tasks. ) custom_introc(sneZdZgdZdeffdd ZddZddZ d*d ed e ed e de j ffdd Z d edd fddZddZe d+de ejde ejde ejde ejde ejde ejde ejde eejde ede ejde ejd e ejd!e e d"e e d#e e d$e e d%e ejdeeeff$d&d'Zd ejfd(d)ZZS),MvpForConditionalGeneration)r4r5lm_head.weightrcsXt|t||_|dtd|jjjft j |j |jjjdd|_ | dS)Nr3rFrR)r1r2r2rregister_bufferr:rzr6r,rrUrlm_headrrr4r(r)r2fs   z$MvpForConditionalGeneration.__init__cC |jSr)rr@r:r(r(r)r@o z'MvpForConditionalGeneration.get_encodercCrVr)r get_decoderr:r(r(r)rXrrWz'MvpForConditionalGeneration.get_decoderNTnew_num_tokenspad_to_multiple_of mean_resizingrgcst|||}|||Sr)r1resize_token_embeddings_resize_final_logits_bias)r3rYrZr[new_embeddingsr4r(r)r\us z3MvpForConditionalGeneration.resize_token_embeddingscCsj|jjd}||kr|jddd|f}ntjd||f|jjd}tj|j|gdd}|d|dS)Nr!rrrhr3)r3r#r:rzr9ryrT)r3rYold_num_tokensnew_bias extra_biasr(r(r)r]|s z5MvpForConditionalGeneration._resize_final_logits_biascC|j|jddSrrrBrUrAr:r(r(r)rB z2MvpForConditionalGeneration.set_lightweight_tuningrrbrCrDrrEr$rFr[rrGlabelsrrerrrfcCs|dur|n|jj}| dur)| rtdd} |dur)| dur)t| |jj|jj}|j|f|||||||| | | | ||||d}||d|j }d}| durat }|| d|jj | d}|sw|f|dd}|duru|f|S|St |||j|j|j|j|j|j|jd S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example of summarization: Fine-tuning a model ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.", ... return_tensors="pt", ... ) >>> labels = tokenizer("Bad Reasons To Quit Your Job", return_tensors="pt")["input_ids"] >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... generated_ids = model.generate(**inputs) >>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True) ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)rbrCrFrDrrEr$r[rrGrrerrrfrr!r losslogitsr[rHrIr%rJrrK)rrr&warningr*rr rrUr3rrvrrr[rHrIr%rJrrK)r3rrbrCrDrrEr$rFr[rrGrerrerrrfr lm_logitsmasked_lm_lossloss_fctoutputr(r(r)r@s^R z#MvpForConditionalGeneration.forwardcCst||jj|jjSr)r*rrr )r3rer(r(r)%prepare_decoder_input_ids_from_labelsszAMvpForConditionalGeneration.prepare_decoder_input_ids_from_labels)NT)NNNNNNNNNNNNNNNNN)rCrDrErOrr2r@rXrGrrrrr\r]rBrr:rrHrPrr rrrr@rnrIr(r(r4r)rR^s         rRz Mvp model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c$seZdZddgZdeffdd ZddZe               dd ee j d ee j d ee j d ee j d ee j dee j dee j dee e j dee j dee j dee j deedeedeedeedeeeff ddZZS)MvpForSequenceClassificationr4r5rc sBtj|fi|t||_t|j|j|j|j|_| dSr) r1r2r2rrr num_labelsclassifier_dropoutclassification_headr)r3rkwargsr4r(r)r2s  z%MvpForSequenceClassification.__init__cCrbr)rrBrrrAr:r(r(r)rB)rdz3MvpForSequenceClassification.set_lightweight_tuningNrrbrCrDrrEr$rFrrGrerrerrrgcCs.|dur|n|jj}| durd} |dur!| dur!td|jj|j||||||||| | | | ||d}|d}||jj|j }t t | ddkrStd||ddf|dd|ddddddf}||}d}| dur|jjdur|jjdkrd |j_n|jjdkr| jt jks| jt jkrd |j_nd |j_|jjd krt}|jjdkr||| }n,||| }n&|jjd krt}||d|jj| d}n|jjd krt}||| }|s|f|dd}|dur|f|S|St|||j|j|j|j|j |j!|j"d S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: Fine-tuning a model on `num_labels` classes ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForSequenceClassification >>> num_labels = 2 # for example, this is a binary classification task >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForSequenceClassification.from_pretrained("RUCAIBox/mvp", num_labels=num_labels) >>> inputs = tokenizer("Classify: Hello, my dog is cute", return_tensors="pt") >>> labels = torch.tensor(1) # the real label for inputs >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax() ``` NFz8Passing input embeddings is currently not supported for rbrCrDrrEr$rFrrGrrerrrrz7All examples must have the same number of tokens.r! regressionsingle_label_classificationmulti_label_classificationrf)#rrNotImplementedErrorr5rCreq eos_token_idr{r9rr:unique_consecutivesumr%rvrmrr problem_typerpr8r<rGrsqueezerrrr[rHrIr%rJrrK)r3rrbrCrDrrEr$rFrrGrerrerrrr`eos_masksentence_representationrhrgrlrmr(r(r)r@-sM $    $       z$MvpForSequenceClassification.forward)NNNNNNNNNNNNNNN)rCrDrErOrr2rBrrr:rrHrPrrrrrr@rIr(r(r4r)rosj       roc&seZdZddgZfddZddZe                ddeej d eej d eej d eej d eej d eej deej dee ej deej deej deej deej dee dee dee dee deeeff"ddZZS)MvpForQuestionAnsweringr4r5csBt|d|_|j|_t||_t|j|j|_| dSr.) r1r2rpr2rrrU hidden_size qa_outputsrrr4r(r)r2s   z MvpForQuestionAnswering.__init__cCrbr)rrBrrAr:r(r(r)rBrdz.MvpForQuestionAnswering.set_lightweight_tuningNrrbrCrDrrEr$rFstart_positions end_positionsrrGrrerrrgcCsz|dur|n|jj}| dur| durd} |j||||||||| | | |||d}|d}||}|jddd\}}|d}|d}d}| dur| durt| dkr^| d} t| dkrk| d} |d}| d|} | d|} t |d}||| }||| }||d }|s||f|dd}|dur|f|S|St ||||j |j |j|j|j|j|jd S) a` decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: Fine-tuning a model for extrative question answering, and our model also supports generative question answering using `BartForConditionalGeneration` ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForQuestionAnswering >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForQuestionAnswering.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Answer the following question: Who was Jim Henson? [SEP] Jim Henson was a nice puppet", ... return_tensors="pt", ... ) >>> target_start_index = torch.tensor([18]) >>> target_end_index = torch.tensor([19]) >>> loss = model(**inputs, start_positions=target_start_index, end_positions=target_end_index).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1] >>> predict_answer = tokenizer.decode(predict_answer_tokens) ``` NFrtrrr!rh) ignore_indexr/) rg start_logits end_logitsr[rHrIr%rJrrK)rrrrrr~ contiguousrrmrrrr[rHrIr%rJrrK)r3rrbrCrDrrEr$rFrrrrGrrerrrsequence_outputrhrr total_loss ignored_indexrl start_lossend_lossrmr(r(r)r@spS           zMvpForQuestionAnswering.forwardrM)rCrDrErOr2rBrrr:rHrrPrrrrrr@rIr(r(r4r)rsp        rcs(eZdZdZfddZddZZS)MvpDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cst|t||_dSr)r1r2rr8rr4r(r)r2us zMvpDecoderWrapper.__init__cOs|j|i|Sr)r8)r3argsrsr(r(r)r@yszMvpDecoderWrapper.forward)rCrDrErFr2r@rIr(r(r4r)ros rc"seZdZdgZfddZddZddZdd Zd d Zd d Z e              d de e j de e jde e jde e jde e jde e jde ede e jde e j de ede ede ede ede e jdeeeffddZZS)!MvpForCausalLMrScsDd|_d|_t|t||_tj|j|j dd|_ | dS)NTFrR) rOis_encoder_decoderr1r2rrrrUrrrUrrr4r(r)r2s    zMvpForCausalLM.__init__cCs |jjjSrrr8rr:r(r(r)r;rWz#MvpForCausalLM.get_input_embeddingscCs||jj_dSrrr=r(r(r)r?sz#MvpForCausalLM.set_input_embeddingscCs ||j_dSrrr8)r3r8r(r(r) set_decoders zMvpForCausalLM.set_decodercCs|jjSrrr:r(r(r)rXszMvpForCausalLM.get_decodercCrbrrcr:r(r(r)rBrdz%MvpForCausalLM.set_lightweight_tuningNrrbrrrr$r[rrerrerrrfrgcCs| dur| n|jj} | dur| n|jj} | dur| n|jj} |jj||||||||| | | | d }||d}d}| durNt}||d|jj | d}| sd|f|dd}|durb|f|S|St |||j |j |j |jdS)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MvpForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForCausalLM.from_pretrained("RUCAIBox/mvp", add_cross_attention=False) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 8, 50267] ```N) rrbrrrr$r[rrrerrrr!r)rgrhr[r`rr%)rrerrrr8rUrrvrrr[r`rr%)r3rrbrrrr$r[rrerrerrrfrrhrgrlrmr(r(r)r@sD-zMvpForCausalLM.forward)NNNNNNNNNNNNNN)rCrDrErOr2r;r?rrXrBrrr:rrHrr rrrrr@rIr(r(r4r)r}sl        r)rrRrror2r)ArFrtypingrrr:rtorch.nnrrr activationsr cache_utilsr r r generationr modeling_attn_mask_utilsrrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrutils.deprecationrconfiguration_mvpr get_loggerrCr&rHrGr*rr+ModulerJrrrrrrrr2rRrorrr__all__r(r(r(r)sf    $    "Cw51-* v