o wiE@sdZddlZddlZddlmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZdd lmZmZdd lmZdd lmZmZmZmZmZmZmZdd lmZdd l m!Z!m"Z"ddl#m$Z$e"%e&Z'dej(de)de)fddZ*Gdddej+Z,Gdddej-Z.GdddeZ/GdddeZ0Gdddej-Z1Gdd d ej-Z2e!Gd!d"d"eZ3Gd#d$d$e3Z4Gd%d&d&e3Z5e!Gd'd(d(e3Z6e!d)d*Gd+d,d,e3eZ7e!d-d*Gd.d/d/e3Z8e!Gd0d1d1e3Z9Gd2d3d3e3Z:Gd4d5d5e3eZ;gd6Zr=cCs ||||j|jddS)Nrr+)viewrJrO transpose contiguous)r/rVr>r=r$r$r%_shape{s zMvpAttention._shapeN hidden_stateskey_value_statespast_key_valueattention_masklayer_head_mask attn_promptoutput_attentionsreturncCs|du}|\} } } |||j} |r"|dur"|d} |d}nZ|r9|||d| } |||d| }nC|durh|||d| } |||d| }tj|d| gdd} tj|d|gdd}n|||d| } |||d| }|jr| |f}|durtj|d | ddd| gdd} tj|d | ddd|gdd}|durt | d| |dd |j }tj||gdd}| |j d|jf}|| | | j|} | j|} |j|}| d}t| | dd}|| |j | |fkrtd| |j | |fd||durG|| d| |fkr2td | d| |fd||| |j | ||}|| |j | |}tjj|dd}|dur||j fkrktd |j fd||dddd|| |j | |}|| |j | |}|r|| |j | |}|| |j | |}nd}tjj||j|jd }t||}|| |j | |jfkrtd | |j | |jfd||| |j | |j}|dd}|| | |j}||}|||fS) z#Input shape: Batch x Time x ChannelNrrrr+dimz$Attention weights should be of size z , but is z!Attention mask should be of size z/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )sizerTrPrZrRrSr6catrLr:zerostor5rJrOrWbmmrXr!r functionalsoftmaxrKrgreshaperIrU)r/r[r\r]r^r_r`rais_cross_attentionr=tgt_len_ query_states key_states value_states prompt_mask proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputr$r$r%r<~s  """      "    zMvpAttention.forward)rHFT)NNNNNF)r@rArBrCrDfloatboolr.r6rErZrtupler<rFr$r$r0r%rG]sP  rGcsdeZdZdeffdd Z d dejdejdejdejd eed e ejeejff d d Z Z S)MvpEncoderLayerconfigcst|j|_t|j|j|jd|_t |j|_ |j |_ t |j |_|j|_t|j|j|_t|j|j|_t |j|_dS)N)rIrJrK)r-r.d_modelrIrGencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrKr activation_function activation_fnactivation_dropoutrQencoder_ffn_dimfc1fc2final_layer_normr/rr0r$r%r.s  zMvpEncoderLayer.__init__Fr[r^r_self_attn_promptrarbc Cs|}|j|||||d\}}}tjj||j|jd}||}||}|}|||}tjj||j|jd}| |}tjj||j|jd}||}| |}|j t j krwt |set |rwt |j jd} t j|| | d}|f} |r| |f7} | S)a@ Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, encoder_attention_heads, pro_len, head_dim)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r[r^r_r`rarei)minmax)rrrmrKrgrrrrrrr4r6float16isinfanyisnanfinforclamp) r/r[r^r_rraresidualryrr clamp_valueoutputsr$r$r%r< s:        zMvpEncoderLayer.forward)F) r@rArBrr.r6 FloatTensorrr~rr<rFr$r$r0r%rs rcseZdZdeffdd Z          ddejdeejd eejd eejd eejd eejd eejdeejdeeejdee dee deej eeej ej fffddZ Z S)MvpDecoderLayerrcst|j|_t|j|j|jdd|_|j|_t |j |_ |j |_ t |j|_t|j|j|jdd|_t |j|_t |j|j|_t |j|j|_t |j|_dS)NT)rIrJrKrL)rKrL)r-r.rrIrGdecoder_attention_headsrrrKr rrrrrr encoder_attnencoder_attn_layer_normrQdecoder_ffn_dimrrrrr0r$r%r.As,  zMvpDecoderLayer.__init__NFTr[r^encoder_hidden_statesencoder_attention_maskr_cross_attn_layer_head_maskrcross_attn_promptr]ra use_cacherbc  Cs^|} | dur | ddnd} |j|| |||| d\}}}tjj||j|jd}| |}||}d}d}|durm|} | durD| ddnd}|j||||||| d\}}}tjj||j|jd}| |}||}||}|} || |}tjj||j |jd}| |}tjj||j|jd}| |}| |}|f}| r|||f7}| r||f7}|S)aC Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. cross_attn_prompt (`torch.FloatTensor`): prompt of cross attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. Nr+)r[r]r^r_r`rare)r[r\r^r_r`r]ra) rrrmrKrgrrrrrrrr)r/r[r^rrr_rrrr]rarrself_attn_past_key_valueself_attn_weightspresent_key_valuecross_attn_present_key_valuecross_attn_weightscross_attn_past_key_valuerr$r$r%r<[sX$         zMvpDecoderLayer.forward) NNNNNNNNFT) r@rArBrr.r6rErrr~rr<rFr$r$r0r%r@sJ     rcsHeZdZdZdedededeffdd Zdejd ejfd d Z Z S) MvpClassificationHeadz-Head for sentence-level classification tasks. input_dim inner_dim num_classespooler_dropoutcs8tt|||_tj|d|_t|||_dS)Nrf)r-r.rrQdenseDropoutrKrU)r/rrrrr0r$r%r.s zMvpClassificationHead.__init__r[rbcCs6||}||}t|}||}||}|SN)rKrr6tanhrU)r/r[r$r$r%r<s     zMvpClassificationHead.forward) r@rArBrCrDr}r.r6rEr<rFr$r$r0r%rs rcs:eZdZdZfddZdejdeejfddZZ S) MvpPromptz)Layer-wise prompt for encoder or decoder.c st|j|_||_||_|j||_tj|j d|_ t |j|j|_ t t |j|jtt |j|d|j|_dS)Nrr+)r-r. prompt_length num_layersrJrrOrrrK Embeddingprompt_embedding SequentialrQprompt_mid_dimGELU prompt_trans)r/rrrJr0r$r%r.s   zMvpPrompt.__init__ prompt_idsrbcCsN|||}||j|jd|j|j}||}|gd d}|S)Nr+)rr+rr) rrrWrrrJrOrKpermutesplit)r/rpromptr$r$r%r<s  zMvpPrompt.forward) r@rArBrCr.r6rErr<rFr$r$r0r%rs "rc@s,eZdZeZdZdZddZeddZ dS)MvpPreTrainedModelmodelTcCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|tj rA|jjjd|d|j durC|jj|j  dSdSdS)NrH)meanstd) rinit_std isinstancerrQr9datanormal_rMzero_r padding_idx)r/modulerr$r$r% _init_weightss    z MvpPreTrainedModel._init_weightscCs>|jj}tjgddddd|gg|jd}|||d}|S)N)r r+r r+r5)r^r)rrr6rVr5ne)r/ pad_tokenr dummy_inputsr$r$r%rs "zMvpPreTrainedModel.dummy_inputsN) r@rArBr config_classbase_model_prefixsupports_gradient_checkpointingrpropertyrr$r$r$r%rs rcseZdZdZ ddedeejdeeffdd Z d d Z d d Z       dd ee j dee jdee jdee jdeedeedeedeeeffddZZS) MvpEncodera Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`MvpEncoderLayer`]. Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt NFr embed_tokens use_promptcstj|_j|_j}j|_j|_ j r!t |nd|_ |dur,||_n tj||j|_tj||_tfddtjD|_t||_||_|rej|_tjj|_d|_| dS)N?cg|]}tqSr$)r.0rrrr$r% ,z'MvpEncoder.__init__..F)!r-r.rKencoder_layerdrop layerdroprrrmax_position_embeddingsmax_source_positionsscale_embeddingmathsqrt embed_scalerrr vocab_sizer'embed_positions ModuleListrangeencoder_layerslayersrlayernorm_embeddingrrrrrgradient_checkpointing post_init)r/rrrrIr0rr%r.s4    zMvpEncoder.__init__cC|jSrrr/r$r$r%get_input_embeddings<zMvpEncoder.get_input_embeddingscC ||_dSrrr/valuer$r$r%set_input_embeddings? zMvpEncoder.set_input_embeddingsrr^ head_mask inputs_embedsraoutput_hidden_states return_dictrbcCsb|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur<|}|j} |d| d}n|durT|dd} |dddddf}ntd|durd|||j }| |} || } | | } t j j| |j|jd} |jrt|j|j} || } |durt||j}|rdnd}|rdnd}|dur|dt|jkrtdt|jd |dd t|jD]G\}}|r|| f}d }|jrtg}||jkrd }|rd }n|| ||dur||nd|jr| |nd|d}|d} |r||df}q|r|| f}|s*tdd| ||fDSt| ||dS)a~ Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsrer$rz&The head_mask should be specified for  layers, but it is for .FT)NN)r_rrarcs|] }|dur|VqdSrr$rvr$r$r% sz%MvpEncoder.forward..last_hidden_stater[ attentions) rraruse_return_dictr!rrWrhrrrrrrmrKrgrr6r7rrkr5rr r4lenr enumeraterandrrr)r/rr^rrrarrinput input_shape embed_posr[rrencoder_statesall_attentionsidx encoder_layerto_dropdropout_probability layer_outputsr$r$r%r<Bs|.           zMvpEncoder.forwardNF)NNNNNNN)r@rArBrCrrrrr~r.rrr6 LongTensorrErrrrr<rFr$r$r0r%r sH & rcseZdZdZ ddedeejdeeffdd Z d d Z d d Z            dd ee j dee jdee jdee j dee jdee jdeee jdee jdeedeedeedeedeeeffddZZS) MvpDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`MvpDecoderLayer`] Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt NFrrrcstj|_j|_j|_j|_j rt j nd|_ |dur*||_n tjj |j|_tjj |_tfddtjD|_tj |_||_|roj|_tjj|_tjj|_d|_ |!dS)Nrcrr$)rrrr$r%rrz'MvpDecoder.__init__..F)"r-r.rKdecoder_layerdroprrrrmax_target_positionsrrrrrrrrrr'rrrdecoder_layersrrrrrrrrrrr)r/rrrr0rr%r.s<   zMvpDecoder.__init__cCrrrrr$r$r%rrzMvpDecoder.get_input_embeddingscCrrrrr$r$r%rrzMvpDecoder.set_input_embeddingsrr^rrrcross_attn_head_maskpast_key_valuesrrrarrrbc !Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} | dur$| n|jj} |dur4|dur4td|durF|} |j}|d|d}n|dur^|dd}|dddddf} ntd|duro|ddjdnd}|dur}| ||j }t ||||}|dur|durt ||j |dd}|| |}||}||}tjj||j|jd}|jrt|j|j}||}||}|jr|jr| rtd d } | rd nd}| rd nd}| r|durd nd}| rd nd}t||gd d gD]*\}}|dur%|dt |j!kr%td|dt |j!d|ddqt"|j!D]\}}| r7||f7}|jrHt#g}||j$krHq+|durQ||nd}||||||dura||nd|durk||nd|jrt||nd|jr}||nd|| | d }|d}| r||| rdndf7}| r||df7}|dur||df7}q+| r||f7}| r|nd} | st%dd|| |||fDSt&|| |||dS)a  Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timerzEYou have to specify either decoder_input_ids or decoder_inputs_embedsrr+)rqrezZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr$rr$zThe `z` should be specified for rr)rr_rrrr]rarrrcsr rr$r r$r$r%r sz%MvpDecoder.forward..)rr%r[rcross_attentions)'rrarrrr!rrWrhrrr r r4rrrrmrKrgrr6r7rrkr5rrrlogger warning_onceziprrrrrrr)!r/rr^rrrr$r%rrrarrrrr2 positionsr[rrrall_hidden_statesall_self_attnsall_cross_attentionsnext_decoder_cache attn_mask mask_namer decoder_layerrr]r next_cacher$r$r%r<sP                  zMvpDecoder.forwardr) NNNNNNNNNNNN)r@rArBrCrrrrr~r.rrr6rrErlistrrrr<rFr$r$r0r%r sf (       r c$s(eZdZdgZddgZdeffdd ZddZd d Zd d Z d dZ ddZ e               d$de ejde ejde ejde ejde ejde ejde ejde eejde eejde ejde ejde ede ede ed e ed!eeeff d"d#ZZS)%MvpModelfinal_logits_biasencoder.embed_tokens.weightdecoder.embed_tokens.weightrcsdt||j|j}}|j|_t||j||_t ||j|j|_ t ||j|j|_ | dSr)r-r.rrrrrrsharedrencoderr decoderr)r/rrrr0r$r%r.s  zMvpModel.__init__cCrr)r8rr$r$r%rrzMvpModel.get_input_embeddingscCs||_|j|j_|j|j_dSr)r8r9rr:rr$r$r%rs zMvpModel.set_input_embeddingscCrr)r9rr$r$r% get_encoderrzMvpModel.get_encodercCrrr:rr$r$r% get_decoderrzMvpModel.get_decodercCsF|jsJd|d|jjd|jjd|jjddS)NzHIf you want to use lightweight tuning, make sure that `use_prompt=True`.FT)rrequires_grad_r9rr:rrr$r$r%set_lightweight_tunings  zMvpModel.set_lightweight_tuningNrr^decoder_input_idsdecoder_attention_maskrdecoder_head_maskr$encoder_outputsr%rdecoder_inputs_embedsrrarrrbcCsH|dur| dur|durtdt||jj|jj}| dur | n|jj} |dur*|n|jj}| dur4| n|jj} |dur>|n|jj}|durS|j |||| | ||d}n$|rwt |t swt |dt |dkrh|dndt |dkrs|dndd}|j |||d|||| | | | ||d }|s||St|j|j|j|j|j|j|j|jd S) a* decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. NzIf no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.)rr^rrrarrrrr+r  rr^rrrr$r%rrrarr)rr%decoder_hidden_statesdecoder_attentionsr&encoder_last_hidden_staterencoder_attentions)r!r&rrrrarrrr9rrrr:rrr%r[rr&)r/rr^r@rArrBr$rCr%rrDrrarrdecoder_outputsr$r$r%r<sn2 zMvpModel.forwardNNNNNNNNNNNNNNN)r@rArB"_keys_to_ignore_on_load_unexpected_tied_weights_keysrr.rrr;r=r?rrr6rrEr3rr~rrrr<rFr$r$r0r%r4st       r4ze The MVP Model with a language modeling head. Can be used for various text generation tasks. ) custom_introc&seZdZgdZdeffdd ZddZddZ d/d ed e ed e de j ffdd Z d edd fddZddZddZddZe d0de ejde ejde ejde ejde ejde ejde ejd e eejd!e eejd"e ejd#e ejd$e ejd%e e d&e e d'e e d(e e deeeff"d)d*Zd$ejfd+d,Zed-d.ZZS)1MvpForConditionalGeneration)r6r7lm_head.weightrcsXt|t||_|dtd|jjjft j |j |jjjdd|_ | dS)Nr5rFrN)r-r.r4rregister_bufferr6rjr8r(rrQrlm_headrrr0r$r%r.ss   z$MvpForConditionalGeneration.__init__cC |jSr)rr;rr$r$r%r;|rz'MvpForConditionalGeneration.get_encodercCrSr)rr=rr$r$r%r=rz'MvpForConditionalGeneration.get_decoderNTnew_num_tokenspad_to_multiple_of mean_resizingrbcst|||}|||Sr)r-resize_token_embeddings_resize_final_logits_bias)r/rTrUrVnew_embeddingsr0r$r%rWs z3MvpForConditionalGeneration.resize_token_embeddingscCsj|jjd}||kr|jddd|f}ntjd||f|jjd}tj|j|gdd}|d|dS)Nrrrrcr5)r5rr6rjr5rirQ)r/rTold_num_tokensnew_bias extra_biasr$r$r%rXs z5MvpForConditionalGeneration._resize_final_logits_biascCrrrRrr$r$r%get_output_embeddingsrz1MvpForConditionalGeneration.get_output_embeddingscCrrr]r/rYr$r$r%set_output_embeddingsrz1MvpForConditionalGeneration.set_output_embeddingscC|j|jddSrrr?rRr>rr$r$r%r? z2MvpForConditionalGeneration.set_lightweight_tuningrr^r@rArrBr$rCr%rrDlabelsrrarrcCs|dur|n|jj}| dur)| rtdd} |dur)| dur)t| |jj|jj}|j||||||||| | | | |||d}||d|j }d}| dur^t }|| d|jj | d}|st|f|dd}|durr|f|S|St |||j|j|j|j|j|j|jd S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example of summarization: Fine-tuning a model ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.", ... return_tensors="pt", ... ) >>> labels = tokenizer("Bad Reasons To Quit Your Job", return_tensors="pt")["input_ids"] >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... generated_ids = model.generate(**inputs) >>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True) ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)r^r@rCrArrBr$r%rrDrrarrrrr losslogitsr%rFrGr&rHrrI)rrr'warningr&rrrrRr5rrWrrr%rFrGr&rHrrI)r/rr^r@rArrBr$rCr%rrDrdrrarrr lm_logitsmasked_lm_lossloss_fctoutputr$r$r%r<sZQ z#MvpForConditionalGeneration.forwardcCst||jj|jjSr)r&rrr)r/rdr$r$r%%prepare_decoder_input_ids_from_labels szAMvpForConditionalGeneration.prepare_decoder_input_ids_from_labelscsBd}|D]}|tfdd|ddD|ddf7}q|S)Nr$c3$|] }|d|jVqdSr? index_selectrkr5r past_statebeam_idxr$r%r )"z=MvpForConditionalGeneration._reorder_cache..r+rr%rtreordered_past layer_pastr$rsr%_reorder_cache#s z*MvpForConditionalGeneration._reorder_cache)NTNNNNNNNNNNNNNNNN) r@rArBrMrr.r;r=rDrr~rrrWrXr^r`r?rr6rrEr3rrrrr<rm staticmethodrzrFr$r$r0r%rOks         rOz Mvp model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c$seZdZddgZdeffdd ZddZe               dd ee j d ee j d ee j d ee j d ee j dee j dee j dee e j dee j dee j dee j deedeedeedeedeeeff ddZZS)MvpForSequenceClassificationr6r7rc sBtj|fi|t||_t|j|j|j|j|_| dSr) r-r.r4rrr num_labelsclassifier_dropoutclassification_headr)r/rkwargsr0r$r%r.8s  z%MvpForSequenceClassification.__init__cCrar)rr?rr>rr$r$r%r?Ercz3MvpForSequenceClassification.set_lightweight_tuningNrr^r@rArrBr$rCrrDrdrrarrrbcCs.|dur|n|jj}| durd} |dur!| dur!td|jj|j||||||||| | | | ||d}|d}||jj|j }t t | ddkrStd||ddf|dd|ddddddf}||}d}| dur|jjdur|jjdkrd |j_n|jjdkr| jt jks| jt jkrd |j_nd |j_|jjd krt}|jjdkr||| }n,||| }n&|jjd krt}||d|jj| d}n|jjd krt}||| }|s|f|dd}|dur|f|S|St|||j|j|j|j|j |j!|j"d S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: Fine-tuning a model on `num_labels` classes ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForSequenceClassification >>> num_labels = 2 # for example, this is a binary classification task >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForSequenceClassification.from_pretrained("RUCAIBox/mvp", num_labels=num_labels) >>> inputs = tokenizer("Classify: Hello, my dog is cute", return_tensors="pt") >>> labels = torch.tensor(1) # the real label for inputs >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax() ``` NFz8Passing input embeddings is currently not supported for r^r@rArrBr$rCrrDrrarrrrz7All examples must have the same number of tokens.r regressionsingle_label_classificationmulti_label_classificationre)#rrNotImplementedErrorr1r@req eos_token_idrkr5rr6unique_consecutivesumr!rWrhr problem_typer~r4r8rDrsqueezerrrr%rFrGr&rHrrI)r/rr^r@rArrBr$rCrrDrdrrarrrr[eos_masksentence_representationrgrfrkrlr$r$r%r<IsM $    $       z$MvpForSequenceClassification.forwardrK)r@rArBrMrr.r?rrr6rrEr3rr~rrrr<rFr$r$r0r%r}/sj       r}c&seZdZddgZfddZddZe                ddeej d eej d eej d eej d eej d eej deej dee ej deej deej deej deej dee dee dee dee deeeff"ddZZS)MvpForQuestionAnsweringr6r7csBt|d|_|j|_t||_t|j|j|_| dSr*) r-r.r~r4rrrQ hidden_size qa_outputsrrr0r$r%r.s   z MvpForQuestionAnswering.__init__cCrar)rr?rr>rr$r$r%r?rcz.MvpForQuestionAnswering.set_lightweight_tuningNrr^r@rArrBr$rCstart_positions end_positionsrrDrrarrrbcCsz|dur|n|jj}| dur| durd} |j||||||||| | | |||d}|d}||}|jddd\}}|d}|d}d}| dur| durt| dkr^| d} t| dkrk| d} |d}| d|} | d|} t |d}||| }||| }||d }|s||f|dd}|dur|f|S|St ||||j |j |j|j|j|j|jd S) a` decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: Fine-tuning a model for extrative question answering, and our model also supports generative question answering using `BartForConditionalGeneration` ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForQuestionAnswering >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForQuestionAnswering.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Answer the following question: Who was Jim Henson? [SEP] Jim Henson was a nice puppet", ... return_tensors="pt", ... ) >>> target_start_index = torch.tensor([18]) >>> target_end_index = torch.tensor([19]) >>> loss = model(**inputs, start_positions=target_start_index, end_positions=target_end_index).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1] >>> predict_answer = tokenizer.decode(predict_answer_tokens) ``` NFrrrrrc) ignore_indexr+) rf start_logits end_logitsr%rFrGr&rHrrI)rrrrrrrYrrhrrrr%rFrGr&rHrrI)r/rr^r@rArrBr$rCrrrrDrrarrrsequence_outputrgrr total_loss ignored_indexrk start_lossend_lossrlr$r$r%r<spS           zMvpForQuestionAnswering.forwardr{)r@rArBrMr.r?rrr6rErr3rr~rrrr<rFr$r$r0r%rsp        rcs(eZdZdZfddZddZZS)MvpDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cst|t||_dSr)r-r.r r:rr0r$r%r.s zMvpDecoderWrapper.__init__cOs|j|i|Srr<)r/argsrr$r$r%r<szMvpDecoderWrapper.forward)r@rArBrCr.r<rFr$r$r0r%rs rc seZdZdgZfddZddZddZdd Zd d Zd d Z ddZ ddZ e             d%de ejde ejde ejde ejde ejde ejde eejde ejde ejde ede ede ede ed eeeffd!d"Zed#d$ZZS)&MvpForCausalLMrPcsNt|}d|_d|_t|t||_tj |j |j dd|_ | dS)NTFrN)copydeepcopyrLis_encoder_decoderr-r.rrrrQrrrRrrr0r$r%r.s    zMvpForCausalLM.__init__cCs |jjjSrrr:rrr$r$r%rrz#MvpForCausalLM.get_input_embeddingscCs||jj_dSrrrr$r$r%rsz#MvpForCausalLM.set_input_embeddingscCrrr]rr$r$r%r^rz$MvpForCausalLM.get_output_embeddingscCrrr]r_r$r$r%r`rz$MvpForCausalLM.set_output_embeddingscCs ||j_dSrrr:)r/r:r$r$r% set_decoders zMvpForCausalLM.set_decodercCs|jjSrrrr$r$r%r=szMvpForCausalLM.get_decodercCrarrbrr$r$r%r?rcz%MvpForCausalLM.set_lightweight_tuningNrr^rrrr$r%rrdrrarrrbcCs| dur| n|jj} | dur| n|jj} | dur| n|jj} |jj||||||||| | | | d }||d}d}| durNt}||d|jj | d}| sd|f|dd}|durb|f|S|St |||j |j |j |jdS)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MvpForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForCausalLM.from_pretrained("RUCAIBox/mvp", add_cross_attention=False) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 8, 50267] ```NrErrr)rfrgr%r[rr&)rrarrrr:rRrrWrrr%r[rr&)r/rr^rrrr$r%rrdrrarrrrgrfrkrlr$r$r%r<sD,zMvpForCausalLM.forwardcs.d}|D]}|tfdd|Df7}q|S)Nr$c3rnr?rorqrsr$r%r ruz0MvpForCausalLM._reorder_cache..rvrwr$rsr%rzs zMvpForCausalLM._reorder_cache) NNNNNNNNNNNNN)r@rArBrMr.rrr^r`rr=r?rrr6rrErr3r~rrrr<r|rzrFr$r$r0r%rsn          Ur)rrOrr}r4r)=rCrrtypingrrr6torch.utils.checkpointrtorch.nnrrr activationsr generationr modeling_attn_mask_utilsr r modeling_layersr modeling_outputsrrrrrrrmodeling_utilsrutilsrrconfiguration_mvpr get_loggerr@r'rErDr&rr'ModulerGrrrrrrr r4rOr}rrr__all__r$r$r$r%sh    $   H~; @-*