o ei!@s"dZddlZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl mZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZmZdd l m!Z!ddl"m#Z#m$Z$m%Z%ddl&m'Z'e$(e)Z*dej+de,de,fddZ-Gdddej.Z/Gdddej0Z1GdddeZ2GdddeZ3Gdddej0Z4Gd d!d!ej0Z5e#Gd"d#d#e!Z6Gd$d%d%e6Z7Gd&d'd'e6Z8e#Gd(d)d)e6Z9e#d*d+Gd,d-d-e6eZ:e#d.d+Gd/d0d0e6Z;e#Gd1d2d2e6Zgd7Z?dS)8zPyTorch MVP model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutput)PreTrainedModel)auto_docstringloggingtorch_compilable_check) MvpConfig input_ids pad_token_iddecoder_start_token_idcCsh||j}|ddddf|ddddf<||dddf<|dur*td||dk||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)rrrshifted_input_idsr'b/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/mvp/modeling_mvp.pyshift_tokens_right.s (r)csPeZdZdZdedeffdd Z d dejd ed ejdBffd d ZZ S)MvpLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcsd|_t||j|dSN)offsetsuper__init__)selfr+r, __class__r'r(r1Dsz&MvpLearnedPositionalEmbedding.__init__rNrpast_key_values_length position_idscs\|dur |jdd\}}tj|||tj|jjd|d}n|d}t ||j S)z3`input_ids' shape is expected to be [bsz x seqlen].Nr.)dtypedevicer r) r"torcharangelongweightr8expand unsqueezer0forwardr/)r2rr5r6bszseq_lenr3r'r(r?Js z%MvpLearnedPositionalEmbedding.forward)rN) __name__ __module__ __qualname____doc__intr1r9Tensorr? __classcell__r'r'r3r(r*?sr*cseZdZdZ    ddedededBd edBd edBd edBf fd d Z      ddej dej dBde dBdej dBdej dBdedej dBde ej ej dBe ej dBffddZ Z S) MvpAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsdropout is_decoderbias layer_idxcst||_||_||_|||_|j||jkr'td|jd|d|jd|_||_||_ t j |||d|_ t j |||d|_ t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rO)r0r1rKrLrMhead_dimr$scalingrNrPrLineark_projv_projq_projout_proj)r2rKrLrMrNrOrPr3r'r(r1]s$    zMvpAttention.__init__ hidden_stateskey_value_statespast_key_valuesattention_mask attn_promptoutput_attentionscache_positionreturncCs|du}|\} } } |||j} d} |dur1t|tr/|j|j} |r+|j}n|j }n|}|r5|n|}|rN|durN| rN|j |jj }|j |jj }nJ| |}||}|| d|j|jdd}|| d|j|jdd}|dur|s||nd}||||jd|i\}}|rt|trd|j|j<|durtj|d| ddd|gdd }tj|d| ddd|gdd }|durt| d| |dd|j}tj||gdd }| |jd|jf}| | | |j|jdd} | j|} |j|}|j|}|d}t| |dd}|| |j| |fkr0td | |j| |fd ||dure|| d| |fkrPtd | d| |fd ||| |j| ||}|| |j| |}tjj|dd }|r|| |j| |}|| |j| |}nd}tjj ||j |j!d }t||}|| |j| |jfkrtd| |j| |jfd ||| |j| |j}|dd}|| | |j"}|#|}||fS)z#Input shape: Batch x Time x ChannelNFr rr.r_Trdimz$Attention weights should be of size z , but is z!Attention mask should be of size ptrainingz `attn_output` should be of size )$sizerWrS isinstancer is_updatedgetrPcross_attention_cacheself_attention_cachelayerskeysvaluesrUrVviewrLrR transposeupdater9catr=zerostor8reshapebmmr$r functionalsoftmaxrMrerKrX)r2rYrZr[r\r]r^r_is_cross_attentionr@tgt_len_ query_statesrhcurr_past_key_valuescurrent_states key_states value_states prompt_mask proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputr'r'r(r?zs      """        zMvpAttention.forward)rJFTN)NNNNFN)rBrCrDrErFfloatboolr1r9rGr tupler?rHr'r'r3r(rIZsT  rIcs^eZdZdeffdd Z d dejdejdejded Bd eejejd Bff d d Z Z S)MvpEncoderLayerconfigcst|j|_t|j|j|jd|_t |j|_ |j |_ t |j |_|j|_t|j|j|_t|j|j|_t |j|_dS)N)rKrLrM)r0r1d_modelrKrIencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrMractivation_function activation_fnactivation_dropoutrTencoder_ffn_dimfc1fc2final_layer_normr2rr3r'r(r1s  zMvpEncoderLayer.__init__FrYr\self_attn_promptr^Nr`cCs|}|j||||d\}}tjj||j|jd}||}||}|}|||}tjj||j|jd}| |}tjj||j|jd}||}| |}|j t j krut |sct |rut |j jd}t j|| |d}||fS)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, encoder_attention_heads, pro_len, head_dim)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rYr\r]r^rci)minmax)rrrwrMrerrrrrrr7r9float16isinfanyisnanfinforclamp)r2rYr\rr^residualr clamp_valuer'r'r(r?s2       zMvpEncoderLayer.forwardF) rBrCrDrr1r9 FloatTensorrrr?rHr'r'r3r(rsrcseZdZddeffdd Z         ddejdejdBd ejdBd ejdBd ejdBd ejdBd edBdedBdedBdejdBde ej e ej ej fdBffddZ Z S)MvpDecoderLayerNrcst|j|_t|j|j|jd|d|_|j|_t |j |_ |j |_ t |j|_t|j|j|jd|d|_t |j|_t |j|j|_t |j|j|_t |j|_dS)NT)rKrLrMrNrP)rMrNrP)r0r1rrKrIdecoder_attention_headsrrrMrrrrrrr encoder_attnencoder_attn_layer_normrTdecoder_ffn_dimrrr)r2rrPr3r'r(r10s0  zMvpDecoderLayer.__init__FTrYr\encoder_hidden_statesencoder_attention_maskrcross_attn_promptr[r^ use_cacher_r`c Cs|} |j|||||| d\}} tjj||j|jd}| |}||}d} |durL|} |j||||||d\}} tjj||j|jd}| |}||}|} || |}tjj||j |jd}| |}tjj||j|jd}| |}| |}|f}|r|| | f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. cross_attn_prompt (`torch.FloatTensor`): prompt of cross attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rYr[r\r]r^r_rcN)rYrZr\r]r[r^) rrrwrMrerrrrrrrr)r2rYr\rrrrr[r^rr_rself_attn_weightscross_attn_weightsoutputsr'r'r(r?LsJ       zMvpDecoderLayer.forwardN) NNNNNNFTN) rBrCrDrr1r9rGr rrrr?rHr'r'r3r(r/sD    rcsHeZdZdZdedededeffdd Zdejd ejfd d Z Z S) MvpClassificationHeadz-Head for sentence-level classification tasks. input_dim inner_dim num_classespooler_dropoutcs8tt|||_tj|d|_t|||_dS)Nrd)r0r1rrTdenseDropoutrMrX)r2rrrrr3r'r(r1s zMvpClassificationHead.__init__rYr`cCs6||}||}t|}||}||}|Sr)rMrr9tanhrX)r2rYr'r'r(r?s     zMvpClassificationHead.forward) rBrCrDrErFrr1r9rGr?rHr'r'r3r(rs rcs:eZdZdZfddZdejdeejfddZZ S) MvpPromptz)Layer-wise prompt for encoder or decoder.c st|j|_||_||_|j||_tj|j d|_ t |j|j|_ t t |j|jtt |j|d|j|_dS)Nrr.)r0r1 prompt_length num_layersrLrrRrrrM Embeddingprompt_embedding SequentialrTprompt_mid_dimGELU prompt_trans)r2rrrLr3r'r(r1s   zMvpPrompt.__init__ prompt_idsr`cCsN|||}||j|jd|j|j}||}|gd d}|S)Nr.)rr.rr) rrrorrrLrRrMpermutesplit)r2rpromptr'r'r(r?s  zMvpPrompt.forward) rBrCrDrEr1r9rGrr?rHr'r'r3r(rs "rcs:eZdZUeed<dZdZfddZeddZ Z S)MvpPreTrainedModelrmodelTcs*t|t|trt|jdSdSr)r0 _init_weightsrgMvpForConditionalGenerationinitzeros_final_logits_bias)r2moduler3r'r(rs  z MvpPreTrainedModel._init_weightscCs>|jj}tjgddddd|gg|jd}|||d}|S)N)r r.r r.r8)r\r)rrr9tensorr8ne)r2 pad_tokenr dummy_inputsr'r'r(rs "zMvpPreTrainedModel.dummy_inputs) rBrCrDr__annotations__base_model_prefixsupports_gradient_checkpointingrpropertyrrHr'r'r3r(rs  rcseZdZdZddedejdBdedBffdd Z      dd e j dBd e j dBd e j dBd edBd edBdedBde eBfddZZS) MvpEncodera Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`MvpEncoderLayer`]. Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt NFr embed_tokens use_promptcstj|_j|_j}j|_j|_ j r!t |nd|_ tj||j|_tj||_tfddtjD|_t||_||_|r]j|_tjj|_d|_| dS)N?csg|]}tqSr')r).0r{rr'r( sz'MvpEncoder.__init__..F)!r0r1rMencoder_layerdrop layerdroprr padding_idxmax_position_embeddingsmax_source_positionsscale_embeddingmathsqrt embed_scalerr vocab_sizerr*embed_positions ModuleListrangeencoder_layersrlrlayernorm_embeddingrrrrrgradient_checkpointing post_init)r2rrrrKr3rr(r1s0    zMvpEncoder.__init__rr\ inputs_embedsr^output_hidden_states return_dictr`cKs |dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur<|}|j} |d| d}n|durT|dd} |dddddf}ntd|durd|||j }| |} || } | | } t j j| |j|jd} |jrt|j|j} || } |durt|j| |d}|rdnd}|rdnd}t|jD]=\}}|r|| f}d}|jrtg}||jkrd }|rd }n|| ||jr| |nd|d }|d } |r||d f}q|r|| f}|stdd| ||fDSt| ||dS)a8 Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timer z5You have to specify either input_ids or inputs_embedsrc)rrr\r'FT)NN)rr^rrcs|] }|dur|VqdSrr'rvr'r'r( sz%MvpEncoder.forward..last_hidden_staterY attentions)rr^ruse_return_dictr$r"rorfrrrrrrwrMrerr9r:rrtr8rr enumeraterlrandrrr)r2rr\rr^rrkwargsinput input_shape embed_posrYrrencoder_statesall_attentionsidx encoder_layerto_dropdropout_probability layer_outputsr'r'r(r?st(         zMvpEncoder.forwardNF)NNNNNN)rBrCrDrErrrrr1r9 LongTensorrGrrrr?rHr'r'r3r(rs0& # rcseZdZdZddededBffdd Z           ddejdBd ej dBd ej dBd ejdBd e dBd ej dBdedBdedBdedBdedBdej dBde e BfddZZS) MvpDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`MvpDecoderLayer`] Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt FrrNcstj|_j|_j|_j|_j rt j nd|_ tjj |j|_tjj |_tfddtjD|_tj |_||_|rgj|_tjj|_tjj|_d|_ |!dS)Nrcsg|]}t|dqS))rP)r)rirr'r(rsz'MvpDecoder.__init__..F)"r0r1rMdecoder_layerdroprrrrmax_target_positionsrrrrrrrrrr*rrrdecoder_layersrlrrrrrrrrrr)r2rrr3rr(r1s8   zMvpDecoder.__init__rr\rrr[rrr^rrr_r`c  Ks>|dur|n|jj}| dur| n|jj} |dur|n|jj}| dur$| n|jj} |dur4|dur4td|durF|} |j}|d|d}n|dur^|dd}|dddddf} ntd|durn| ||j }|j r}|j r}|r}t dd}|r|dur|dus|jjrtt|jdt|jdnt|jd}|dur|nd}| durtj|||jd |jd } t|j||| |d }|dur|durt|j|||d }|| |}||}||}tjj||j|j d }|jrt|j|j}| |}|!|}| r dnd}|rdnd}|r|durdnd}t"|j#D]V\}}| r2||f7}|j rCt$g}||j%krCq&||||||jrP||nd|jrY||nd|||| d }|d}|r{||d f7}|dur{||df7}q&| r||f7}| st&dd|||||fDSt'|||||dS)aU Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer zEYou have to specify either decoder_input_ids or decoder_inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frrrr)rrr\r_r[)rrr\rrcr')rrrr[r^rr_r.csrrr'rr'r'r(rdsz%MvpDecoder.forward..)rr[rYrcross_attentions)(rr^rrrr$r"rorfrrrrelogger warning_onceis_encoder_decoderr r get_seq_lengthr9r:r8rr rrrrwrMrrrtrrrrlrrrr)r2rr\rrr[rrr^rrr_rrrr5 positionsrYrrrall_hidden_statesall_self_attnsall_cross_attentionsr  decoder_layerrrr'r'r(r?sA               zMvpDecoder.forwardr) NNNNNNNNNNN)rBrCrDrErrr1r9rrGrr rrr?rHr'r'r3r(rsN $    rcseZdZdgZdddZdeffdd ZddZd d Zd d Z e dde j d Bde j d Bde j d Bde j d Bdee jd Bded Bde jd Bde jd Bded Bded Bded Bded Bde j d BdeeBfddZZS)MvpModelrz shared.weight)zencoder.embed_tokens.weightzdecoder.embed_tokens.weightrcs\t||j|j}}|j|_t||j||_t ||j|_ t ||j|_ | dSr)r0r1rrrrrrsharedrencoderrdecoderr)r2rrrr3r'r(r1zs  zMvpModel.__init__cCs|jSr)r#r2r'r'r(get_input_embeddingsszMvpModel.get_input_embeddingscCs||_|j|j_|j|j_dSr)r#r$rr%r2valuer'r'r(set_input_embeddingss zMvpModel.set_input_embeddingscCsF|jsJd|d|jjd|jjd|jjddS)NzHIf you want to use lightweight tuning, make sure that `use_prompt=True`.FT)rrequires_grad_r$rr%rr&r'r'r(set_lightweight_tunings  zMvpModel.set_lightweight_tuningNrr\decoder_input_idsdecoder_attention_maskencoder_outputsr[rdecoder_inputs_embedsrr^rrr_r`c KsD|dur|dur|durtdt||jj|jj}| dur | n|jj} | dur*| n|jj} | dur4| n|jj} | dur>| n|jj} |durR|j |||| | | d}n$| rvt |t svt |dt |dkrg|dndt |dkrr|dndd}|j |||d|||| | | | | d }| s||St|j|j|j|j|j|j|j|jd S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. NzIf no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.)rr\rr^rrrrr.r) rr\rrr[rrr^rrr_)rr[decoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentions)r$r)rrrr^rrrr$rgrlenr%rrr[rYrr)r2rr\r-r.r/r[rr0rr^rrr_rdecoder_outputsr'r'r(r?sj+ zMvpModel.forward NNNNNNNNNNNNN)rBrCrD"_keys_to_ignore_on_load_unexpected_tied_weights_keysrr1r'r*r,rr9rrGlistrr rrrr?rHr'r'r3r(r"rsh       r"ze The MVP Model with a language modeling head. Can be used for various text generation tasks. ) custom_introc!s6eZdZddiZdeffdd Z d$ded edBd ed ej ffd d Z ded dfddZ ddZ e              d%dejdBdejdBdejdBdejdBdeejdBdedBdejdBdejdBdejdBdedBdedBdedBdedBdejdBd eeBfd d!Zdejfd"d#ZZS)&rlm_head.weightzmodel.shared.weightrcsXt|t||_|dtd|jjjft j |j |jjjdd|_ | dS)NrrFrQ)r0r1r"rregister_bufferr9rsr#r+rrTrlm_headrrr3r'r(r1s   z$MvpForConditionalGeneration.__init__NTnew_num_tokenspad_to_multiple_of mean_resizingr`cst|||}|||Sr)r0resize_token_embeddings_resize_final_logits_bias)r2r?r@rAnew_embeddingsr3r'r(rBs z3MvpForConditionalGeneration.resize_token_embeddingscCsj|jjd}||kr|jddd|f}ntjd||f|jjd}tj|j|gdd}|d|dS)Nr rrrar)rr"r9rsr8rrr=)r2r?old_num_tokensnew_bias extra_biasr'r'r(rCs z5MvpForConditionalGeneration._resize_final_logits_biascC|j|jddSrrr,r>r+r&r'r'r(r,' z2MvpForConditionalGeneration.set_lightweight_tuningrr\r-r.r/r[rr0labelsrr^rrr_cKs | dur| n|jj} | dur)| rtdd} |dur)|dur)t| |jj|jj}|j||||||||| | | | |d }||d|j }d}| dur\t }|| d|jj | d}| sr|f|dd}|durp|f|S|St |||j|j|j|j|j|j|jd S) a\ decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example of summarization: Fine-tuning a model ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.", ... return_tensors="pt", ... ) >>> labels = tokenizer("Bad Reasons To Quit Your Job", return_tensors="pt")["input_ids"] >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... generated_ids = model.generate(**inputs) >>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True) ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F) r\r-r/r.r[rr0rr^rrr_rr r losslogitsr[r1r2rr3rr4)rrrwarningr)rrrr>rrrorrr[r1r2rr3rr4)r2rr\r-r.r/r[rr0rKrr^rrr_rr lm_logitsmasked_lm_lossloss_fctoutputr'r'r(r?+sVJ z#MvpForConditionalGeneration.forwardcCst||jj|jjSr)r)rrr)r2rKr'r'r(%prepare_decoder_input_ids_from_labelsszAMvpForConditionalGeneration.prepare_decoder_input_ids_from_labels)NT)NNNNNNNNNNNNNN)rBrCrDr9rr1rFrrrrBrCr,rr9rrGr:rr rrr?rTrHr'r'r3r(rs~         zrz Mvp model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZdeffdd ZddZe            ddejdBdej dBd ejdBd ejdBd e ej dBd ej dBd ej dBdejdBde dBde dBde dBde dBde eBfddZZS)MvpForSequenceClassificationrc sBtj|fi|t||_t|j|j|j|j|_| dSr) r0r1r"rrr num_labelsclassifier_dropoutclassification_headr)r2rrr3r'r(r1s  z%MvpForSequenceClassification.__init__cCrHr)rr,rXr+r&r'r'r(r,rJz3MvpForSequenceClassification.set_lightweight_tuningNrr\r-r.r/rr0rKrr^rrr`c  Ks"| dur| n|jj} |durd} |dur!|dur!td|jj|j|||||||| | | | d }|d}||jj|j }t t | ddkd||ddf|dd|ddddddf}||}d}|dur|jjdur|jjdkrd |j_n|jjdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jjdkr|||}n,|||}n&|jjd krt}||d|jj|d}n|jjd krt}|||}| s|f|dd}|dur|f|S|St|||j|j|j|j|j |j!|j"d S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: Fine-tuning a model on `num_labels` classes ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForSequenceClassification >>> num_labels = 2 # for example, this is a binary classification task >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForSequenceClassification.from_pretrained("RUCAIBox/mvp", num_labels=num_labels) >>> inputs = tokenizer("Classify: Hello, my dog is cute", return_tensors="pt") >>> labels = torch.tensor(1) # the real label for inputs >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax() ``` NFz8Passing input embeddings is currently not supported for r\r-r.r/rr0rr^rrrrz7All examples must have the same number of tokens.r regressionsingle_label_classificationmulti_label_classificationrL)#rrNotImplementedErrorr4rBreq eos_token_idrtr8rr9unique_consecutivesumnumelrorfrX problem_typerVr7r;rFrsqueezerrrr[r1r2rr3rr4)r2rr\r-r.r/rr0rKrr^rrrrrYeos_masksentence_representationrNrMrRrSr'r'r(r?s~E  $    $       z$MvpForSequenceClassification.forward) NNNNNNNNNNNN)rBrCrDrr1r,rr9rrGr:rrrrr?rHr'r'r3r(rUsV       rUcseZdZfddZddZe             ddejdBdejdBdejdBd ejdBd e ej dBd ejdBd ejdBd ej dBdej dBde dBde dBde dBde dBde e BfddZZS)MvpForQuestionAnsweringcsBt|d|_|j|_t||_t|j|j|_| dSr-) r0r1rVr"rrrT hidden_size qa_outputsrrr3r'r(r1Ss   z MvpForQuestionAnswering.__init__cCrHr)rr,rir+r&r'r'r(r,_rJz.MvpForQuestionAnswering.set_lightweight_tuningNrr\r-r.r/start_positions end_positionsrr0rr^rrr`c Kst| dur| n|jj} |dur|durd} |j||||||| | | | | d }|d}||}|jddd\}}|d}|d}d}|dur|durt|dkr[|d}t|dkrh|d}|d}| d|}| d|}t |d}|||}|||}||d }| s||f|dd}|dur|f|S|St ||||j |j |j|j|j|j|jd S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. Example: Fine-tuning a model for extrative question answering, and our model also supports generative question answering using `BartForConditionalGeneration` ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForQuestionAnswering >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForQuestionAnswering.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Answer the following question: Who was Jim Henson? [SEP] Jim Henson was a nice puppet", ... return_tensors="pt", ... ) >>> target_start_index = torch.tensor([18]) >>> target_end_index = torch.tensor([19]) >>> loss = model(**inputs, start_positions=target_start_index, end_positions=target_end_index).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1] >>> predict_answer = tokenizer.decode(predict_answer_tokens) ``` NFrYrrr ra) ignore_indexr.) rM start_logits end_logitsr[r1r2rr3rr4)rrrrirrd contiguousr5rfrrrr[r1r2rr3rr4)r2rr\r-r.r/rjrkrr0rr^rrrrsequence_outputrNrmrn total_loss ignored_indexrR start_lossend_lossrSr'r'r(r?csjK           zMvpForQuestionAnswering.forwardr7)rBrCrDr1r,rr9rGrr:rrrrr?rHr'r'r3r(rgQs\        rgcs(eZdZdZfddZddZZS)MvpDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cs"t|t||_|dSr)r0r1rr%rrr3r'r(r1s   zMvpDecoderWrapper.__init__cOs|j|i|Sr)r%)r2argsrr'r'r(r?szMvpDecoderWrapper.forward)rBrCrDrEr1r?rHr'r'r3r(rus rucseZdZddiZfddZddZddZd d Ze dd e j d Bde j d Bde j d Bde j d Bde d Bde j d Bde j d Bded Bded Bded Bded Bde j d Bdee j BdeeBfddZZS)MvpForCausalLMr<z!model.decoder.embed_tokens.weightcsDd|_d|_t|t||_tj|j|j dd|_ | dS)NTFrQ) rNrr0r1rurrrTrhrr>rrr3r'r(r1s    zMvpForCausalLM.__init__cCs |jjjSrrr%rr&r'r'r(r' s z#MvpForCausalLM.get_input_embeddingscCs||jj_dSrrxr(r'r'r(r*sz#MvpForCausalLM.set_input_embeddingscCrHrrIr&r'r'r(r,rJz%MvpForCausalLM.set_lightweight_tuningNrrr\rrr[rrKrr^rrr_logits_to_keepr`c Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} |jj|||||||| | | d }|d}t| tr=t| dn| }| |dd|ddf}d}|durdt }|| d|jj | d}| sz|f|dd}|durx|f|S|St |||j|j|j|jdS)ap labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MvpForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForCausalLM.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 8, 50267] ```N) rr\rrr[rrr^rrrr r)rMrNr[rYrr)rr^rrrr%rgrFslicer>rrorrr[rYrr)r2rr\rrr[rrKrr^rrr_ryrrrY slice_indicesrNrMrRrSr'r'r(r?sD( zMvpForCausalLM.forward) NNNNNNNNNNNNr)rBrCrDr9r1r'r*r,rr9rrGrr rrFrrr?rHr'r'r3r(rwsb       rw)rwrrgrUr"r)@rErr9rtorch.nnrrrrr activationsr cache_utilsr r r generationr masking_utilsr rmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrconfiguration_mvpr get_loggerrBrrGrFr)rr*ModulerIrrrrrrrr"rrUrgrurw__all__r'r'r'r(sb      $   ?n$l"" l