o eiQI@sldZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZmZmZdd lmZdd lmZd d lmZmZmZmZmZd dlmZd dlmZddlm Z GdddeZ!eGdddeZ"GdddeZ#GdddeZ$eGddde"Z%eddGddde"e Z&Gd d!d!eZ'Gd"d#d#eZ(Gd$d%d%eZ)gd&Z*dS)'zPyTorch PLBART model.N)nn)CrossEntropyLoss)initialization)Cache)GenerationMixin)BaseModelOutputSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)auto_docstring)BartClassificationHead BartDecoder BartEncoderBartForCausalLMBartScaledWordEmbedding)'BigBirdPegasusForSequenceClassification)shift_tokens_right) PLBartConfigc@ eZdZdS)PLBartScaledWordEmbeddingN__name__ __module__ __qualname__rrg/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/plbart/modular_plbart.pyr,rcsBeZdZUeed<dZdZddgZdZdZ dZ fddZ Z S)PLBartPreTrainedModelconfigmodelTPLBartDecoderLayerPLBartEncoderLayercs*t|t|trt|jdSdSN)super _init_weights isinstancePLBartForConditionalGenerationinitzeros_final_logits_bias)selfmodule __class__rrr':s  z#PLBartPreTrainedModel._init_weights) rrrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_supports_flex_attnr' __classcell__rrr/rr 0s r c@r) PLBartEncoderNrrrrrr9@rr9c@r) PLBartDecoderNrrrrrr:Drr:cseZdZdddZdeffdd ZddZdd Ze dd e j d Bd e j d Bd e j d Bde j d Bde e j d Bded Bde j d Bde j d Bded Bded Bded Bded Bde j d Bdee j eBfddZZS) PLBartModelz shared.weight)zencoder.embed_tokens.weightzdecoder.embed_tokens.weightr!csdt||j|j}}|jrt|jnd}t||j||d|_ t ||_ t ||_ |dS)Ng?) embed_scale)r&__init__ pad_token_id vocab_sizescale_embeddingmathsqrtd_modelrsharedr9encoderr:decoder post_init)r-r! padding_idxr?r<r/rrr=Os    zPLBartModel.__init__cCs|jSr%)rD)r-rrrget_input_embeddings[sz PLBartModel.get_input_embeddingscCs||_|j|j_|j|j_dSr%)rDrE embed_tokensrF)r-valuerrrset_input_embeddings^s z PLBartModel.set_input_embeddingsN input_idsattention_maskdecoder_input_idsdecoder_attention_maskencoder_outputspast_key_values inputs_embedsdecoder_inputs_embeds use_cacheoutput_attentionsoutput_hidden_states return_dictcache_positionreturnc Ks.| dur| n|jj} | dur| n|jj} | dur| n|jj} | dur$| n|jj} |dur7|dur7t||jj}|durG|j|||| | | d}n$| rkt|t skt |dt |dkr\|dndt |dkrg|dndd}|j |||d|||| | | | | d }| s||St |j |j|j|j|j|j |j|jdS) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. N)rMrNrSrVrWrXrrr )last_hidden_state hidden_states attentions) rMrNencoder_hidden_statesencoder_attention_maskrRrSrUrVrWrXrY)r[rRdecoder_hidden_statesdecoder_attentionscross_attentionsencoder_last_hidden_stater^encoder_attentions)r!rVrWrUuse_return_dictrr>rEr(rlenrFr r[rRr\r]rb)r-rMrNrOrPrQrRrSrTrUrVrWrXrYkwargsdecoder_outputsrrrforwardcs^( zPLBartModel.forward) NNNNNNNNNNNNN)rrr_tied_weights_keysrr=rIrLr torch LongTensorTensorlist FloatTensorrbooltupler rir8rrr/rr;Hsd        r;zv The PLBART Model with a language modeling head. Can be used for code-to-text, text-to-code and code-to-code. ) custom_introc!s>eZdZdZdgZddiZdeffdd Z d$d ed edBd e d e j ffdd Z d ed dfddZ e              d%dejdBdejdBdejdBdejdBdeejdBdedBdejdBdejdBdejdBde dBde dBde dBde dBdejdBd eejeBfd d!Zdejfd"d#ZZS)&r)r"r,zlm_head.weightzmodel.shared.weightr!csXt|t||_|dtd|jjjft j |j |jjjdd|_ | dS)Nr,rF)bias)r&r=r;r"register_bufferrkzerosrDnum_embeddingsrLinearrClm_headrG)r-r!r/rrr=s   z'PLBartForConditionalGeneration.__init__NTnew_num_tokenspad_to_multiple_of mean_resizingrZcs&t|||}||jjd|S)Nr)r&resize_token_embeddings_resize_final_logits_biasweightshape)r-ryrzr{new_embeddingsr/rrr|sz6PLBartForConditionalGeneration.resize_token_embeddingscCsj|jjd}||kr|jddd|f}ntjd||f|jjd}tj|j|gdd}|d|dS)Nr)device)dimr,)r,rrkrurcatrt)r-ryold_num_tokensnew_bias extra_biasrrrr}s z8PLBartForConditionalGeneration._resize_final_logits_biasrMrNrOrPrQrRrSrTlabelsrUrVrWrXrYcKs| dur| n|jj} | dur|dur|durt| |jj}|j||||||||| | | | |d }||d}||j|j}d}| durVt }|| d|jj | d}| sl|f|dd}|durj|f|S|St |||j |j|j|j|j|j|jd S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Mask-filling: ```python >>> from transformers import AutoTokenizer, PLBartForConditionalGeneration >>> model = PLBartForConditionalGeneration.from_pretrained("uclanlp/plbart-base") >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> # en_XX is the language symbol id for English >>> TXT = " Is 0 the Fibonacci number ? en_XX" >>> input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt").input_ids >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['first', 'same', 'highest', 'result', 'number'] ``` N) rNrOrQrPrRrSrTrUrVrWrXrYrrr) losslogitsrRr`rarbrcr^rd)r!rerr>r"rxr,torrviewr?r rRr`rarbrcr^rd)r-rMrNrOrPrQrRrSrTrrUrVrWrXrYrgoutputs lm_logitsmasked_lm_lossloss_fctoutputrrrrisNBz&PLBartForConditionalGeneration.forwardcCst||jjSr%)rr!r>)r-rrrr%prepare_decoder_input_ids_from_labelsYszDPLBartForConditionalGeneration.prepare_decoder_input_ids_from_labels)NT)NNNNNNNNNNNNNN)rrrr2_keys_to_ignore_on_load_missingrjrr=intrpr Embeddingr|r}r rkrlrmrnrorrqr rirr8rrr/rr)s          nr)c@r)PLBartClassificationHeadNrrrrrr]rrcseZdZfddZZS)PLBartForSequenceClassificationc tjdi|dS)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr&ri super_kwargsr/rrribsz'PLBartForSequenceClassification.forward)rrrrir8rrr/rrasrcs eZdZefddZZS)PLBartForCausalLMc r)aF labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PLBartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> model = PLBartForCausalLM.from_pretrained("uclanlp/plbart-base") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```Nrrrr/rrriszPLBartForCausalLM.forward)rrrr rir8rrr/rrsr)rr)rr;r )+__doc__rArkrtorch.nnrrr* cache_utilsr generationrmodeling_outputsrr r modeling_utilsr utilsr bart.modeling_bartrrrrr(bigbird_pegasus.modeling_bigbird_pegasusrmbart.modeling_mbartrconfiguration_plbartrrr r9r:r;r)rrr__all__rrrrs>           }