o ei@s^dZddlZddlmZddlmZmZmZddlmZ ddl m Z m Z m Z ddlmZmZddlmZmZmZmZmZmZmZmZdd lmZdd lmZdd lmZm Z m!Z!dd l"m#Z#m$Z$dd l%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:e!;e<Z=Gddde)Z>Gddde8Z?Gddde(Z@Gddde3ZAGddde7ZBGddde5ZCGddde*ZDe Gd d!d!eZEGd"d#d#e6ZFGd$d%d%e/ZGGd&d'd'e.ZHGd(d)d)e4ZIGd*d+d+e+ZJGd,d-d-e-ZKGd.d/d/e1ZLGd0d1d1e,ZMGd2d3d3e2ZNGd4d5d5e0ZOgd6ZPdS)7zPyTorch ERNIE model.N)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)Cache DynamicCacheEncoderDecoderCache)create_bidirectional_maskcreate_causal_mask),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs)BertCrossAttentionBertEmbeddings BertEncoderBertForMaskedLMBertForMultipleChoiceBertForNextSentencePredictionBertForPreTrainingBertForPreTrainingOutputBertForQuestionAnsweringBertForSequenceClassificationBertForTokenClassification BertLayerBertLMHeadModelBertLMPredictionHead BertModel BertPoolerBertSelfAttention) ErnieConfigcsteZdZdZfddZ      ddejdBdejdBdejdBd ejdBd ejdBd ed ej fd dZ Z S)ErnieEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cs4t||j|_|jrt|j|j|_dSdS)N)super__init__ use_task_idnn Embeddingtask_type_vocab_size hidden_sizetask_type_embeddings)selfconfig __class__e/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/ernie/modular_ernie.pyr2Bs zErnieEmbeddings.__init__Nr input_idstoken_type_ids task_type_ids position_ids inputs_embedspast_key_values_lengthreturncCs6|dur |}n|dd}|\}} |dur&|jdd|| |f}|durSt|drH|j|jdd} tj| d|d} | || }n tj|tj |jj d}|dur\| |}| |} | | j }|| } ||} | | } |jr|durtj|tj |jj d}||}| |7} || } || } | S)Nr@rr.)dimindex)dtypedevice)sizerBhasattrr@expandshapetorchgatherzeroslongrJword_embeddingstoken_type_embeddingstoposition_embeddingsr3r8 LayerNormdropout)r9r?r@rArBrCrD input_shape batch_size seq_lengthbuffered_token_type_idsrT embeddingsrVr8r=r=r>forwardIs6         zErnieEmbeddings.forward)NNNNNr) __name__ __module__ __qualname____doc__r2rO LongTensor FloatTensorintTensorr^ __classcell__r=r=r;r>r0?s0  r0c@ eZdZdS)ErnieSelfAttentionNr_r`rar=r=r=r>riric@rh)ErnieCrossAttentionNrjr=r=r=r>rlrkrlc@rh) ErnieLayerNrjr=r=r=r>rmrkrmc@rh) ErniePoolerNrjr=r=r=r>rnrkrnc@rh)ErnieLMPredictionHeadNrjr=r=r=r>rorkroc@rh) ErnieEncoderNrjr=r=r=r>rprkrpcsLeZdZeZdZdZdZdZdZ dZ e e e dZefddZZS)ErniePreTrainedModelernieT) hidden_states attentionscross_attentionscsft|t|trt|jdSt|tr1t|j t |j j d dt|jdSdS)zInitialize the weightsrF)r.rFN)r1 _init_weights isinstanceroinitzeros_biasr0copy_rBrOarangerNrMr@)r9moduler;r=r>rvs   "z"ErniePreTrainedModel._init_weights)r_r`rar/ config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrmrirl_can_record_outputsrOno_gradrvrgr=r=r;r>rqsrqcseZdZdgZdfdd Zeee           ddej dBdej dBdej dBd ej dBd ej dBd ej dBd ej dBd ej dBde dBde dBdej dBde e deej eBfddZddZZS) ErnieModelrmTcsLt||||_d|_t||_t||_|rt|nd|_ | dS)NF) r1r2r:gradient_checkpointingr0r]rpencoderrnpooler post_init)r9r:add_pooling_layerr;r=r>r2s   zErnieModel.__init__Nr?attention_maskr@rArBrCencoder_hidden_statesencoder_attention_maskpast_key_values use_cachecache_positionkwargsrEc  Ks|jjr| dur | n|jj} nd} |jr |jr | r tdd} | rA| durA|dus.|jjr;tt |jdt |jdnt |jd} |durM|durMt d|dur\| ||| } n|duri| dd} nt d| \}}|durx|j n|j }| dur| nd}| durtj||||d } |j||||||d }|j||||| | d \}}|j|f|||| | | |d | }|d}|jdur||nd}t|||jd S) task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] NFzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...)r:zDYou cannot specify both input_ids and inputs_embeds at the same timerFz5You have to specify either input_ids or inputs_embedsr)rJ)r?rBr@rArCrD)rrembedding_outputrrr)rrrrrrrB)last_hidden_state pooler_outputr)r: is_decoderrrtraininglogger warning_onceis_encoder_decoderr r ValueError%warn_if_padding_and_no_attention_maskrKrJget_seq_lengthrOr|r]_create_attention_masksrrr r)r9r?rr@rArBrCrrrrrrrYrZr[rJrDrencoder_outputssequence_output pooled_outputr=r=r>r^s~          zErnieModel.forwardcCsP|jjrt|j||||d}nt|j||d}|dur$t|j|||d}||fS)N)r:rCrrr)r:rCr)r:rCrr)r:rr r )r9rrrrrrr=r=r>r&s* z"ErnieModel._create_attention_masks)T) NNNNNNNNNNN)r_r`ra_no_split_modulesr2rrrrOrfrboolrrtupler r^rrgr=r=r;r>rsZ       brc@rh)ErnieForPreTrainingOutputNrjr=r=r=r>rIrkrc@seZdZdddZee        ddejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd ejdBd ee de eje BfddZ dS)ErnieForPreTrainingcls.predictions.bias'ernie.embeddings.word_embeddings.weightzcls.predictions.decoder.biaszcls.predictions.decoder.weightNr?rr@rArBrClabelsnext_sentence_labelrrEc  Ks|j|f|||||dd| } | dd\} } || | \} }d}|durK|durKt}|| d|jj|d}||dd|d}||}t|| || j| jdS)a: task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, ErnieForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForPreTraining.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` Trr@rArBrC return_dictNrrF)lossprediction_logitsseq_relationship_logitsrsrt) rrclsrviewr: vocab_sizerrsrt)r9r?rr@rArBrCrrroutputsrrprediction_scoresseq_relationship_score total_lossloss_fctmasked_lm_lossnext_sentence_lossr=r=r>r^Ss6/ zErnieForPreTraining.forwardNNNNNNNN) r_r`ra_tied_weights_keysrrrOrfrrrrr^r=r=r=r>rMsF   rc"@seZdZee             ddejdBdejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd eejdBd edBdejdBde ejBde e de eje BfddZdS)ErnieForCausalLMNrr?rr@rArBrCrrrrrrlogits_to_keeprrEcKs| durd} |j|f|||||||| | | dd |}|j}t| tr)t| dn| }||dd|ddf}d}| durM|jd|| |jjd|}t |||j |j |j |j dS)a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` NFT) rr@rArBrCrrrrrr)logitsrr)rrrrsrtrur=)rrrrwreslicer loss_functionr:rr rrsrtru)r9r?rr@rArBrCrrrrrrrrrrs slice_indicesrrr=r=r>r^sB zErnieForCausalLM.forward) NNNNNNNNNNNNr)r_r`rarrrOrflistrrerrrr r^r=r=r=r>rs^      rc@seZdZdddZee         ddejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd ejdBd ejdBdee de eje BfddZ dS)ErnieForMaskedLMrrrNr?rr@rArBrCrrrrrEc  Ksx|j|f|||||||dd| } | d} || } d}| dur2t}|| d|jj| d}t|| | j| jdS)as task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rr@rArBrCrrrrNrFrrrsrt) rrrrrr:rrrsrt)r9r?rr@rArBrCrrrrrrrrrr=r=r>r^s4  zErnieForMaskedLM.forward) NNNNNNNNN) r_r`rarrrrOrfrrrrr^r=r=r=r>rsL    rc@eZdZee       d dejdBdejdBdejdBdejdBdejdBdejdBdejdBd eed e eje Bfd d Z dS)ErnieForNextSentencePredictionNr?rr@rArBrCrrrEc Ksp|j|f|||||dd|} | d} || } d} |dur.t} | | dd|d} t| | | j| jdS)a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` Trr.NrFrr)rrrrrrrsrt)r9r?rr@rArBrCrrrrseq_relationship_scoresrrr=r=r>r^#s0- z&ErnieForNextSentencePrediction.forwardNNNNNNN) r_r`rarrrOrfrrrrr^r=r=r=r>r":  rc@r)ErnieForSequenceClassificationNr?rr@rArBrCrrrEc Ks6|j|f|||||dd|} | d} || } || } d} |dur|jjdurN|jdkr4d|j_n|jdkrJ|jtjksE|jtj krJd|j_nd|j_|jjdkrlt } |jdkrf| | | } n+| | |} n%|jjdkrt } | | d|j| d} n|jjdkrt} | | |} t| | | j| jd S) a^ task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trr.N regressionsingle_label_classificationmulti_label_classificationrFr)rrrX classifierr: problem_type num_labelsrIrOrRrersqueezerrrrrsrt)r9r?rr@rArBrCrrrrrrrr=r=r>r^msP     "       z&ErnieForSequenceClassification.forwardr) r_r`rarrrOrfrrrrr^r=r=r=r>rlrrc@r)ErnieForMultipleChoiceNr?rr@rArBrCrrrEc Ks2|dur |jdn|jd} |dur|d|dnd}|dur*|d|dnd}|dur9|d|dnd}|durH|d|dnd}|dur[|d|d|dnd}|j|f|||||dd|} | d} || } || } | d| } d}|durt}|| |}t|| | j| j dS)a9 input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) task_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr.rFTrr) rNrrKrrrXrrrrsrt)r9r?rr@rArBrCrr num_choicesrrrreshaped_logitsrrr=r=r>r^sF0    zErnieForMultipleChoice.forwardr) r_r`rarrrOrfrrrrr^r=r=r=r>rrrc@r)ErnieForTokenClassificationNr?rr@rArBrCrrrEc Ks||j|f|||||dd|} | d} || } || } d} |dur4t} | | d|j|d} t| | | j| jdS)a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrrNrFr) rrrXrrrrrrsrt)r9r?rr@rArBrCrrrrrrrr=r=r>r^s2  z#ErnieForTokenClassification.forwardr) r_r`rarrrOrfrrrrr^r=r=r=r>rrrc@seZdZee        ddejdBdejdBdejdBdejdBdejdBdejdBdejdBd ejdBd eed e eje Bfd d Z dS)ErnieForQuestionAnsweringNr?rr@rArBrCstart_positions end_positionsrrEc  Ks |j|f|||||dd| } | d} || } | jddd\} }| d} |d}d}|durz|durzt|dkrG|d}t|dkrT|d}| d}|d|}|d|}t|d}|| |}|||}||d }t || || j | j d S) rTrrr.rF)rGN) ignore_indexr)r start_logits end_logitsrsrt) rr qa_outputssplitr contiguouslenrKclamprrrsrt)r9r?rr@rArBrCrrrrrrrrr ignored_indexr start_lossend_lossr=r=r>r^DsJ          z!ErnieForQuestionAnswering.forwardr) r_r`rarrrOrfrrrrr^r=r=r=r>rCs@   r) rrrrrrrrrrq)QrbrOtorch.nnr4rrrrrx cache_utilsrrr masking_utilsr r modeling_outputsr r rrrrrrmodeling_utilsrprocessing_utilsrutilsrrr utils.genericrrutils.output_capturingrbert.modeling_bertrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-configuration_ernier/ get_loggerr_rr0rirlrmrnrorprqrrrrrrrrrr__all__r=r=r=r>sJ   (  L  @SE=JH[4 B