o ei{@sdZddlZddlmZddlmZmZmZddlmZ ddl m Z ddl m Z ddlmZmZmZmZmZmZmZdd lmZdd lmZdd lmZmZmZdd lmZd dl m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'e(e)Z*Gddde"Z+Gddde%Z,Gddde!Z-Gddde#Z.eGdddeZ/Gddde$Z0eddGdd d e/e Z1eGd!d"d"e/Z2Gd#d$d$ej3Z4ed%dGd&d'd'e/Z5eGd(d)d)e/Z6eGd*d+d+e/Z7Gd,d-d-ej3Z8eGd.d/d/e/Z9gd0Z:dS)1zPyTorch RoBERTa model.N)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)gelu)GenerationMixin),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuple)BertCrossAttentionBertEmbeddings BertLayer BertModelBertSelfAttention) RobertaConfigc sxeZdZfddZ     ddejdBdejdBdejdBdejdBd ef d d Ze d d Z e dddZ Z S)RobertaEmbeddingscs8t||`|`|j|_tj|j|j|jd|_dS)N) padding_idx) super__init__ pad_token_idposition_embeddingsrnn Embeddingmax_position_embeddings hidden_sizeselfconfig __class__i/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/roberta/modular_roberta.pyr!-s   zRobertaEmbeddings.__init__Nr input_idstoken_type_ids position_ids inputs_embedspast_key_values_lengthc Cs|dur|dur|||j|}n|||j}|dur!|}n|dd}|\}}|durZt|drO|j|jdd} tj | d|d} | ||}n tj |tj |j j d}|durc||}||} || } ||} | | } || } || } | S)Nr0rr)dimindexdtypedevice)"create_position_ids_from_input_idsr&create_position_ids_from_inputs_embedssizehasattrr0expandshapetorchgatherzeroslongr1r9word_embeddingstoken_type_embeddingsr# LayerNormdropout) r)r/r0r1r2r3 input_shape batch_size seq_lengthbuffered_token_type_idsrE embeddingsr#r-r-r.forward8s2       zRobertaEmbeddings.forwardcCsJ|dd}|d}tj|d||dtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr4rr7r)r<r@arangerCr9 unsqueezer>)r2rrHsequence_lengthr1r-r-r.r;hs  z8RobertaEmbeddings.create_position_ids_from_inputs_embedscCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr5)neintr@cumsumtype_asrC)r/rr3maskincremental_indicesr-r-r.r:zs  z4RobertaEmbeddings.create_position_ids_from_input_ids)NNNNr)r) __name__ __module__ __qualname__r!r@ LongTensor FloatTensorrSrM staticmethodr;r: __classcell__r-r-r+r.r,s,   0 rc@ eZdZdS)RobertaSelfAttentionNrXrYrZr-r-r-r.r`r`c@r_)RobertaCrossAttentionNrar-r-r-r.rcrbrcc@r_) RobertaLayerNrar-r-r-r.rdrbrdcsLeZdZeZdZdZdZdZdZ dZ e e e dZefddZZS)RobertaPreTrainedModelrobertaT) hidden_states attentionscross_attentionscsft|t|trt|jdSt|tr1t|j t |j j d dt|jdSdS)zInitialize the weightsr4)rr4N)r _init_weights isinstance RobertaLMHeadinitzeros_biasrcopy_r1r@rNr?r>r0)r)moduler+r-r.rjs   "z$RobertaPreTrainedModel._init_weights)rXrYrZr config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrdr`rc_can_record_outputsr@no_gradrjr^r-r-r+r.resrecseZdZdfdd ZZS) RobertaModelTcst||dSN)r r!)r)r*add_pooling_layerr+r-r.r!szRobertaModel.__init__)T)rXrYrZr!r^r-r-r+r.r{sr{zS RoBERTa Model with a `language modeling` head on top for CLM fine-tuning. ) custom_introc seZdZdddZfddZddZdd Zee dd e j d Bd e j d Bde j d Bde j d Bde j d Bde j d Bde j d Bde j d Bde e e j d Bde d Bde jd Bdee jBdeede e jeBfddZZS)RobertaForCausalLM)roberta.embeddings.word_embeddings.weight lm_head.biaszlm_head.decoder.weightzlm_head.decoder.biascs@t||jstdt|dd|_t||_| dS)NzOIf you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`Fr} r r! is_decoderloggerwarningr{rfrllm_head post_initr(r+r-r.r!s    zRobertaForCausalLM.__init__cC|jjSr|rdecoderr)r-r-r.get_output_embeddingsz(RobertaForCausalLM.get_output_embeddingscC ||j_dSr|rr)new_embeddingsr-r-r.set_output_embeddings z(RobertaForCausalLM.set_output_embeddingsNrr/attention_maskr0r1r2encoder_hidden_statesencoder_attention_masklabelspast_key_values use_cachecache_positionlogits_to_keepkwargsreturnc  Ks|durd} |j|f||||||| | | dd | }|j}t| tr(t| dn| }||dd|ddf}d}|durL|jd|||jjd| }t |||j |j |j |j dS)am token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, RobertaForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base") >>> config = AutoConfig.from_pretrained("FacebookAI/roberta-base") >>> config.is_decoder = True >>> model = RobertaForCausalLM.from_pretrained("FacebookAI/roberta-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) rr0r1r2rrrrr return_dict)logitsr vocab_size)lossrrrgrhrir-)rflast_hidden_staterkrSslicer loss_functionr*rr rrgrhri)r)r/rr0r1r2rrrrrrrroutputsrg slice_indicesrrr-r-r.rMs@1 zRobertaForCausalLM.forward) NNNNNNNNNNNr)rXrYrZ_tied_weights_keysr!rrrrr@r[r\tupleboolTensorrSrrr rMr^r-r-r+r.rsd        rcseZdZdddZfddZddZdd Zee dd e j d Bd e j d Bd e j d Bde j d Bde j d Bde j d Bde j d Bde j d Bde e dee jeBfddZZS)RobertaForMaskedLMrrrcs@t||jrtdt|dd|_t||_| dS)NznIf you want to use `RobertaForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Frrr(r+r-r.r!0s   zRobertaForMaskedLM.__init__cCrr|rrr-r-r.r?rz(RobertaForMaskedLM.get_output_embeddingscCrr|rrr-r-r.rBrz(RobertaForMaskedLM.set_output_embeddingsNr/rr0r1r2rrrrrc  Ks|j|f||||||dd| } | d} || } d} |dur7|| j}t}|| d|jj|d} t| | | j | j dS)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rr0r1r2rrrrNr4rrrgrh) rfrtor9rviewr*rr rgrh)r)r/rr0r1r2rrrrrsequence_outputprediction_scoresmasked_lm_lossloss_fctr-r-r.rMEs4   zRobertaForMaskedLM.forward)NNNNNNNN)rXrYrZrr!rrrrr@r[r\rrrrr rMr^r-r-r+r.r)sL    rc(eZdZdZfddZddZZS)rlz*Roberta Head for masked language modeling.csZtt|j|j|_tj|j|jd|_t|j|j |_ t t |j |_dS)N)eps)r r!r$Linearr'denserFlayer_norm_eps layer_normrr Parameterr@rBror(r+r-r.r!s zRobertaLMHead.__init__cKs*||}t|}||}||}|Sr|)rrrrr)featuresrxr-r-r.rMs   zRobertaLMHead.forwardrXrYrZ__doc__r!rMr^r-r-r+r.rls rlz RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfddZee      ddejdBdejdBdejdBdejdBdejdBd ejdBd e e d e ej e Bfd d ZZS) RobertaForSequenceClassificationcs>t||j|_||_t|dd|_t||_|dSNFr) r r! num_labelsr*r{rfRobertaClassificationHead classifierrr(r+r-r.r!s   z)RobertaForSequenceClassification.__init__Nr/rr0r1r2rrrc Ks6|j|f||||dd|}|d} || } d} |dur|| j}|jjdurN|jdkr4d|j_n|jdkrJ|jtj ksE|jtj krJd|j_nd|j_|jjdkrlt } |jdkrf| | | } n+| | |} n%|jjdkrt } | | d |j|d } n|jjdkrt} | | |} t| | |j|jd S) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trr0r1r2rrNr regressionsingle_label_classificationmulti_label_classificationr4r)rfrrr9r* problem_typerr8r@rCrSrsqueezerrrrrgrh r)r/rr0r1r2rrrrrrrr-r-r.rMsN     "       z(RobertaForSequenceClassification.forwardNNNNNN)rXrYrZr!rrr@r[r\rrrrrrMr^r-r-r+r.rs6   rcseZdZfddZee      ddejdBdejdBdejdBdejdBdejdBd ejdBd e e d e ej e Bfd d ZZS)RobertaForMultipleChoicecs@t|t||_t|j|_t|j d|_ | dS)Nr) r r!r{rfr$Dropouthidden_dropout_probrGrr'rrr(r+r-r.r!s   z!RobertaForMultipleChoice.__init__Nr/r0rrr1r2rrcKs<|dur |jdn|jd}|dur|d|dnd} |dur*|d|dnd} |dur9|d|dnd} |durH|d|dnd} |dur[|d|d|dnd} |j| f| | | | dd|}|d}||}||}|d|}d}|dur||j}t}|||}t |||j |j dS)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nrr4T)r1r0rr2rr) r?rr<rfrGrrr9rr rgrh)r)r/r0rrr1r2r num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsr pooled_outputrreshaped_logitsrrr-r-r.rMsF+     z RobertaForMultipleChoice.forwardr)rXrYrZr!rrr@r[r\rrrrr rMr^r-r-r+r.rs6   rcr)RobertaForTokenClassificationcsbt||j|_t|dd|_|jdur|jn|j}t||_ t |j |j|_ | dSr)r r!rr{rfclassifier_dropoutrr$rrGrr'rrr)r*rr+r-r.r!Rs   z&RobertaForTokenClassification.__init__Nr/rr0r1r2rrrc Ks|j|f||||dd|}|d} || } || } d} |dur9|| j}t} | | d|j|d} t| | |j |j dS)a- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrrNr4r) rfrGrrr9rrrrrgrhrr-r-r.rM`s2   z%RobertaForTokenClassification.forwardr)rXrYrZr!rrr@r[r\rrrrrrMr^r-r-r+r.rPs6  rcr)rz-Head for sentence-level classification tasks.csTtt|j|j|_|jdur|jn|j}t||_ t|j|j |_ dSr|) r r!r$rr'rrrrrGrout_projrr+r-r.r!s  z"RobertaClassificationHead.__init__cKsL|dddddf}||}||}t|}||}||}|S)Nr)rGrr@tanhrrr-r-r.rMs     z!RobertaClassificationHead.forwardrr-r-r+r.rs  rcseZdZfddZee       ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd e e d e ej e Bfd dZZS)RobertaForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSr) r r!rr{rfr$rr' qa_outputsrr(r+r-r.r!s  z$RobertaForQuestionAnswering.__init__Nr/rr0r1r2start_positions end_positionsrrcKs|j|f||||dd|} | d} || } | jddd\} } | d} | d} d}|dury|duryt|dkrF|d}t|dkrS|d}| d}|d|}|d|}t|d}|| |}|| |}||d }t || | | j | j d S) a[ token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) Trrrr4rQN) ignore_indexr)r start_logits end_logitsrgrh) rfrsplitr contiguouslenr<clamprr rgrh)r)r/rr0r1r2rrrrrrrr total_loss ignored_indexr start_lossend_lossr-r-r.rMsH          z#RobertaForQuestionAnswering.forward)NNNNNNN)rXrYrZr!rrr@r[r\rrrrr rMr^r-r-r+r.rs<    r)rrrrrrr{re);rr@torch.nnr$rrrrrm activationsr generationrmodeling_outputsr r r r r rrmodeling_utilsrprocessing_utilsrutilsrrr utils.genericrbert.modeling_bertrrrrrconfiguration_robertar get_loggerrXrrr`rcrdrer{rrModulerlrrrrr__all__r-r-r-r.sR    $    _nUT`F N