o wi@sdZddlZddlmZmZddlZddlZddlmZddlm Z ddl m Z m Z m Z ddlmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZmZm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)m*Z*ddl+m,Z,e*-e.Z/Gddde j0Z1Gddde j0Z2Gddde2Z3Gddde j0Z4e2e3dZ5Gddde j0Z6Gddde j0Z7Gdd d e j0Z8Gd!d"d"eZ9Gd#d$d$e j0Z:Gd%d&d&e j0Z;e(Gd'd(d(e"Ze(Gd0d1d1e<Z?Gd2d3d3e j0Z@e(d4d*Gd5d6d6e<ZAe(Gd7d8d8e<ZBe(Gd9d:d:e<ZCGd;d<dd>e<ZEdBd?d@ZFgdAZGdS)CzPyTorch RoBERTa model.N)OptionalUnion)version)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNgelu)GenerationMixin)#_prepare_4d_attention_mask_for_sdpa*_prepare_4d_causal_attention_mask_for_sdpa)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringget_torch_versionlogging) RobertaConfigcs4eZdZdZfddZ d ddZdd ZZS) RobertaEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dd|j|_tj|j|j|jd|_ dS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rF) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr%register_buffertorcharangeexpandzerosr'sizelongr"selfconfig __class__i/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.pyr.8s"   zRobertaEmbeddings.__init__Nrc Cs|dur|durt||j|}n||}|dur|}n|dd}|d}|durTt|drI|jddd|f}||d|} | }n tj|tj |j j d}|dur]| |}| |} || } |jdkrt||} | | 7} || } || } | S)Nr(rr*rr,devicer&)"create_position_ids_from_input_idsr"&create_position_ids_from_inputs_embedsrChasattrr*rAr?rBrDr'rMr3r7r%r5r8r<) rF input_idsr*r' inputs_embedspast_key_values_length input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr7 embeddingsr5rJrJrKforwardQs0         zRobertaEmbeddings.forwardcCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr(rrLr)rCr?r@r"rDrM unsqueezerA)rFrRrTsequence_lengthr'rJrJrKrOys  z8RobertaEmbeddings.create_position_ids_from_inputs_embeds)NNNNr)__name__ __module__ __qualname____doc__r.rYrO __classcell__rJrJrHrKr!2s   (r!cseZdZdfdd ZdejdejfddZ      dd ejd eejd eejd eejd eejdee e ejdee de ejfddZ Z S)RobertaSelfAttentionNcst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|p\t|dd|_|jdksh|jd kry|j|_t d |jd |j|_|j|_dS) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r%r& relative_keyrelative_key_queryr)r-r.r1num_attention_headsrP ValueErrorintattention_head_size all_head_sizerLinearquerykeyvaluer:attention_probs_dropout_probr<r=r%r4r/distance_embedding is_decoderrFrGr%rHrJrKr.s*   zRobertaSelfAttention.__init__xreturncCs6|dd|j|jf}||}|ddddS)Nr(rrfrr )rCrgrjviewpermute)rFrt new_x_shaperJrJrKtranspose_for_scoress z)RobertaSelfAttention.transpose_for_scoresF hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionscCs||}|du} | r|dur|d} |d} |}nP| r/|||} |||} |}n;|durZ|||} |||} tj|d| gdd} tj|d| gdd} n|||} |||} ||} |du} |jrz| | f}t| | dd}|j dks|j dkr | j d| j d}}| rtj |dtj |j d dd}ntj|tj |j d dd}tj|tj |j d dd}||}|||jd}|j| jd }|j dkrtd | |}||}n|j dkr td | |}td | |}|||}|t|j}|dur||}tjj|dd}||}|dur0||}t|| }|dddd }|dd|jf}||}|rX||fn|f}|jrd||f}|S)Nrrrfdimr(rdrerLr+zbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rmryrnror?catrrmatmul transposer%shapetensorrDrMrvr@rqr4tor,einsummathsqrtrjr functionalsoftmaxr<rw contiguousrCrk)rFrzr{r|r}r~rrmixed_query_layeris_cross_attention key_layer value_layer query_layer use_cacheattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputsrJrJrKrYsn             zRobertaSelfAttention.forwardNNNNNNF) r\r]r^r.r?Tensorryr FloatTensortupleboolrYr`rJrJrHrKras4 racseZdZdfdd Z      ddejdeejdeejdeejd eejd eeeejd ee d eejffd d Z Z S)RobertaSdpaSelfAttentionNcs4tj||d|j|_tttdk|_dS)Nr%z2.2.0)r-r.rp dropout_probrparserrequire_contiguous_qkvrsrHrJrKr.sz!RobertaSdpaSelfAttention.__init__Frzr{r|r}r~rrruc s|jdks |s |durtdt|||||||S|\}} } |||} |du} | r3|n|} | r9|n|}| rP|rP|djd| jdkrP|\}}n,|| | }|| | }|dur|| s|t j |d|gdd}t j |d|gdd}|j r||f}|jr| jjdkr|dur| } |}|}|j r| s|dur| dkrdnd }t jjj| ||||jr|jnd |d }|dd}||| |j}|f}|j r||f}|S) Nr&aRobertaSdpaSelfAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support non-absolute `position_embedding_type` or `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rrfrrcudaTF) attn_mask dropout_p is_causal)r%logger warning_oncer-rYrCryrmrrnror?rrrrrMtyperrrscaled_dot_product_attentiontrainingrrreshaperk)rFrzr{r|r}r~rrbsztgt_len_rrcurrent_statesrrr attn_outputrrHrJrKrYs^          z RobertaSdpaSelfAttention.forwardrr) r\r]r^r.r?rrrrrrYr`rJrJrHrKrs2  rc8eZdZfddZdejdejdejfddZZS)RobertaSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr#) r-r.rrlr1denser8r9r:r;r<rErHrJrKr.z zRobertaSelfOutput.__init__rz input_tensorrucC&||}||}|||}|Srrr<r8rFrzrrJrJrKrY  zRobertaSelfOutput.forwardr\r]r^r.r?rrYr`rJrJrHrKry $r)eagersdpacseZdZdfdd ZddZ      ddejdeejd eejd eejd eejd ee e ejd ee de ejfddZ Z S)RobertaAttentionNcs4tt|j||d|_t||_t|_dS)Nr) r-r.ROBERTA_SELF_ATTENTION_CLASSES_attn_implementationrFroutputset pruned_headsrsrHrJrKr.s   zRobertaAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)lenrrFrgrjrrrmrnrorrrkunion)rFheadsindexrJrJrK prune_headss zRobertaAttention.prune_headsFrzr{r|r}r~rrruc Cs<||||||||}||d|} | f|dd} | S)Nrr)rFr) rFrzr{r|r}r~rr self_outputsattention_outputrrJrJrKrYs  zRobertaAttention.forwardrr) r\r]r^r.rr?rrrrrrYr`rJrJrHrKrs4 rc2eZdZfddZdejdejfddZZS)RobertaIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r-r.rrlr1intermediate_sizer isinstance hidden_actstrr intermediate_act_fnrErHrJrKr.s   zRobertaIntermediate.__init__rzrucCs||}||}|Sr)rr)rFrzrJrJrKrYs  zRobertaIntermediate.forwardrrJrJrHrKrs rcr) RobertaOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r-r.rrlrr1rr8r9r:r;r<rErHrJrKr.rzRobertaOutput.__init__rzrrucCrrrrrJrJrKrYrzRobertaOutput.forwardrrJrJrHrKrrrcseZdZfddZ      ddejdeejdeejdeejd eejd eeeejd ee d eejfd dZ ddZ Z S) RobertaLayercsrt|j|_d|_t||_|j|_|j|_|jr-|js&t|dt|dd|_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is addedr&r)r-r.chunk_size_feed_forward seq_len_dimr attentionrradd_cross_attentionrhcrossattentionr intermediaterrrErHrJrKr.s   zRobertaLayer.__init__NFrzr{r|r}r~rrruc Cs|dur |ddnd}|j|||||d} | d} |jr(| dd} | d} n| dd} d} |jro|durot|dsDtd|d|durN|d dnd}|| ||||||}|d} | |dd} |d} | | } t|j|j|j| }|f| } |jr| | f} | S) Nrf)rrrrr(rz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrrPrhrrfeed_forward_chunkrr)rFrzr{r|r}r~rrself_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputrJrJrKrYsP         zRobertaLayer.forwardcCs||}|||}|Sr)rr)rFrintermediate_outputrrJrJrKr1s  zRobertaLayer.feed_forward_chunkr) r\r]r^r.r?rrrrrrYrr`rJrJrHrKrs4  ArcseZdZfddZ         ddejdeejdeejd eejd eejd eeeejd ee d ee dee dee de eeje ffddZ Z S)RobertaEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSrJ)r).0rrGrJrK <sz+RobertaEncoder.__init__..F) r-r.rGr ModuleListrangenum_hidden_layerslayergradient_checkpointingrErHrrKr.9s   zRobertaEncoder.__init__NFTrzr{r|r}r~past_key_valuesrroutput_hidden_states return_dictruc  Cs8| rdnd} |r dnd} |r|jjrdnd} |jr%|jr%|r%tdd}|r)dnd}t|jD]K\}}| r;| |f} |durC||nd}|durM||nd}||||||||d}|d}|rg||df7}|r{| |df} |jjr{| |df} q0| r| |f} | std d ||| | | fDSt ||| | | d S) NrJzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r~rrrr(rrfcss|] }|dur|VqdSrrJ)rvrJrJrK usz)RobertaEncoder.forward..)last_hidden_staterrz attentionscross_attentions) rGrrrrr enumeraterrr)rFrzr{r|r}r~rrrrrall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_maskr layer_outputsrJrJrKrY?sd       zRobertaEncoder.forward) NNNNNNFFT)r\r]r^r.r?rrrrrrrrYr`rJrJrHrKr8sD      rcr) RobertaPoolercs*tt|j|j|_t|_dSr)r-r.rrlr1rTanh activationrErHrJrKr.s zRobertaPooler.__init__rzrucCs(|dddf}||}||}|SNr)rr)rFrzfirst_token_tensor pooled_outputrJrJrKrYs  zRobertaPooler.forwardrrJrJrHrKr s r c@s,eZdZeZdZdZgdZdZddZ dS)RobertaPreTrainedModelrobertaT)r!rarcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSt|tre|jj dSdS)zInitialize the weightsr)meanstdNg?)rrrlweightdatanormal_rGinitializer_rangebiaszero_r/r"r8fill_ RobertaLMHead)rFmodulerJrJrK _init_weightss       z$RobertaPreTrainedModel._init_weightsN) r\r]r^r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_sdparrJrJrJrKrs ra  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ) custom_introc seZdZddgZdfdd ZddZdd Zd d Ze dd e e j de e j de e j de e j de e j de e j de e j de e j de e e j de ede ede ede edeee j effddZZS) RobertaModelr!rTcsTt|||_t||_t||_|rt|nd|_|j |_ |j |_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r-r.rGr!rXrencoderr poolerrattn_implementationr% post_init)rFrGadd_pooling_layerrHrJrKr.s    zRobertaModel.__init__cC|jjSrrXr3rFrJrJrKget_input_embeddingsz!RobertaModel.get_input_embeddingscC ||j_dSrr-)rFrorJrJrKset_input_embeddings z!RobertaModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr'rrr)rFheads_to_prunerrrJrJrK _prune_headsszRobertaModel._prune_headsNrQr{r*r'r|rRr}r~rrrrrruc Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} |jjr-| dur(| n|jj} nd} |dur;|dur;td|durJ||||}n|durW|dd}ntd|\}}|durf|j n|j }| durv| ddj dnd}|durt |j dr|j j ddd|f}|||}|}n tj|tj|d}|j |||||d }|durtj|||f|d }|jd ko|jd ko|duo| }|r|dkr|jjrt||||}nt||j|d }n|||}|jjr'|dur'|\}}}||f}|durtj||d }|r!|dkr!t||j|d }n||}nd}|||jj}|j|||||| | | | | d }|d}|jdurO||nd}| s^||f|ddSt|||j |j!|j"|j#dS)NFzDYou cannot specify both input_ids and inputs_embeds at the same timer(z5You have to specify either input_ids or inputs_embedsrrfr*rL)rQr'r*rRrS)rMrr&)r) r{r|r}r~rrrrrr)r pooler_outputrrzrr)$rGrruse_return_dictrrrrh%warn_if_padding_and_no_attention_maskrCrMrrPrXr*rAr?rBrDonesr)r%rrr r,get_extended_attention_maskinvert_attention_mask get_head_maskrr'r(rrrzrr) rFrQr{r*r'r|rRr}r~rrrrrrT batch_sizerUrMrSrVrWembedding_outputuse_sdpa_attention_masksextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskencoder_outputssequence_outputrrJrJrKrYs         zRobertaModel.forward)T) NNNNNNNNNNNNN)r\r]r^r#r.r/r2r6rrr?rlistrrrrrrYr`rJrJrHrKr&sb     r&zS RoBERTa Model with a `language modeling` head on top for CLM fine-tuning. c"seZdZddgZfddZddZddZe dd ee j d ee j d ee j d ee j dee j dee j dee j dee j dee j dee e e j dee dee dee dee dee e jeffddZddZZS)RobertaForCausalLMlm_head.decoder.weightlm_head.decoder.biascs@t||jstdt|dd|_t||_| dS)NzOIf you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`Fr+ r-r.rrrwarningr&rrlm_headr*rErHrJrKr.~s    zRobertaForCausalLM.__init__cCr,rrOdecoderr.rJrJrKget_output_embeddingsr0z(RobertaForCausalLM.get_output_embeddingscCr1rrPrFnew_embeddingsrJrJrKset_output_embeddingsr3z(RobertaForCausalLM.set_output_embeddingsNrQr{r*r'r|rRr}r~labelsrrrrrrucKs|dur|n|jj}| durd} |j||||||||| | | | |d }|d}||}d}| durE| |j} |j|| fd|jji|}|s[|f|dd}|durY|f|S|St|||j |j |j |j dS)am token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, RobertaForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base") >>> config = AutoConfig.from_pretrained("FacebookAI/roberta-base") >>> config.is_decoder = True >>> model = RobertaForCausalLM.from_pretrained("FacebookAI/roberta-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NF) r{r*r'r|rRr}r~rrrrrrr0rf)losslogitsrrzrr) rGr8rrOrrM loss_functionr0rrrzrr)rFrQr{r*r'r|rRr}r~rVrrrrrkwargsrrGprediction_scoreslm_lossrrJrJrKrYsT2  zRobertaForCausalLM.forwardcs.d}|D]}|tfdd|Df7}q|S)NrJc3s$|] }|d|jVqdS)rN) index_selectrrM)r past_statebeam_idxrJrKrs"z4RobertaForCausalLM._reorder_cache..)r)rFrr`reordered_past layer_pastrJr_rK_reorder_caches z!RobertaForCausalLM._reorder_cache)NNNNNNNNNNNNNN)r\r]r^_tied_weights_keysr.rRrUrrr? LongTensorrrrrrrrYrcr`rJrJrHrKrIvsh        `rIcseZdZddgZfddZddZddZe dd ee j d ee j d ee j d ee j dee j dee j dee j dee j dee j dee dee dee de ee jeffddZZS)RobertaForMaskedLMrJrKcs@t||jrtdt|dd|_t||_| dS)NznIf you want to use `RobertaForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FrLrMrErHrJrKr.s   zRobertaForMaskedLM.__init__cCr,rrPr.rJrJrKrR r0z(RobertaForMaskedLM.get_output_embeddingscCr1rrPrSrJrJrKrUr3z(RobertaForMaskedLM.set_output_embeddingsNrQr{r*r'r|rRr}r~rVrrrruc  Cs| dur| n|jj} |j||||||||| | | d } | d}||}d}| dur@| |j} t}||d|jj| d}| sV|f| dd}|durT|f|S|St ||| j | j dS)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` N) r{r*r'r|rRr}r~rrrrr(rfrWrXrzr) rGr8rrOrrMrrvr0rrzr)rFrQr{r*r'r|rRr}r~rVrrrrrGr[masked_lm_lossloss_fctrrJrJrKrYs<  zRobertaForMaskedLM.forward) NNNNNNNNNNNN)r\r]r^rdr.rRrUrrr?rerrrrrrrYr`rJrJrHrKrfsZ      rfcs0eZdZdZfddZddZddZZS)rz*Roberta Head for masked language modeling.csdtt|j|j|_tj|j|jd|_t|j|j |_ t t |j |_|j|j _dSr)r-r.rrlr1rr8r9 layer_normr0rQ Parameterr?rBrrErHrJrKr.Zs zRobertaLMHead.__init__cKs*||}t|}||}||}|Sr)rr rjrQrFfeaturesrZrtrJrJrKrYcs   zRobertaLMHead.forwardcCs,|jjjjdkr|j|j_dS|jj|_dS)Nmeta)rQrrMrr.rJrJrK _tie_weightsmszRobertaLMHead._tie_weights)r\r]r^r_r.rYror`rJrJrHrKrWs    rz RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfddZe          ddeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e ej e ffddZZS) RobertaForSequenceClassificationcs>t||j|_||_t|dd|_t||_|dSNFrL) r-r. num_labelsrGr&rRobertaClassificationHead classifierr*rErHrJrKr.}s   z)RobertaForSequenceClassification.__init__NrQr{r*r'r|rRrVrrrruc  Cst| dur| n|jj} |j|||||||| | d } | d} || } d}|dur|| j}|jjdurW|jdkr=d|j_n|jdkrS|jt j ksN|jt j krSd|j_nd|j_|jjdkrut }|jdkro|| | }n+|| |}n%|jjdkrt}|| d|j|d}n|jjdkrt}|| |}| s| f| d d}|dur|f|S|St|| | j| jd S) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr{r*r'r|rRrrrrr regressionsingle_label_classificationmulti_label_classificationr(rfrg)rGr8rrurrM problem_typersr,r?rDrirsqueezerrvrrrzrrFrQr{r*r'r|rRrVrrrrrGrXrWrirrJrJrKrYsV     "       z(RobertaForSequenceClassification.forward NNNNNNNNNN)r\r]r^r.rrr?rerrrrrrrYr`rJrJrHrKrqvsH      rqcseZdZfddZe          ddeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e ej e ffddZZS)RobertaForMultipleChoicecs@t|t||_t|j|_t|j d|_ | dS)Nr) r-r.r&rrr:r;r<rlr1rur*rErHrJrKr.s   z!RobertaForMultipleChoice.__init__NrQr*r{rVr'r|rRrrrruc  Csz| dur| n|jj} |dur|jdn|jd} |dur%|d|dnd} |dur4|d|dnd} |durC|d|dnd}|durR|d|dnd}|dure|d|d|dnd}|j| | |||||| | d }|d}||}||}|d| }d}|dur||j }t }|||}| s|f|dd}|dur|f|S|St |||j |j dS)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nrr(r)r'r*r{r|rRrrrrfrg)rGr8rrvrCrr<rurrMrrrzr)rFrQr*r{rVr'r|rRrrr num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsrrrXreshaped_logitsrWrirrJrJrKrYsN-     z RobertaForMultipleChoice.forwardr})r\r]r^r.rrr?rerrrrrrrYr`rJrJrHrKr~sH      r~crp)RobertaForTokenClassificationcsbt||j|_t|dd|_|jdur|jn|j}t||_ t |j |j|_ | dSrr)r-r.rsr&rclassifier_dropoutr;rr:r<rlr1rur*rFrGrrHrJrKr.Fs   z&RobertaForTokenClassification.__init__NrQr{r*r'r|rRrVrrrruc  Cs| dur| n|jj} |j|||||||| | d } | d} || } || } d}|durB|| j}t}|| d|j |d}| sX| f| dd}|durV|f|S|St || | j | j dS)a- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrvrr(rfrg) rGr8rr<rurrMrrvrsrrzrr|rJrJrKrYTs:   z%RobertaForTokenClassification.forwardr})r\r]r^r.rrr?rerrrrrrrYr`rJrJrHrKrDsH     rcs(eZdZdZfddZddZZS)rtz-Head for sentence-level classification tasks.csTtt|j|j|_|jdur|jn|j}t||_ t|j|j |_ dSr) r-r.rrlr1rrr;r:r<rsout_projrrHrJrKr.s  z"RobertaClassificationHead.__init__cKsL|dddddf}||}||}t|}||}||}|Sr)r<rr?tanhrrlrJrJrKrYs     z!RobertaClassificationHead.forward)r\r]r^r_r.rYr`rJrJrHrKrts  rtcseZdZfddZe           ddeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee de e ej e ffddZZS)RobertaForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSrr) r-r.rsr&rrrlr1 qa_outputsr*rErHrJrKr.s  z$RobertaForQuestionAnswering.__init__NrQr{r*r'r|rRstart_positions end_positionsrrrruc  CsH| dur| n|jj} |j||||||| | | d } | d} || }|jddd\}}|d}|d}d}|dur|durt|dkrO|d}t|dkr\|d}|d}| d|}| d|}t |d}|||}|||}||d}| s||f| dd}|dur|f|S|St |||| j | j d S) a[ token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) Nrvrrr(r) ignore_indexrf)rW start_logits end_logitsrzr)rGr8rrsplitr{rrrCclamprrrzr)rFrQr{r*r'r|rRrrrrrrrGrXrr total_loss ignored_indexri start_lossend_lossrrJrJrKrYsP          z#RobertaForQuestionAnswering.forward) NNNNNNNNNNN)r\r]r^r.rrr?rerrrrrrrYr`rJrJrHrKrsN       rcCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)nerir?cumsumtype_asrD)rQr"rSmaskincremental_indicesrJrJrKrNs  rN)rIrfr~rrqrr&r)r)Hr_rtypingrrr?torch.utils.checkpoint packagingrrtorch.nnrrr activationsr r generationr modeling_attn_mask_utilsr rmodeling_layersrmodeling_outputsrrrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrrconfiguration_robertar get_loggerr\rModuler!rarrrrrrrrr rr&rIrfrrqr~rrtrrN__all__rJrJrJrKsv    (   Zf4WR 5\^iP X