o ei@sddlmZddlZddlmZddlmZmZmZddlm Z ddl m Z m Z ddlmZmZmZddlmZdd lmZmZdd lmZdd lmZmZmZmZmZmZm Z m!Z!dd l"m#Z#m$Z$dd l%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3e,4e5Z6Gdddej7Z8  dLdej7dej9dej9dej9dej9dBde:dBde:de&e*fdd Z;Gd!d"d"ej7ZGd'd(d(ej7Z?Gd)d*d*ej7Z@Gd+d,d,ej7ZAGd-d.d.eZBe+Gd/d0d0e$ZCGd1d2d2ej7ZDGd3d4d4ej7ZEe+d5d6Gd7d8d8eCZFe+d9d6Gd:d;d;eCeZGe+Gdd?d?ej7ZIe+d@d6GdAdBdBeCZJe+GdCdDdDeCZKe+GdEdFdFeCZLGdGdHdHej7ZMe+GdIdJdJeCZNgdKZOdS)M)CallableN)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FNgelu)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs) RobertaConfigc seZdZdZfddZ     ddejdBdejdBdejdBd ejdBd ed ej f d d Z e ddZ e dddZ ZS)RobertaEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j |_|jdt|jddd|jdtj|jtjddd|j|_tj|j|j|jd|_dS) N) padding_idxeps position_idsr#F) persistenttoken_type_ids)dtype)super__init__nn Embedding vocab_size hidden_size pad_token_idword_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangemax_position_embeddingsexpandzerosr)sizelongr&position_embeddingsselfconfig __class__j/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.pyr0;s   zRobertaEmbeddings.__init__Nr input_idsr-r) inputs_embedspast_key_values_lengthreturnc Cs|dur|dur|||j|}n|||j}|dur!|}n|dd}|\}}|durZt|drO|j|jdd} tj | d|d} | ||}n tj |tj |j j d}|durc||}||} || } ||} | | } || } || } | S)Nr+r-rr#)dimindexr.device)"create_position_ids_from_input_idsr&&create_position_ids_from_inputs_embedsrDhasattrr-rBshaper?gatherrCrEr)rUr6r8rFr9r=) rHrNr-r)rOrP input_shape batch_size seq_lengthbuffered_token_type_idsr8 embeddingsrFrLrLrMforwardOs2       zRobertaEmbeddings.forwardcCsJ|dd}|d}tj|d||dtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr+r#rTr)rDr?r@rErU unsqueezerB)rOr&r[sequence_lengthr)rLrLrMrWs  z8RobertaEmbeddings.create_position_ids_from_inputs_embedscCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor r#rR)neintr?cumsumtype_asrE)rNr&rPmaskincremental_indicesrLrLrMrVs  z4RobertaEmbeddings.create_position_ids_from_input_ids)NNNNr)r)__name__ __module__ __qualname____doc__r0r? LongTensor FloatTensorreTensorr` staticmethodrWrV __classcell__rLrLrJrMr%8s2  0 r%modulequerykeyvalueattention_maskscalingr=kwargsc Ks|dur |dd}t||dd|}|dur||}tjj|dd}tjj|||jd}t||} | dd } | |fS)Nr+rrc)ptrainingr#) rDr?matmul transposer1 functionalsoftmaxr=r~ contiguous) rtrurvrwrxryr=rz attn_weights attn_outputrLrLrMeager_attention_forwards  rcsdeZdZd fdd Z   ddejdejdBdedBdejdBd ee d e ejf d d Z Z S)RobertaSelfAttentionFNcst|j|jdkrt|dstd|jd|jd||_|j|_t|j|j|_|j|j|_ |jd|_ t |j|j |_ t |j|j |_t |j|j |_t |j|_|j|_||_||_dSNrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r{)r/r0r4num_attention_headsrX ValueErrorrIreattention_head_size all_head_sizeryr1Linearrurvrwr;attention_probs_dropout_probr= is_decoder is_causal layer_idxrHrIrrrJrLrMr0s&    zRobertaSelfAttention.__init__ hidden_statesrxpast_key_valuescache_positionrzrQcKs|jdd}g|d|jR}||j|dd}||j|dd} ||j|dd} |durP|} t|trC|j } | | | |j d|i\} } t |jjt} | ||| | |f|jsddn|jj|jd|\} }| jg|dR} | |fS)Nr+r#r|rrsr=ry)rYrruviewrrvrw isinstancer self_attention_cacheupdaterr get_interfacerI_attn_implementationrr~r=r}ryreshaper)rHrrxrrrzr[ hidden_shape query_layer key_layer value_layercurrent_past_key_valuesattention_interfacerrrLrLrMr`s@   zRobertaSelfAttention.forwardFNNNN rjrkrlr0r?rpror rrtupler`rrrLrLrJrMrs$rcsdeZdZd fdd Z   ddejdejdBdejdBdedBd ee d e ejf d d Z Z S)RobertaCrossAttentionFNcst|j|jdkrt|dstd|jd|jd||_|j|_t|j|j|_|j|j|_ |jd|_ t |j|j |_ t |j|j |_t |j|j |_t |j|_||_||_dSr)r/r0r4rrXrrIrerrryr1rrurvrwr;rr=rrrrJrLrMr0s$    zRobertaCrossAttention.__init__rencoder_hidden_statesrxrrzrQcKsN|jdd\}}|jd}||d|jf} ||d|jf} ||j| dd} |dur3|j|jnd} |durL| rL|jj |jj } |jj |jj }n-| |j| dd} | |j| dd}|dury|j| ||j\} }d|j|j<t|jjt}||| | ||f|jsdn|jj|jd|\}}|||d}||fS)Nr+r#r|FTrsr)rYrrurr is_updatedgetrcross_attention_cachelayerskeysvaluesrvrwrrrrIrrr~r=r}ryrr)rHrrrxrrzbsztgt_lensrc_len q_input_shapekv_input_shaperrrrrrrrLrLrMr`sB     zRobertaCrossAttention.forwardrr) rjrkrlr0r?rpror rrrr`rrrLrLrJrMrs$rc8eZdZfddZdejdejdejfddZZS)RobertaSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr') r/r0r1rr4denser9r:r;r<r=rGrJrLrMr0U zRobertaSelfOutput.__init__r input_tensorrQcC&||}||}|||}|SNrr=r9rHrrrLrLrMr`[  zRobertaSelfOutput.forwardrjrkrlr0r?rpr`rrrLrLrJrMrT $rcs|eZdZdfdd Z     ddejdejdBdejdBdejdBd edBd ejdBd ee d e ejfd dZ Z S)RobertaAttentionFNcs:t||_|r tnt}||||d|_t||_dS)Nrr)r/r0is_cross_attentionrrrHroutput)rHrIrrrattention_classrJrLrMr0cs  zRobertaAttention.__init__rrxrencoder_attention_maskrrrzrQc KsB|js|n|}|j|f||||d|\}} |||}|| fS)N)rrxrr)rrHr) rHrrxrrrrrzattention_outputrrLrLrMr`js  zRobertaAttention.forward)FNFNNNNNrrLrLrJrMrbs0  rc2eZdZfddZdejdejfddZZS)RobertaIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r/r0r1rr4intermediate_sizerr hidden_actstrrintermediate_act_fnrGrJrLrMr0s   zRobertaIntermediate.__init__rrQcCs||}||}|Sr)rr)rHrrLrLrMr`s  zRobertaIntermediate.forwardrrLrLrJrMrs rcr) RobertaOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r/r0r1rrr4rr9r:r;r<r=rGrJrLrMr0rzRobertaOutput.__init__rrrQcCrrrrrLrLrMr`rzRobertaOutput.forwardrrLrLrJrMrrrcseZdZdfdd Z     ddejdejdBdejdBdejdBdedBd ejdBd ee d e ejfd d Z ddZ Z S) RobertaLayerNcs~t|j|_d|_t||j|d|_|j|_|j|_|jr3|js*t|dt|d|dd|_ t ||_ t ||_ dS)Nr#rz> should be used as a decoder model if cross attention is addedFT)rrr)r/r0chunk_size_feed_forward seq_len_dimrr attentionadd_cross_attentionrcrossattentionr intermediaterr)rHrIrrJrLrMr0s"  zRobertaLayer.__init__rrxrrrrrzrQc Ks|j||f||d|\}} |} |jr7|dur7t|ds%td|d|j|d||fd|i|\} } | } t|j|j|j| } | S)N)rrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrXrrrfeed_forward_chunkrr) rHrrxrrrrrzself_attention_output_rcross_attention_output layer_outputrLrLrMr`s<     zRobertaLayer.forwardcCs||}|||}|Sr)rr)rHrintermediate_outputrrLrLrMrs  zRobertaLayer.feed_forward_chunkrr)rjrkrlr0r?rpror rrrr`rrrrLrLrJrMrs2 )rcsLeZdZeZdZdZdZdZdZ dZ e e e dZefddZZS)RobertaPreTrainedModelrobertaT)r attentionscross_attentionscsft|t|trt|jdSt|tr1t|j t |j j d dt|jdSdS)zInitialize the weightsr+r*N)r/ _init_weightsr RobertaLMHeadinitzeros_biasr%copy_r)r?r@rYrBr-)rHrtrJrLrMrs   "z$RobertaPreTrainedModel._init_weights)rjrkrlr$ config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrrr_can_record_outputsr?no_gradrrrrLrLrJrMrsrcseZdZfddZ      ddejdejdBdejdBdejdBdedBd edBd ejdBd e e d e eje Bfd dZ ZS)RobertaEncodercs4t|_tfddtjD|_dS)Ncsg|]}t|dqS))r)r).0irIrLrM sz+RobertaEncoder.__init__..)r/r0rIr1 ModuleListrangenum_hidden_layerslayerrGrJrrMr0s $zRobertaEncoder.__init__Nrrxrrr use_cacherrzrQc KsHt|jD]\} } | |||f|||d|}qt||r |dSddS)N)rrr)last_hidden_stater) enumeraterr) rHrrxrrrrrrzr layer_modulerLrLrMr`s$  zRobertaEncoder.forwardNNNNNN)rjrkrlr0r?rpror boolrrrrr`rrrLrLrJrMrs6   rcr) RobertaPoolercs*tt|j|j|_t|_dSr)r/r0r1rr4rTanh activationrGrJrLrMr0s zRobertaPooler.__init__rrQcCs(|dddf}||}||}|SNr)rr)rHrfirst_token_tensor pooled_outputrLrLrMr`#s  zRobertaPooler.forwardrrLrLrJrMrs ra  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ) custom_introcseZdZddgZdfdd ZddZdd Zeee dd e j d Bd e j d Bd e j d Bde j d Bde j d Bde j d Bde j d Bde d Bde d Bde j d Bdeedee j eBfddZddZZS) RobertaModelr%rTcsJt|||_d|_t||_t||_|rt|nd|_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) r/r0rIgradient_checkpointingr%r_rencoderrpooler post_init)rHrIadd_pooling_layerrJrLrMr0;s    zRobertaModel.__init__cC|jjSrr_r6rHrLrLrMget_input_embeddingsLz!RobertaModel.get_input_embeddingscC ||j_dSrr)rHrwrLrLrMset_input_embeddingsO z!RobertaModel.set_input_embeddingsNrNrxr-r)rOrrrrrrzrQc  KsT|jjr| dur | n|jj} nd} | r2|dur2|dus|jjr,tt|jdt|jdnt|jd}|du|duAr>td|durK|j} |jd} n|j} |jd} |dur[| nd}| durkt j ||| | d} |j |||||d}|j ||||| |d\}}|j|f||||| | |d | }|j}|jdur||nd}t|||jd S) NFrz:You must specify exactly one of input_ids or inputs_embedsr#r)rU)rNr)r-rOrP)rxrembedding_outputrrr)rxrrrrrr))r pooler_outputr)rIrris_encoder_decoderr r rrUrYget_seq_lengthr?r@r__create_attention_masksr rr rr)rHrNrxr-r)rOrrrrrrzrUr]rPrencoder_outputssequence_outputrrLrLrMr`Rsj        zRobertaModel.forwardcCsP|jjrt|j||||d}nt|j||d}|dur$t|j|||d}||fS)N)rIrOrxrr)rIrOrx)rIrOrxr)rIrrr)rHrxrrrrrrLrLrMrs* z$RobertaModel._create_attention_masks)T) NNNNNNNNNN)rjrkrl_no_split_modulesr0rrr!r"rr?rpr rrrrrr`rrrrLrLrJrMr ,sX      Mr zS RoBERTa Model with a `language modeling` head on top for CLM fine-tuning. c seZdZdddZfddZddZdd Zee dd e j d Bd e j d Bde j d Bde j d Bde j d Bde j d Bde j d Bde j d Bde e e j d Bde d Bde jd Bdee jBdeede e jeBfddZZS)RobertaForCausalLM)roberta.embeddings.word_embeddings.weight lm_head.biaszlm_head.decoder.weightzlm_head.decoder.biascs@t||jstdt|dd|_t||_| dS)NzOIf you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`Fr r/r0rloggerwarningr rrlm_headr rGrJrLrMr0s    zRobertaForCausalLM.__init__cCrrr'decoderrrLrLrMget_output_embeddingsrz(RobertaForCausalLM.get_output_embeddingscCrrr(rHnew_embeddingsrLrLrMset_output_embeddingsrz(RobertaForCausalLM.set_output_embeddingsNrrNrxr-r)rOrrlabelsrrrlogits_to_keeprzrQc  Ks|durd} |j|f||||||| | | dd | }|j}t| tr(t| dn| }||dd|ddf}d}|durL|jd|||jjd| }t |||j |j |j |j dS)am token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, RobertaForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base") >>> config = AutoConfig.from_pretrained("FacebookAI/roberta-base") >>> config.is_decoder = True >>> model = RobertaForCausalLM.from_pretrained("FacebookAI/roberta-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) rxr-r)rOrrrrr return_dict)logitsr.r3)lossr1rrrrrL)rrrreslicer' loss_functionrIr3rrrrr)rHrNrxr-r)rOrrr.rrrr/rzoutputsr slice_indicesr1r2rLrLrMr`s@1 zRobertaForCausalLM.forward) NNNNNNNNNNNr)rjrkrl_tied_weights_keysr0r*r-r rr?rnrorrrprerrrr`rrrLrLrJrMrsd        rcseZdZdddZfddZddZdd Zee dd e j d Bd e j d Bd e j d Bde j d Bde j d Bde j d Bde j d Bde j d Bde e dee jeBfddZZS)RobertaForMaskedLMr r!r"cs@t||jrtdt|dd|_t||_| dS)NznIf you want to use `RobertaForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Fr#r$rGrJrLrMr0?s   zRobertaForMaskedLM.__init__cCrrr(rrLrLrMr*Nrz(RobertaForMaskedLM.get_output_embeddingscCrrr(r+rLrLrMr-Qrz(RobertaForMaskedLM.set_output_embeddingsNrNrxr-r)rOrrr.rzrQc  Ks|j|f||||||dd| } | d} || } d} |dur7|| j}t}|| d|jj|d} t| | | j | j dS)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rxr-r)rOrrr0rNr+r2r1rr) rr'torUrrrIr3rrr)rHrNrxr-r)rOrrr.rzr5rprediction_scoresmasked_lm_lossloss_fctrLrLrMr`Ts4   zRobertaForMaskedLM.forward)NNNNNNNN)rjrkrlr7r0r*r-r rr?rnrorrrrprr`rrrLrLrJrMr88sL    r8c(eZdZdZfddZddZZS)rz*Roberta Head for masked language modeling.csZtt|j|j|_tj|j|jd|_t|j|j |_ t t |j |_dSr)r/r0r1rr4rr9r: layer_normr3r) Parameterr?rCrrGrJrLrMr0s zRobertaLMHead.__init__cKs*||}t|}||}||}|Sr)rr r?r)rHfeaturesrzxrLrLrMr`s   zRobertaLMHead.forwardrjrkrlrmr0r`rrrLrLrJrMrs rz RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfddZee      ddejdBdejdBdejdBdejdBdejdBd ejdBd e e d e ej e Bfd d ZZS) RobertaForSequenceClassificationcs>t||j|_||_t|dd|_t||_|dSNFr#) r/r0 num_labelsrIr rRobertaClassificationHead classifierr rGrJrLrMr0s   z)RobertaForSequenceClassification.__init__NrNrxr-r)rOr.rzrQc Ks6|j|f||||dd|}|d} || } d} |dur|| j}|jjdurN|jdkr4d|j_n|jdkrJ|jtj ksE|jtj krJd|j_nd|j_|jjdkrlt } |jdkrf| | | } n+| | |} n%|jjdkrt } | | d |j|d } n|jjdkrt} | | |} t| | |j|jd S) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trxr-r)rOr0rNr# regressionsingle_label_classificationmulti_label_classificationr+r9)rrJr:rUrI problem_typerHr.r?rErersqueezerrrrrr rHrNrxr-r)rOr.rzr5rr1r2r=rLrLrMr`sN     "       z(RobertaForSequenceClassification.forwardr)rjrkrlr0r rr?rnrorrrrprr`rrrLrLrJrMrFs6   rFcseZdZfddZee      ddejdBdejdBdejdBdejdBdejdBd ejdBd e e d e ej e Bfd d ZZS)RobertaForMultipleChoicecs@t|t||_t|j|_t|j d|_ | dS)Nr#) r/r0r rr1r;r<r=rr4rJr rGrJrLrMr0s   z!RobertaForMultipleChoice.__init__NrNr-rxr.r)rOrzrQcKs<|dur |jdn|jd}|dur|d|dnd} |dur*|d|dnd} |dur9|d|dnd} |durH|d|dnd} |dur[|d|d|dnd} |j| f| | | | dd|}|d}||}||}|d|}d}|dur||j}t}|||}t |||j |j dS)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nr#r+T)r)r-rxrOr0r9) rYrrDrr=rJr:rUrrrr)rHrNr-rxr.r)rOrz num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsr5rr1reshaped_logitsr2r=rLrLrMr` sF+     z RobertaForMultipleChoice.forwardr)rjrkrlr0r rr?rnrorrrrprr`rrrLrLrJrMrRs6   rRcrE)RobertaForTokenClassificationcsbt||j|_t|dd|_|jdur|jn|j}t||_ t |j |j|_ | dSrG)r/r0rHr rclassifier_dropoutr<r1r;r=rr4rJr rHrIr\rJrLrMr0as   z&RobertaForTokenClassification.__init__NrNrxr-r)rOr.rzrQc Ks|j|f||||dd|}|d} || } || } d} |dur9|| j}t} | | d|j|d} t| | |j |j dS)a- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrKrNr+r9) rr=rJr:rUrrrHrrrrQrLrLrMr`os2   z%RobertaForTokenClassification.forwardr)rjrkrlr0r rr?rnrorrrrprr`rrrLrLrJrMr[_s6  r[cr>)rIz-Head for sentence-level classification tasks.csTtt|j|j|_|jdur|jn|j}t||_ t|j|j |_ dSr) r/r0r1rr4rr\r<r;r=rHout_projr]rJrLrMr0s  z"RobertaClassificationHead.__init__cKsL|dddddf}||}||}t|}||}||}|Sr)r=rr?tanhr^rArLrLrMr`s     z!RobertaClassificationHead.forwardrDrLrLrJrMrIs  rIcseZdZfddZee       ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd e e d e ej e Bfd dZZS)RobertaForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSrG) r/r0rHr rr1rr4 qa_outputsr rGrJrLrMr0s  z$RobertaForQuestionAnswering.__init__NrNrxr-r)rOstart_positions end_positionsrzrQcKs|j|f||||dd|} | d} || } | jddd\} } | d} | d} d}|dury|duryt|dkrF|d}t|dkrS|d}| d}|d|}|d|}t|d}|| |}|| |}||d }t || | | j | j d S) a[ token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) TrKrr#r+rcN) ignore_indexr|)r2 start_logits end_logitsrr) rrasplitrPrlenrDclamprrrr)rHrNrxr-r)rOrbrcrzr5rr1rerf total_loss ignored_indexr= start_lossend_lossrLrLrMr`sH          z#RobertaForQuestionAnswering.forward)NNNNNNN)rjrkrlr0r rr?rnrorrrrprr`rrrLrLrJrMr`s<    r`)rr8rRr`rFr[r r)Nrs)Pcollections.abcrr?torch.nnr1rrrrr activationsrr cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrr utils.genericr r!utils.output_capturingr"configuration_robertar$ get_loggerrjr%Moduler%rpfloatrrrrrrrrrrrr rr8rrFrRr[rIr`__all__rLrLrLrMs     (     p IMC" nUT`F N