o i@sdZddlZddlmZmZddlZddlmZddlmZm Z m Z ddl m Z m Z ddlmZmZmZdd lmZdd lmZdd lmZmZmZmZmZmZmZmZdd lm Z dd l!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+e',e-Z.dZ/Gdddej0Z1Gdddej0Z2Gdddej0Z3de2iZ4Gdddej0Z5Gdddej0Z6Gdddej0Z7Gd d!d!eZ8Gd"d#d#ej0Z9Gd$d%d%ej0Z:e&Gd&d'd'e Z;e&Gd(d)d)e;ZGd0d1d1ej0Z?e&d2d+Gd3d4d4e;Z@e&Gd5d6d6e;ZAe&Gd7d8d8e;ZBGd9d:d:ej0ZCe&Gd;d<dZEgd?ZFdS)AzPyTorch Data2VecText model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNgelu)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging)deprecate_kwarg)Data2VecTextConfigcs4eZdZdZfddZ d ddZdd ZZS) Data2VecTextForTextEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dd|j|_tj|j|j|jd|_ dS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rF) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr&register_buffertorcharangeexpandzerosr(sizelongr#selfconfig __class__g/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/data2vec/modeling_data2vec_text.pyr/:s"   z&Data2VecTextForTextEmbeddings.__init__Nrc Cs|dur|durt||j|}n||}|dur|}n|dd}|d}|durTt|drI|jddd|f}||d|} | }n tj|tj |j j d}|dur]| |}| |} || } |jdkrt||} | | 7} || } || } | S)Nr)rr+rr-devicer')"create_position_ids_from_input_idsr#&create_position_ids_from_inputs_embedsrDhasattrr+rBr@rCrEr(rNr4r8r&r6r9r=) rG input_idsr+r( inputs_embedspast_key_values_length input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr8 embeddingsr6rKrKrLforwardSs0         z%Data2VecTextForTextEmbeddings.forwardcCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr)rrMr)rDr@rAr#rErN unsqueezerB)rGrSrUsequence_lengthr(rKrKrLrP{s  zDData2VecTextForTextEmbeddings.create_position_ids_from_inputs_embeds)NNNNr)__name__ __module__ __qualname____doc__r/rZrP __classcell__rKrKrIrLr"4s   (r"cseZdZdfdd Zedddd      dd ejd eejd eejd eejdee d ee deejde ejfddZ Z S)Data2VecTextSelfAttentionNcst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|p\t|dd|_|jdksh|jd kry|j|_t d |jd |j|_|j|_||_dS) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r&r' relative_keyrelative_key_queryr!r)r.r/r2num_attention_headsrQ ValueErrorintattention_head_size all_head_sizerLinearquerykeyvaluer;attention_probs_dropout_probr=r>r&r5r0distance_embedding is_decoder layer_idxrGrHr&rsrIrKrLr/s,   z"Data2VecTextSelfAttention.__init__past_key_valuepast_key_values4.58new_nameversionF hidden_statesattention_mask head_maskencoder_hidden_statesoutput_attentionscache_positionreturncCs|j\}} } ||} | |d|j|jdd} d} |du} |dur;t|tr9|j |j } | r5|j }n|j }n|}| r?|n|}| rX|durX| rX|j |j j}|j |j j}nJ||}||d|j|jdd}||}||d|j|jdd}|dur| s|nd}||||j d|i\}}| rt|trd|j|j <t| |dd}|jdks|jd kr4| jd|jd}}|durtj|dtj|jd dd}ntj|tj|jd dd}tj|tj|jd dd}||}|||jd}|j| jd }|jdkrtd | |}||}n|jd kr4td | |}td ||}|||}|t |j}|durE||}t!j"j#|dd}|$|}|dur[||}t||}|%dddd&}|'dd|j(f}||}||fS)Nr)rr!FrTrerfrMr,zbhld,lrd->bhlrzbhrd,lrd->bhlrdimrr))shapermviewrgrj transpose isinstancer is_updatedgetrscross_attention_cacheself_attention_cachelayerskeysvaluesrnroupdater@matmulr&tensorrErNrArqr5tor-einsummathsqrtr functionalsoftmaxr=permute contiguousrDrk)rGr{r|r}r~rvrr batch_sizerV_ query_layerris_cross_attentioncurr_past_key_valuecurrent_states key_layer value_layerattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shaperKrKrLrZs                 z!Data2VecTextSelfAttention.forwardNNNNNNFN)r]r^r_r/rr@Tensorr FloatTensorr booltuplerZrarKrKrIrLrbs4  rbc8eZdZfddZdejdejdejfddZZS)Data2VecTextSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr$) r.r/rrlr2denser9r:r;r<r=rFrIrKrLr/ zData2VecTextSelfOutput.__init__r{ input_tensorrcC&||}||}|||}|SNrr=r9rGr{rrKrKrLrZ  zData2VecTextSelfOutput.forwardr]r^r_r/r@rrZrarKrKrIrLr $reagercseZdZdfdd ZddZedddd      dd ejd eej d eej deej dee dee deejde ejfddZ ZS)Data2VecTextAttentionNcs6tt|j|||d|_t||_t|_dS)Nr&rs) r.r/$DATA2VEC_TEXT_SELF_ATTENTION_CLASSES_attn_implementationrGroutputset pruned_headsrtrIrKrLr/)s   zData2VecTextAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)lenrrGrgrjrrrmrnrorrrkunion)rGheadsindexrKrKrL prune_heads3s z!Data2VecTextAttention.prune_headsrurvrwrxFr{r|r}r~rrrc Cs>|j|||||||d}||d|} | f|dd} | S)Nr|r}r~rvrrrr)rGr) rGr{r|r}r~rvrr self_outputsattention_outputoutputsrKrKrLrZEs  zData2VecTextAttention.forwardrr)r]r^r_r/rrr@rrrr rrrZrarKrKrIrLr(s6  rc2eZdZfddZdejdejfddZZS)Data2VecTextIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r.r/rrlr2intermediate_sizerr hidden_actstrr intermediate_act_fnrFrIrKrLr/`s   z!Data2VecTextIntermediate.__init__r{rcCs||}||}|Sr)rr)rGr{rKrKrLrZhs  z Data2VecTextIntermediate.forwardrrKrKrIrLr_s rcr)Data2VecTextOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r.r/rrlrr2rr9r:r;r<r=rFrIrKrLr/przData2VecTextOutput.__init__r{rrcCrrrrrKrKrLrZvrzData2VecTextOutput.forwardrrKrKrIrLrorrcseZdZdfdd Zedddd       dd ejd eejd eejd eejd eejdee dee deejde ejfddZ ddZ ZS)Data2VecTextLayerNcsxt|j|_d|_t||d|_|j|_|j|_|jr0|js(t|dt|d|d|_ t ||_ t ||_ dS)Nrrsz> should be used as a decoder model if cross attention is addedr'r)r.r/chunk_size_feed_forward seq_len_dimr attentionrradd_cross_attentionrhcrossattentionr intermediaterrrGrHrsrIrKrLr/s  zData2VecTextLayer.__init__rurvrwrxFr{r|r}r~encoder_attention_maskrrrc  Cs|j||||||d} | d} | dd} |jrA|durAt|ds)td|d|j| ||||||d} | d} | | dd} t|j|j|j| } | f| } | S)N)r|r}rrvrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrrQrhrrfeed_forward_chunkrr)rGr{r|r}r~rrvrrself_attention_outputsrrcross_attention_outputs layer_outputrKrKrLrZs>      zData2VecTextLayer.forwardcCs||}|||}|Sr)rr)rGrintermediate_outputrrKrKrLrs  z$Data2VecTextLayer.feed_forward_chunkr)NNNNNFN)r]r^r_r/rr@rrrr rrrZrrarKrKrIrLr~s<   0rcseZdZdfdd Z          ddejdeejdeejd eejd eejd eed ee d ee dee dee deejde e eje ffddZ ZS)Data2VecTextEncoderNcs:t|_tfddtjD|_d|_dS)Ncsg|]}t|dqS)r)r).0irHrKrL sz0Data2VecTextEncoder.__init__..F) r.r/rHr ModuleListrangenum_hidden_layerslayergradient_checkpointingrrIrrLr/s   zData2VecTextEncoder.__init__FTr{r|r}r~rrv use_cacheroutput_hidden_states return_dictrrc  Cs`| rdnd} |r dnd} |r|jjrdnd}|jr%|jr%|r%tdd}|r<|jjr<|dur sz.Data2VecTextEncoder.forward..)last_hidden_statervr{ attentionscross_attentions)rHrrtraininglogger warning_oncerrr r rrfrom_legacy_cache enumeraterr)rGr{r|r}r~rrvrrrrrall_hidden_statesall_self_attentionsall_cross_attentionsr layer_modulelayer_head_mask layer_outputsrKrKrLrZsl        zData2VecTextEncoder.forwardr) NNNNNNFFTN)r]r^r_r/r@rrrr rrrrrZrarKrKrIrLrsJ      rcr)Data2VecTextPoolercs*tt|j|j|_t|_dSr)r.r/rrlr2rTanh activationrFrIrKrLr/#s zData2VecTextPooler.__init__r{rcCs(|dddf}||}||}|SNr)rr )rGr{first_token_tensor pooled_outputrKrKrLrZ(s  zData2VecTextPooler.forwardrrKrKrIrLr"s rc@s.eZdZUeed<dZdZddgZddZdS) Data2VecTextPreTrainedModelrH data2vec_textTr"rcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rlt |drY|jdurY|jj t |drn|jdurp|jjddSdSdSdS)zInitialize the weightsg)meanstdNbiasweightg?)rrrlrdatanormal_rHinitializer_rangerzero_r0r#r9rQfill_)rGmodulerKrKrL _init_weights8s"      z)Data2VecTextPreTrainedModel._init_weightsN) r]r^r_r __annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesrrKrKrKrLr 1s  r c"seZdZdZdfdd ZddZddZd d Ze dd e e j d e e j de e j de e j de e j de e j de e j de e j de e de e de e de e de e de e j deee j effddZZS)Data2VecTextModela2 The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in *Attention is all you need*_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. .. _*Attention is all you need*: https://huggingface.co/papers/1706.03762 TcsDt|||_t||_t||_|rt|nd|_| dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r.r/rHr"rYrencoderrpooler post_init)rGrHadd_pooling_layerrIrKrLr/\s    zData2VecTextModel.__init__cC|jjSrrYr4rGrKrKrLget_input_embeddingslz&Data2VecTextModel.get_input_embeddingscC ||j_dSrr$)rGrorKrKrLset_input_embeddingso z&Data2VecTextModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rGheads_to_prunerrrKrKrL _prune_headsrszData2VecTextModel._prune_headsNrRr|r+r(r}rSr~rrvrrrrrrc Csd| dur| n|jj} | dur| n|jj} | dur| n|jj} |jjr-| dur(| n|jj} nd} |dur;|dur;td|durJ||||}n|durW|dd}ntd|\}}|durf|j n|j }d}| durt | t s}| ddj dn| }|durtj|||f|d}|durt|jdr|jjddd|f}|||}|}n tj|tj|d }|||}|jjr|dur|\}}}||f}|durtj||d}||}nd}|||jj}|j|||||d }|j|||||| | | | | |d }|d}|jdur||nd}| s$||f|d dSt|||j|j|j|j d S)NFzDYou cannot specify both input_ids and inputs_embeds at the same timer)z5You have to specify either input_ids or inputs_embedsrr)rNr+rM)rRr(r+rSrT) r|r}r~rrvrrrrrr)r pooler_outputrvr{rr)!rHrruse_return_dictrrrrh%warn_if_padding_and_no_attention_maskrDrNrr rget_seq_lengthr@onesrQrYr+rBrCrEget_extended_attention_maskinvert_attention_mask get_head_maskrrr rrvr{rr) rGrRr|r+r(r}rSr~rrvrrrrrrUrrVrNrTrWrXextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputssequence_outputr rKrKrLrZzs       zData2VecTextModel.forward)T)NNNNNNNNNNNNNN)r]r^r_r`r/r&r)r-rrr@rr rrrrrZrarKrKrIrLrKsh     rzX Data2VecText Model with a `language modeling` head on top for CLM fine-tuning. ) custom_introc$seZdZddgZfddZddZddZe dd ee j d ee j d ee j d ee j dee j dee j dee j dee j dee j dee dee dee dee dee dee jdeeeff ddZZS)Data2VecTextForCausalLMlm_head.decoder.weightlm_head.decoder.biascs@t||jstdt|dd|_t||_| dS)NzTIf you want to use `Data2VecTextLMHeadModel` as a standalone, add `is_decoder=True.`Fr" r.r/rrrwarningrrData2VecTextLMHeadlm_headr!rFrIrKrLr/s    z Data2VecTextForCausalLM.__init__cCr#rrFdecoderr%rKrKrLget_output_embeddingsr'z-Data2VecTextForCausalLM.get_output_embeddingscCr(rrGrGnew_embeddingsrKrKrLset_output_embeddingsr*z-Data2VecTextForCausalLM.set_output_embeddingsNrRr|r+r(r}rSr~rlabelsrvrrrrrrcKs|dur|n|jj}| durd} |j||||||||| | | | ||d}|d}||}d}| dur@|j|| fd|jji|}|sV|f|dd}|durT|f|S|St|||j|j|j |j dS)aA labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, Data2VecTextForCausalLM, Data2VecTextConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/data2vec-text-base") >>> config = Data2VecTextConfig.from_pretrained("facebook/data2vec-text-base") >>> config.is_decoder = True >>> model = Data2VecTextForCausalLM.from_pretrained("facebook/data2vec-text-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NF) r|r+r(r}rSr~rrvrrrrrrr1r!)losslogitsrvr{rr) rHr/rrF loss_functionr1rrvr{rr)rGrRr|r+r(r}rSr~rrMrvrrrrrkwargsrr=prediction_scoreslm_lossrrKrKrLrZ sT* zData2VecTextForCausalLM.forward)NNNNNNNNNNNNNNN)r]r^r__tied_weights_keysr/rIrLrrr@ LongTensorrr rrrrrrZrarKrKrIrLr?sl        r?cseZdZddgZfddZddZddZe dd ee j d ee j d ee j d ee j dee j dee j dee j dee j dee j dee dee dee de eeffddZZS)Data2VecTextForMaskedLMr@rAcs@t||jrtdt|dd|_t||_| dS)NzsIf you want to use `Data2VecTextForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FrBrCrFrIrKrLr/hs   z Data2VecTextForMaskedLM.__init__cCr#rrGr%rKrKrLrIwr'z-Data2VecTextForMaskedLM.get_output_embeddingscCr(rrGrJrKrKrLrLzr*z-Data2VecTextForMaskedLM.set_output_embeddingsNrRr|r+r(r}rSr~rrMrrrrc  Cs| dur| n|jj} |j||||||||| | | d } | d}||}d}| dur@t}| |j} ||d|jj| d}| sV|f| dd}|durT|f|S|St ||| j | j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` N) r|r+r(r}rSr~rrrrrr)r!rNrOr{r) rHr/rrFrrrNrr1rr{r)rGrRr|r+r(r}rSr~rrMrrrrr=rRmasked_lm_lossloss_fctrrKrKrLrZ}s<  zData2VecTextForMaskedLM.forward) NNNNNNNNNNNN)r]r^r_rTr/rIrLrrr@rUrrrrrrZrarKrKrIrLrVdsZ       rVcs0eZdZdZfddZddZddZZS)rEz/Data2VecText Head for masked language modeling.csdtt|j|j|_tj|j|jd|_t|j|j |_ t t |j |_|j|j _dSr)r.r/rrlr2rr9r: layer_normr1rH Parameterr@rCrrFrIrKrLr/s zData2VecTextLMHead.__init__cKs*||}t|}||}||}|Sr)rr rZrHrGfeaturesrQxrKrKrLrZs   zData2VecTextLMHead.forwardcCs,|jjjjdkr|j|j_dS|jj|_dS)Nmeta)rHrrNtyper%rKrKrL _tie_weightsszData2VecTextLMHead._tie_weights)r]r^r_r`r/rZrararKrKrIrLrEs    rEz Data2VecText Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfddZe          ddeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e e ffddZ ZS)%Data2VecTextForSequenceClassificationcs>t||j|_||_t|dd|_t||_|dSNFrB) r.r/ num_labelsrHrrData2VecTextClassificationHead classifierr!rFrIrKrLr/s   z.Data2VecTextForSequenceClassification.__init__NrRr|r+r(r}rSrMrrrrc  Cst| dur| n|jj} |j|||||||| | d } | d} || } d}|dur|| j}|jjdurW|jdkr=d|j_n|jdkrS|jt j ksN|jt j krSd|j_nd|j_|jjdkrut }|jdkro|| | }n+|| |}n%|jjdkrt}|| d|j|d}n|jjdkrt}|| |}| s| f| d d}|dur|f|S|St|| | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr|r+r(r}rSrrrrr regressionsingle_label_classificationmulti_label_classificationr)r!rW)rHr/rrgrrN problem_typerer-r@rErirsqueezerrrrr{rrGrRr|r+r(r}rSrMrrrrr=rOrNrYrrKrKrLrZsV     "       z-Data2VecTextForSequenceClassification.forward NNNNNNNNNN)r]r^r_r/rrr@rUrrrrrrZrarKrKrIrLrcsH      rccseZdZfddZe          ddeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e e ffddZ ZS)Data2VecTextForMultipleChoicecs@t|t||_t|j|_t|j d|_ | dS)Nr) r.r/rrrr;r<r=rlr2rgr!rFrIrKrLr/5s   z&Data2VecTextForMultipleChoice.__init__NrRr+r|rMr(r}rSrrrrc  Csz| dur| n|jj} |dur|jdn|jd} |dur%|d|dnd} |dur4|d|dnd} |durC|d|dnd}|durR|d|dnd}|dure|d|d|dnd}|j| | |||||| | d }|d}||}||}|d| }d}|durt}| |j }|||}| s|f|dd}|dur|f|S|St |||j |j dS)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nrr)r)r(r+r|r}rSrrrr!rW)rHr/rrrDrr=rgrrrNrr{r)rGrRr+r|rMr(r}rSrrr num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsrr rOreshaped_logitsrNrYrrKrKrLrZ?sN,     z%Data2VecTextForMultipleChoice.forwardro)r]r^r_r/rrr@rUrrrrrrZrarKrKrIrLrp3sH      rpcrb)"Data2VecTextForTokenClassificationcsbt||j|_t|dd|_|jdur|jn|j}t||_ t |j |j|_ | dSrd)r.r/rerrclassifier_dropoutr<rr;r=rlr2rgr!rGrHryrIrKrLr/s   z+Data2VecTextForTokenClassification.__init__NrRr|r+r(r}rSrMrrrrc  Cs| dur| n|jj} |j|||||||| | d } | d} || } || } d}|durBt}|| j}|| d|j |d}| sX| f| dd}|durV|f|S|St || | j | j dS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrhrr)r!rW) rHr/rr=rgrrrNrrerr{rrnrKrKrLrZs:   z*Data2VecTextForTokenClassification.forwardro)r]r^r_r/rrr@rUrrrrrrZrarKrKrIrLrxsH     rxcs(eZdZdZfddZddZZS)rfz-Head for sentence-level classification tasks.csTtt|j|j|_|jdur|jn|j}t||_ t|j|j |_ dSr) r.r/rrlr2rryr<r;r=reout_projrzrIrKrLr/s  z'Data2VecTextClassificationHead.__init__cKsL|dddddf}||}||}t|}||}||}|Sr )r=rr@tanhr{r\rKrKrLrZs     z&Data2VecTextClassificationHead.forward)r]r^r_r`r/rZrarKrKrIrLrfs  rfcseZdZfddZe           ddeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee de e e ffddZ ZS) Data2VecTextForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSrd) r.r/rerrrrlr2 qa_outputsr!rFrIrKrLr/s  z)Data2VecTextForQuestionAnswering.__init__NrRr|r+r(r}rSstart_positions end_positionsrrrrc  CsH| dur| n|jj} |j||||||| | | d } | d} || }|jddd\}}|d}|d}d}|dur|durt|dkrO|d}t|dkr\|d}|d}| d|}| d|}t |d}|||}|||}||d}| s||f| dd}|dur|f|S|St |||| j | j dS) Nrhrrr)r) ignore_indexr!)rN start_logits end_logitsr{r)rHr/rr~splitrmrrrDclamprrr{r)rGrRr|r+r(r}rSrrrrrrr=rOrr total_loss ignored_indexrY start_lossend_lossrrKrKrLrZsP          z(Data2VecTextForQuestionAnswering.forward) NNNNNNNNNNN)r]r^r_r/rrr@rUrrrrrrZrarKrKrIrLr}sN       r}cCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)nerir@cumsumtype_asrE)rRr#rTmaskincremental_indicesrKrKrLrOIs  rO)r?rVrpr}rcrxrr )r)Gr`rtypingrrr@rtorch.nnrrr activationsr r cache_utilsr r r generationrmodeling_layersrmodeling_outputsrrrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrutils.deprecationrconfiguration_data2vec_textr get_loggerr]r_HIDDEN_STATES_START_POSITIONModuler"rbrrrrrrrrr rr?rVrErcrprxrfr}rO__all__rKrKrKrLsn   (    Z7I[&nTUhH M