o ei@sddlmZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZmZmZmZmZmZmZdd lmZmZdd lm Z ddl!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*ddl+m,Z,e%-e.Z/Gdddej0Z1ej2e1dZ3Gdddej0Z4  dadej0dej5dej5dej5dej5dBde6dBde6d e e#fd!d"Z7Gd#d$d$ej0Z8Gd%d&d&ej0Z9Gd'd(d(ej0Z:Gd)d*d*ej0Z;Gd+d,d,ej0ZGd1d2d2ej0Z?Gd3d4d4ej0Z@Gd5d6d6ej0ZAGd7d8d8eZBGd9d:d:ej0ZCGd;d<dd>ej0ZEGd?d@d@ej0ZFGdAdBdBej0ZGGdCdDdDej0ZHe$GdEdFdFeZIee$dGdHGdIdJdJe"ZJe$GdKdLdLeIZKe$dMdHGdNdOdOeIZLe$GdPdQdQeIZMGdRdSdSej0ZNe$dTdHGdUdVdVeIZOe$dWdHGdXdYdYeIZPe$GdZd[d[eIZQe$Gd\d]d]eIZRe$Gd^d_d_eIZSgd`ZTdS)b)Callable) dataclassN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs)MobileBertConfigcs4eZdZdfdd ZdejdejfddZZS) NoNormNcs2ttt||_tt||_dSN) super__init__r Parametertorchzerosbiasonesweight)self feat_sizeeps __class__p/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.pyr$8s zNoNorm.__init__ input_tensorreturncCs||j|jSr")r*r()r+r2r0r0r1forward=szNoNorm.forwardr"__name__ __module__ __qualname__r$r&Tensorr4 __classcell__r0r0r.r1r!7sr!) layer_normno_normc sbeZdZdZfddZ    d dejdBdejdBdejdBdejdBd ejf d d Z Z S) MobileBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cst|j|_|j|_|j|_tj|j|j|jd|_ t|j |j|_ t|j |j|_ |jr4dnd}|j|}t||j|_t|j|j|_t|j|_|jdt|j ddddS)N) padding_idxrr position_idsrF) persistent)r#r$ trigram_inputembedding_size hidden_sizer Embedding vocab_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddingsLinearembedding_transformationNORM2FNnormalization_type LayerNormDropouthidden_dropout_probdropoutregister_bufferr&arangeexpand)r+configembed_dim_multiplierembedded_input_sizer.r0r1r$Gs   zMobileBertEmbeddings.__init__N input_idstoken_type_idsr? inputs_embedsr3c Cs*|dur |}n|dd}|d}|dur$|jddd|f}|dur3tj|tj|jjd}|dur<||}|jrktjt j j |ddddfgddd|t j j |ddddfgdddgdd }|jst|j |j kry||}||}||}|||} || } || } | S) NrAr)dtypedevice)rrrrrr)value)rrrrrrdim)sizer?r&r'longr`rIrCcatr functionalpadrDrErOrKrMrRrU) r+r\r]r?r^ input_shape seq_lengthrKrM embeddingsr0r0r1r4]s4  $$      zMobileBertEmbeddings.forward)NNNN) r6r7r8__doc__r$r& LongTensor FloatTensorr9r4r:r0r0r.r1r=Ds$ r=ramodulequerykeyrbattention_maskscalingrUkwargsc Ks|dur |dd}t||dd|}|dur||}tjj|dd}tjj|||jd}t||} | dd } | |fS)NrArcrrd)ptrainingr) rfr&matmul transposerrisoftmaxrUry contiguous) rqrrrsrbrtrurUrv attn_weights attn_outputr0r0r1eager_attention_forwards  rcsXeZdZfddZ d dejdejdejdejdBdeed e ejf d d Z Z S) MobileBertSelfAttentioncst||_|j|_t|j|j|_|j|j|_|jd|_t |j|j|_ t |j|j|_ t |j r<|jn|j|j|_t |j|_d|_dS)NrwF)r#r$rYnum_attention_headsinttrue_hidden_sizeattention_head_size all_head_sizerurrNrrrsuse_bottleneck_attentionrErbrSattention_probs_dropout_probrU is_causalr+rYr.r0r1r$s   z MobileBertSelfAttention.__init__N query_tensor key_tensor value_tensorrtrvr3cKs|jdd}g|d|jR}||j|dd}||j|dd} ||j|dd} t|j j t } | ||| | |f|j sIdn|j j|jd|\} } | jg|dR} | | fS)NrArrcra)rUru)shaperrrviewr{rsrbr get_interfacerY_attn_implementationrryrUrxrureshaper})r+rrrrtrvrk hidden_shape query_layer key_layer value_layerattention_interfacerr~r0r0r1r4s,  zMobileBertSelfAttention.forwardr" r6r7r8r$r&r9rprrtupler4r:r0r0r.r1rs  rc8eZdZfddZdejdejdejfddZZS)MobileBertSelfOutputcsXt|j|_t|j|j|_t|j|j|j d|_ |js*t |j |_ dSdSNr-)r#r$use_bottleneckrrNrdenserPrQlayer_norm_epsrRrSrTrUrr.r0r1r$s zMobileBertSelfOutput.__init__ hidden_statesresidual_tensorr3cCs,||}|js ||}|||}|Sr")rrrUrRr+rr layer_outputsr0r0r1r4s  zMobileBertSelfOutput.forwardr5r0r0r.r1rs $rcs^eZdZfddZ d dejdejdejdejdejdBd eed e ejfd d Z Z S)MobileBertAttentionc"tt||_t||_dSr")r#r$rr+routputrr.r0r1r$  zMobileBertAttention.__init__Nrrr layer_inputrtrvr3c Ks0|j||||fi|\}}|||}||fSr")r+r) r+rrrrrtrvattention_outputr~r0r0r1r4s  zMobileBertAttention.forwardr"rr0r0r.r1rs$  rc2eZdZfddZdejdejfddZZS)MobileBertIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr") r#r$rrNrintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrr.r0r1r$s   zMobileBertIntermediate.__init__rr3cCs||}||}|Sr")rrr+rr0r0r1r4  zMobileBertIntermediate.forwardr5r0r0r.r1rs rcr)OutputBottleneckcsFtt|j|j|_t|j|j|j d|_ t |j |_ dSr)r#r$rrNrrErrPrQrrRrSrTrUrr.r0r1r$s zOutputBottleneck.__init__rrr3cCs&||}||}|||}|Sr")rrUrRrr0r0r1r4$s  zOutputBottleneck.forwardr5r0r0r.r1rs $rcs>eZdZfddZdejdejdejdejfddZZS) MobileBertOutputcs\t|j|_t|j|j|_t|j |j|_ |js't |j |_ dSt||_dSr")r#r$rrrNrrrrPrQrRrSrTrUr bottleneckrr.r0r1r$,s zMobileBertOutput.__init__intermediate_statesresidual_tensor_1residual_tensor_2r3cCsJ||}|js||}|||}|S|||}|||}|Sr")rrrUrRr)r+rrr layer_outputr0r0r1r46s   zMobileBertOutput.forwardr5r0r0r.r1r+s  rcr)BottleneckLayerc8tt|j|j|_t|j|j|j d|_ dSr) r#r$rrNrEintra_bottleneck_sizerrPrQrrRrr.r0r1r$D zBottleneckLayer.__init__rr3cCs||}||}|Sr"rrR)r+rrr0r0r1r4IrzBottleneckLayer.forwardr5r0r0r.r1rC rcs6eZdZfddZdejdeejfddZZS) Bottleneckcs<t|j|_|j|_t||_|jrt||_dSdSr")r#r$key_query_shared_bottleneckrrinput attentionrr.r0r1r$Ps  zBottleneck.__init__rr3cCsB||}|jr |fdS|jr||}||||fS||||fS)N)rrrr)r+rbottlenecked_hidden_statesshared_attention_inputr0r0r1r4Xs     zBottleneck.forward r6r7r8r$r&r9rr4r:r0r0r.r1rOs "rcr) FFNOutputcrr) r#r$rrNrrrrPrQrrRrr.r0r1r$trzFFNOutput.__init__rrr3cCs||}|||}|Sr"rrr0r0r1r4ys zFFNOutput.forwardr5r0r0r.r1rss $rcr)FFNLayercrr")r#r$r intermediaterrrr.r0r1r$rzFFNLayer.__init__rr3cCs||}|||}|Sr")rr)r+rintermediate_outputrr0r0r1r4s  zFFNLayer.forwardr5r0r0r.r1rrrc sLeZdZfddZ d dejdejdBdeede ejfdd Z Z S) MobileBertLayercs~tj|_j|_t|_t|_t|_ |jr$t |_ jdkr=t fddtjdD|_dSdS)Nrcg|]}tqSr0)r.0_rYr0r1 z,MobileBertLayer.__init__..)r#r$rnum_feedforward_networksrrrrrrrrr ModuleListrangeffnrr.rr1r$s      (zMobileBertLayer.__init__Nrrtrvr3cKs|jr ||\}}}}n |gd\}}}}|j|||||fi|\}} |} |jdkr6|jD]} | | } q/|| } || | |} | S)Nrr)rrrrrrr)r+rrtrvrrrrself_attention_outputrr ffn_modulerrr0r0r1r4s&     zMobileBertLayer.forwardr"rr0r0r.r1rs rc sJeZdZfddZ d dejdejdBdeede e Bfdd Z Z S) MobileBertEncodercs.ttfddtjD|_dS)Ncrr0)rrrr0r1rrz.MobileBertEncoder.__init__..)r#r$rrrnum_hidden_layerslayerrr.rr1r$s $zMobileBertEncoder.__init__Nrrtrvr3cKs0t|jD] \}}|||fi|}qt|dS)N)last_hidden_state) enumeraterr )r+rrtrvi layer_moduler0r0r1r4s zMobileBertEncoder.forwardr") r6r7r8r$r&r9rprrrr r4r:r0r0r.r1rs rcr)MobileBertPoolercs2t|j|_|jrt|j|j|_dSdSr")r#r$classifier_activation do_activaterrNrErrr.r0r1r$s zMobileBertPooler.__init__rr3cCs2|dddf}|js |S||}t|}|S)Nr)rrr&tanh)r+rfirst_token_tensor pooled_outputr0r0r1r4s   zMobileBertPooler.forwardr5r0r0r.r1rs rcr)!MobileBertPredictionHeadTransformcsXtt|j|j|_t|jtrt |j|_ n|j|_ t d|j|j d|_ dS)Nr;r)r#r$rrNrErrrrr transform_act_fnrPrrRrr.r0r1r$s  z*MobileBertPredictionHeadTransform.__init__rr3cCs"||}||}||}|Sr")rrrRrr0r0r1r4s   z)MobileBertPredictionHeadTransform.forwardr5r0r0r.r1r  rcr)MobileBertLMPredictionHeadcs^tt||_tj|j|j|jdd|_ tj|j|jdd|_ t t |j|_dS)NF)r(T)r#r$r transformrrNrGrErDrdecoderr%r&r'r(rr.r0r1r$s  z#MobileBertLMPredictionHead.__init__rr3cCs>||}|tj|jj|jjgdd}||jj7}|S)Nrrd) rrzr&rhrr*trr(rr0r0r1r4s $ z"MobileBertLMPredictionHead.forwardr5r0r0r.r1rrrcr)MobileBertOnlyMLMHeadcstt||_dSr")r#r$r predictionsrr.r0r1r$s zMobileBertOnlyMLMHead.__init__sequence_outputr3cC||}|Sr")r)r+rprediction_scoresr0r0r1r4 zMobileBertOnlyMLMHead.forwardr5r0r0r.r1r rcs<eZdZfddZdejdejdeejfddZZS)MobileBertPreTrainingHeadscs(tt||_t|jd|_dSNrc)r#r$rrrrNrEseq_relationshiprr.r0r1r$ s  z#MobileBertPreTrainingHeads.__init__rrr3cCs||}||}||fSr")rr)r+rrrseq_relationship_scorer0r0r1r4s  z"MobileBertPreTrainingHeads.forwardrr0r0r.r1r s (rcsPeZdZUeed<dZdZdZdZdZ dZ e e dZ efddZZS)MobileBertPreTrainedModelrY mobilebertT)r attentionscst|t|trt|jt|jdSt|t r&t|jdSt|t r>t |j t |j jdddSdS)zInitialize the weightsrAr@N)r# _init_weightsrr!initzeros_r(ones_r*rr=copy_r?r&rWrrX)r+rqr.r0r1r$s     &z'MobileBertPreTrainedModel._init_weights)r6r7r8r __annotations__base_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrr_can_record_outputsr&no_gradrr:r0r0r.r1rs rz6 Output type of [`MobileBertForPreTraining`]. ) custom_introc@steZdZUdZdZejdBed<dZejdBed<dZ ejdBed<dZ e ejdBed<dZ e ejdBed<dS)MobileBertForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsrr) r6r7r8rnr r&rprrrrrrr0r0r0r1r 1s  r cseZdZdZdfdd ZddZddZeee dd e j d Bd e j d Bd e j d Bd e j d Bde j d Bde edeeBfddZZS)MobileBertModelz2 https://huggingface.co/papers/2004.02984 TcsJt|||_d|_t||_t||_|rt|nd|_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) r#r$rYgradient_checkpointingr=rmrencoderrpooler post_init)r+rYadd_pooling_layerr.r0r1r$Ps    zMobileBertModel.__init__cCs|jjSr"rmrIr+r0r0r1get_input_embeddingsasz$MobileBertModel.get_input_embeddingscCs ||j_dSr"r)r+rbr0r0r1set_input_embeddingsds z$MobileBertModel.set_input_embeddingsNr\rtr]r?r^rvr3c Ks||du|duAr td|j||||d}t|j||d}|j|fd|i|}|d} |jdur6|| nd} t| | dS)Nz:You must specify exactly one of input_ids or inputs_embeds)r\r?r]r^)rYr^rtrtr)r pooler_output) ValueErrorrmr rYrrr) r+r\rtr]r?r^rvembedding_outputencoder_outputsrrr0r0r1r4gs4 zMobileBertModel.forward)T)NNNNN)r6r7r8rnr$rrrrrr&rorprrrrr4r:r0r0r.r1rJs8rz MobileBert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. cseZdZdddZfddZddZdd Zdd ed Bd ej ffd d Z e e dde jd Bde jd Bde jd Bde jd Bde jd Bde jd Bde jd Bdeed eeBfddZZS)MobileBertForPreTrainingcls.predictions.bias,mobilebert.embeddings.word_embeddings.weightzcls.predictions.decoder.biaszcls.predictions.decoder.weightc,t|t||_t||_|dSr")r#r$rrrclsrrr.r0r1r$s    z!MobileBertForPreTraining.__init__cC |jjjSr"r#rrrr0r0r1get_output_embeddings z.MobileBertForPreTraining.get_output_embeddingscC||jj_|j|jj_dSr"r#rrr(r+new_embeddingsr0r0r1set_output_embeddings z.MobileBertForPreTraining.set_output_embeddingsNnew_num_tokensr3c*|j|jjj|dd|jj_tj|dSNT)r. transposed)r._get_resized_lm_headr#rrr#resize_token_embeddingsr+r.r.r0r1r4s  z0MobileBertForPreTraining.resize_token_embeddingsr\rtr]r?r^labelsnext_sentence_labelrvcKs|j|f||||dd|} | dd\} } || | \} } d}|durJ|durJt}|| d|jj|d}|| dd|d}||}t|| | | j| jdS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Examples: ```python >>> from transformers import AutoTokenizer, MobileBertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForPreTraining.from_pretrained("google/mobilebert-uncased") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) >>> # Batch size 1 >>> outputs = model(input_ids) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ```Trtr]r?r^ return_dictNrcrA)r rrrr) rr#rrrYrGr rr)r+r\rtr]r?r^r6r7rvoutputsrrrr total_lossloss_fctmasked_lm_lossnext_sentence_lossr0r0r1r4s4) z MobileBertForPreTraining.forwardr"NNNNNNN)r6r7r8_tied_weights_keysr$r&r,rrrFr4rrr&rorprrrr r4r:r0r0r.r1rsH   rcseZdZdddZfddZddZdd Zdd ed Bd ej ffd d Z e e dde jd Bde jd Bde jd Bde jd Bde jd Bde jd Bdeed eeBfddZZS)MobileBertForMaskedLMrr r!cs6t|t|dd|_t||_||_|dSNF)r)r#r$rrrr#rYrrr.r0r1r$s   zMobileBertForMaskedLM.__init__cCr$r"r%rr0r0r1r& r'z+MobileBertForMaskedLM.get_output_embeddingscCr(r"r)r*r0r0r1r, r-z+MobileBertForMaskedLM.set_output_embeddingsNr.r3cr/r0r2r5r.r0r1r4s  z-MobileBertForMaskedLM.resize_token_embeddingsr\rtr]r?r^r6rvc Ksr|j|f||||dd|}|d} || } d} |dur/t} | | d|jj|d} t| | |j|jdS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Tr8rNrAr logitsrr) rr#rrrYrGrrr) r+r\rtr]r?r^r6rvr:rrr=r<r0r0r1r4s. zMobileBertForMaskedLM.forwardr"NNNNNN)r6r7r8r@r$r&r,rrrFr4rrr&rorprrrrr4r:r0r0r.r1rAsB   rAcr)MobileBertOnlyNSPHeadcstt|jd|_dSr)r#r$rrNrErrr.r0r1r$Ds zMobileBertOnlyNSPHead.__init__rr3cCrr")r)r+rrr0r0r1r4HrzMobileBertOnlyNSPHead.forwardr5r0r0r.r1rFCrrFzZ MobileBert Model with a `next sentence prediction (classification)` head on top. cseZdZfddZee      ddejdBdejdBdejdBdejdBdejdBd ejdBd e e d e e Bfd d Z ZS)#MobileBertForNextSentencePredictioncr"r")r#r$rrrFr#rrr.r0r1r$Ss    z,MobileBertForNextSentencePrediction.__init__Nr\rtr]r?r^r6rvr3c Ksn|j|f||||dd|}|d} || } d} |dur-t} | | dd|d} t| | |j|jdS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`. - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Examples: ```python >>> from transformers import AutoTokenizer, MobileBertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForNextSentencePrediction.from_pretrained("google/mobilebert-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> loss = outputs.loss >>> logits = outputs.logits ```Tr8rNrArcrC)rr#rrrrr) r+r\rtr]r?r^r6rvr:rrr>r<r0r0r1r4\s.& z+MobileBertForNextSentencePrediction.forwardrE)r6r7r8r$rrr&rorprrrrr4r:r0r0r.r1rGMs6   rGz MobileBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfddZee      ddejdBdejdBdejdBdejdBdejdBd ejdBd ee d e eje Bfd d Z Z S)#MobileBertForSequenceClassificationcsdt||j|_||_t||_|jdur|jn|j}t ||_ t |j |j|_ |dSr")r#r$ num_labelsrYrrclassifier_dropoutrTrrSrUrNrE classifierrr+rYrKr.r0r1r$s    z,MobileBertForSequenceClassification.__init__Nr\rtr]r?r^r6rvr3c Ks4|j|f||||dd|}|d} || } || } d} |dur|jjdurM|jdkr3d|j_n|jdkrI|jtjksD|jtj krId|j_nd|j_|jjdkrkt } |jdkre| | | } n+| | |} n%|jjdkrt } | | d|j| d} n|jjdkrt} | | |} t| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Tr8rN regressionsingle_label_classificationmulti_label_classificationrArC)rrUrLrY problem_typerJr_r&rgrrsqueezerrrrrr) r+r\rtr]r?r^r6rvr:rrDr r<r0r0r1r4sN     "       z+MobileBertForSequenceClassification.forwardrE)r6r7r8r$rrr&r9rrrrr4r:r0r0r.r1rIs6  rIcseZdZfddZee       ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd ee d e eje Bfd dZ Z S)MobileBertForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSrB) r#r$rJrrrrNrE qa_outputsrrr.r0r1r$s  z'MobileBertForQuestionAnswering.__init__Nr\rtr]r?r^start_positions end_positionsrvr3cKs|j|f||||dd|} | d} || } | jddd\} } | d} | d} d}|dury|duryt|dkrF|d}t|dkrS|d}| d}|d|}|d|}t|d}|| |}|| |}||d}t || | | j | j d S) NTr8rrrArd) ignore_indexrc)r  start_logits end_logitsrr) rrTsplitrRr}lenrfclamprrrr)r+r\rtr]r?r^rUrVrvr:rrDrXrYr; ignored_indexr< start_lossend_lossr0r0r1r4sH           z&MobileBertForQuestionAnswering.forwardr?)r6r7r8r$rrr&r9rrrrr4r:r0r0r.r1rSs<    rScrH)MobileBertForMultipleChoicecsTt|t||_|jdur|jn|j}t||_t |j d|_ | dS)Nr) r#r$rrrKrTrrSrUrNrErLrrMr.r0r1r$;s    z$MobileBertForMultipleChoice.__init__Nr\rtr]r?r^r6rvr3cKs0|dur |jdn|jd}|dur|d|dnd}|dur*|d|dnd}|dur9|d|dnd}|durH|d|dnd}|dur[|d|d|dnd}|j|f||||dd|} | d} || } || } | d|} d} |durt}|| |} t| | | j| j dS)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) NrrATr8rC) rrrfrrUrLrrrr)r+r\rtr]r?r^r6rv num_choicesr:rrDreshaped_logitsr r<r0r0r1r4HsD*    z#MobileBertForMultipleChoice.forwardrE)r6r7r8r$rrr&r9rrrrr4r:r0r0r.r1r`8s6   r`crH) MobileBertForTokenClassificationcsbt||j|_t|dd|_|jdur|jn|j}t||_ t |j |j|_ | dSrB)r#r$rJrrrKrTrrSrUrNrErLrrMr.r0r1r$s   z)MobileBertForTokenClassification.__init__Nr\rtr]r?r^r6rvr3c Ksz|j|f||||dd|}|d} || } || } d} |dur3t} | | d|j|d} t| | |j|jdS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Tr8rNrArC) rrUrLrrrJrrr) r+r\rtr]r?r^r6rvr:rrDr r<r0r0r1r4s0  z(MobileBertForTokenClassification.forwardrE)r6r7r8r$rrr&r9rrrrr4r:r0r0r.r1rds6  rd) rAr`rGrrSrIrdrrr)Nra)Ucollections.abcr dataclassesrr&rtorch.nnrrrr r activationsr masking_utilsr modeling_layersr modeling_outputsr rrrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_mobilebertr get_loggerr6loggerModuler!rRrPr=r9floatrrrrrrrrrrrrrrrrrrrr rrrArFrGrIrSr`rd__all__r0r0r0r1s       (     S 4 $  ,  FbI JPCa ;