o ۷i-@sdZddlZddlZddlmZmZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZmZmZdd lmZdd lmZdd lmZmZmZmZmZmZmZmZdd lm Z dd l!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+e',e-Z.ddZ/Gdddej0Z1Gdddej0Z2Gdddej0Z3Gdddej0Z4Gdddej0Z5Gdddej0Z6Gd d!d!ej0Z7Gd"d#d#eZ8Gd$d%d%ej0Z9Gd&d'd'ej0Z:Gd(d)d)ej0Z;Gd*d+d+ej0Ze&Gd2d3d3e=Z?e&d4d/Gd5d6d6e=eZ@e&d7d/Gd8d9d9e=ZAe&Gd:d;d;e=ZBe&Gdd?d?e=ZDgd@ZEdS)AzPyTorch RemBERT model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging)deprecate_kwarg) RemBertConfigc szddl}ddl}ddl}Wn tytdwtj|}t d||j |}g}g} |D],\} t fdddDrFq6t dd | |j |} || | q6t|| D]\} d d d t d dDrt dd qh|} D]}} |d| r|d| }n| g}|ddks|ddkrt| d} nH|ddks|ddkrt| d} n6|ddkrt| d} n*|ddkrt| d} nz t| |d} Wntyt dd Yqwt|dkr t|d}| |} q| dddkrt| d} n | dkr'|| } z| j| jkrGsz-load_tf_weights_in_rembert..)adam_vadam_moutput_embeddingclszLoading TF weight z with shape zbert/zrembert//css|]}|dvVqdS))r(r)AdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepNr!)r"nr!r!r&r'Xs  z Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifierz Skipping {}ri _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )!renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variablesany load_variableappendzipreplacesplitjoin fullmatchgetattrAttributeErrorformatlenint transposeshape ValueErrorAssertionErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathr<nptftf_path init_varsnamesarraysrVarraypointerm_name scope_namesnumer!r$r&load_tf_weights_in_rembert1s                     rlc sheZdZdZfddZ     ddeejdeejdeejd eejd e d ej f d d Z Z S)RemBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|jdt|jddddS)N) padding_idxeps position_ids)rF) persistent)super__init__r Embedding vocab_sizeinput_embedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferrZarangeexpandselfr^ __class__r!r&rus   zRemBertEmbeddings.__init__Nr input_idstoken_type_idsrq inputs_embedspast_key_values_lengthreturnc Cs|dur |}n|dd}|d}|dur&|jdd|||f}|dur5tj|tj|jjd}|dur>||}||}||} ||} | | 7} | | } | | } | S)Nrrrdtypedevice) sizerqrZzeroslongrrzr~r|rr) rrrrqrr input_shape seq_lengthr~ embeddingsr|r!r!r&forwards"      zRemBertEmbeddings.forward)NNNNr) __name__ __module__ __qualname____doc__rurrZ LongTensor FloatTensorrTTensorr __classcell__r!r!rr&rms* rmc2eZdZfddZdejdejfddZZS) RemBertPoolercs*tt|j|j|_t|_dSr )rtrurLinear hidden_sizedenseTanh activationrrr!r&rus zRemBertPooler.__init__ hidden_statesrcCs(|dddf}||}||}|S)Nr)rr)rrfirst_token_tensor pooled_outputr!r!r&rs  zRemBertPooler.forwardrrrrurZrrrr!r!rr&rs rcseZdZdfdd Zedddd      dd ejd eejd eejd eejdee d e deejde fddZ Z S)RemBertSelfAttentionNcst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|j|_||_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())rtrurnum_attention_headshasattrrWrTattention_head_size all_head_sizerrquerykeyvaluerattention_probs_dropout_probr is_decoder layer_idxrr^rrr!r&rus   zRemBertSelfAttention.__init__past_key_valuepast_key_valuesz4.58)new_nameversionFrattention_mask head_maskencoder_hidden_statesoutput_attentionscache_positionrcCs|j\}} } |||d|j|jdd} d} |du} |dur9t|tr7|j |j } | r3|j }n|j }n|}| r=|n|}| rV|durV| rV|j |j j}|j |j j}nF|||d|j|jdd}|||d|j|jdd}|dur| s|nd}||||j d|i\}}| rt|trd|j|j <t| |dd}|t|j}|dur||}tjj|dd}||}|dur||}t||}|d ddd }|dd|jf}|j|}||fS) Nrrrr:FrTdimrr)rVrviewrrrU isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrupdaterZmatmulmathsqrtr functionalsoftmaxrpermute contiguousrr)rrrrrrrr batch_sizer_ query_layerris_cross_attentioncurr_past_key_valuecurrent_states key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shaper!r!r&rs\        zRemBertSelfAttention.forwardr NNNNFN)rrrrurrZrrrr booltuplerrr!r!rr&rs4  rc8eZdZfddZdejdejdejfddZZS)RemBertSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNro) rtrurrrrrrrrrrrr!r&ru8 zRemBertSelfOutput.__init__r input_tensorrcC&||}||}|||}|Sr rrrrrrr!r!r&r>  zRemBertSelfOutput.forwardrr!r!rr&r7 $rcseZdZdfdd ZddZ      ddejdeejd eejd eejd ee d ee d eejde ejfddZ Z S)RemBertAttentionNcs.tt||d|_t||_t|_dS)Nr)rtrurrroutputset pruned_headsrrr!r&ruFs   zRemBertAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)rSrrrrrrrrrrrrunion)rheadsindexr!r!r& prune_headsMs zRemBertAttention.prune_headsFrrrrrrrrc Cs>|j|||||||d}||d|} | f|dd} | S)Nrrrrrrrr)rr) rrrrrrrr self_outputsattention_outputoutputsr!r!r&r`s  zRemBertAttention.forwardr r)rrrrurrZrrrr rrrrr!r!rr&rEs4 rcr)RemBertIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr ) rtrurrrintermediate_sizerr hidden_actstrr intermediate_act_fnrrr!r&ruzs   zRemBertIntermediate.__init__rrcCs||}||}|Sr )rrrrr!r!r&rs  zRemBertIntermediate.forwardrr!r!rr&rys rcr) RemBertOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rtrurrrrrrrrrrrrr!r&rurzRemBertOutput.__init__rrrcCrr rrr!r!r&rrzRemBertOutput.forwardrr!r!rr&rrrcseZdZdfdd Z       ddejdeejdeejdeejd eejd eed ee d eejd e ejfddZ ddZ Z S) RemBertLayerNcstt|j|_d|_t|||_|j|_|j|_|jr.|js't|dt||d|_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is addedr)rtruchunk_size_feed_forward seq_len_dimr attentionradd_cross_attentionrWcrossattentionr intermediaterrrrr!r&rus   zRemBertLayer.__init__Frrrrencoder_attention_maskrrrrc  Cs|j||||||d} | d} | dd} |jrA|durAt|ds)td|d|j| ||||||d} | d} | | dd} t|j|j|j| } | f| } | S)N)rrrrrrrr z'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrrWr rfeed_forward_chunkrr)rrrrrr rrrself_attention_outputsrrcross_attention_outputs layer_outputr!r!r&rs>      zRemBertLayer.forwardcCs||}|||}|Sr )r r)rrintermediate_outputrr!r!r&r s  zRemBertLayer.feed_forward_chunkr )NNNNNFN)rrrrurZrrrr rrrr rr!r!rr&rs:  1rcseZdZfddZ          ddejdeejdeejd eejd eejd eed ee d e de de deejde e e ffddZ ZS)RemBertEncodercsLt|_tjj|_tfddt j D|_ d|_ dS)Ncsg|]}t|dqS)r)r)r"ir^r!r& sz+RemBertEncoder.__init__..F) rtrur^rrrxrembedding_hidden_mapping_in ModuleListrangenum_hidden_layerslayergradient_checkpointingrrrr&rus   zRemBertEncoder.__init__NFTrrrrr r use_cacheroutput_hidden_states return_dictrrc  CsV|jr|jr|rtdd}|r"|dur"tt|jdt|jd}|r3t|tr3tdt |}| |}| r.)last_hidden_staterr attentionscross_attentions)rtrainingr@ warning_oncer r r^rrfrom_legacy_cacherr  enumeraterr)rrrrrr rrrrrrall_hidden_statesall_self_attentionsall_cross_attentionsr layer_modulelayer_head_mask layer_outputsr!r!r&rsl          zRemBertEncoder.forward) NNNNNNFFTN)rrrrurZrrrr rrrrrrr!r!rr&rsJ       rcr)RemBertPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dSr) rtrurrrrrrrr transform_act_fnrrrrr!r&ru;s  z'RemBertPredictionHeadTransform.__init__rrcCs"||}||}||}|Sr )rr.rrr!r!r&rDs   z&RemBertPredictionHeadTransform.forwardrr!r!rr&r-:s  r-cr)RemBertLMPredictionHeadcsRtt|j|j|_t|j|j|_t |j |_ tj |j|j d|_ dSr)rtrurrroutput_embedding_sizerrwdecoderr rrrrrrr!r&ruLs  z RemBertLMPredictionHead.__init__rrcCs,||}||}||}||}|Sr )rrrr1rr!r!r&rSs    zRemBertLMPredictionHead.forwardrr!r!rr&r/Ks r/cr)RemBertOnlyMLMHeadcstt||_dSr )rtrur/ predictionsrrr!r&ru]s zRemBertOnlyMLMHead.__init__sequence_outputrcCs||}|Sr )r3)rr4prediction_scoresr!r!r&ras zRemBertOnlyMLMHead.forwardrr!r!rr&r2\s r2c@s*eZdZUeed<eZdZdZddZ dS)RemBertPreTrainedModelr^rembertTcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsg)meanstdNg?)rrrr3r\normal_r^initializer_ranger6zero_rvrnrfill_)rmoduler!r!r& _init_weightsms      z$RemBertPreTrainedModel._init_weightsN) rrrr__annotations__rlload_tf_weightsbase_model_prefixsupports_gradient_checkpointingr?r!r!r!r&r6fs  r6a  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ) custom_introc"seZdZdfdd ZddZddZdd Ze dd ee j d ee j d ee j dee j dee j dee j dee j dee j dee dee dee dee dee dee jdeeeffddZZS) RemBertModelTcsDt|||_t||_t||_|rt|nd|_| dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rtrur^rmrrencoderrpooler post_init)rr^add_pooling_layerrr!r&rus    zRemBertModel.__init__cCs|jjSr rrzrr!r!r&get_input_embeddingssz!RemBertModel.get_input_embeddingscCs ||j_dSr rJ)rrr!r!r&set_input_embeddingss z!RemBertModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrFrrr)rheads_to_prunerrr!r!r& _prune_headsszRemBertModel._prune_headsNrrrrqrrrr rrrrrrrc Cs,| dur| n|jj} | dur| n|jj} | dur| n|jj} |jjr-| dur(| n|jj} nd} |dur;|dur;td|durJ||||}n|durW|dd}ntd|\}}|durf|j n|j }d}| durt | t s}| ddj dn| }|durtj|||f|d}|durtj|tj|d}|||}|jjr|dur|\}}}||f}|durtj||d}||}nd}|||jj}|j|||||d }|j|||||| | | | | |d }|d}|jdur||nd}| s||f|d dSt|||j|j|j|jd S) NFzDYou cannot specify both input_ids and inputs_embeds at the same timerrz5You have to specify either input_ids or inputs_embedsrr)rr)rrqrrr) rrrr rrrrrrr)r  pooler_outputrrr!r")r^rruse_return_dictrrrW%warn_if_padding_and_no_attention_maskrrrr rVget_seq_lengthrZonesrrget_extended_attention_maskinvert_attention_mask get_head_maskrrrFrGrrrr!r")rrrrrqrrrr rrrrrrrrrrrextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputsr4rr!r!r&rs     zRemBertModel.forward)TNNNNNNNNNNNNNN)rrrrurLrMrPrrrZrrr rrrrrrrr!r!rr&rE~sf       rEcseZdZdgZfddZddZddZe            dd ee j d ee j d ee j d ee j d ee j dee j dee j dee j dee j dee dee dee de eeffddZdddZede fddZZS)RemBertForMaskedLMcls.predictions.decoder.weightcs@t||jrtdt|dd|_t||_| dS)NznIf you want to use `RemBertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FrI rtrurr@warningrEr7r2r+rHrrr!r&rus   zRemBertForMaskedLM.__init__cC |jjjSr r+r3r1rKr!r!r&get_output_embeddings- z(RemBertForMaskedLM.get_output_embeddingscC||jj_dSr rgrnew_embeddingsr!r!r&set_output_embeddings0z(RemBertForMaskedLM.set_output_embeddingsNrrrrqrrrr labelsrrrrc  Cs| dur| n|jj} |j||||||||| | | d } | d}||}d}| dur:t}||d|jj| d}| sP|f| dd}|durN|f|S|St||| j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. N) rrrqrrrr rrrrrrr:losslogitsrr!) r^rRr7r+rrrwrrr!)rrrrrqrrrr rorrrrr4r5masked_lm_lossloss_fctrr!r!r&r3s: zRemBertForMaskedLM.forwardcKs~|j}|d}|jjdusJdtj|||jddfgdd}tj|df|jjtj|jd}tj||gdd}||dS)Nrz.The PAD token should be defined for generationrrrrr)rr) rVr^ryrZcat new_zerosfullrr)rrr model_kwargsreffective_batch_size dummy_tokenr!r!r&prepare_inputs_for_generationls" z0RemBertForMaskedLM.prepare_inputs_for_generationcCsdS)z Legacy correction: RemBertForMaskedLM can't call `generate()` from `GenerationMixin`, even though it has a `prepare_inputs_for_generation` method. Fr!)r+r!r!r& can_generatezszRemBertForMaskedLM.can_generate) NNNNNNNNNNNNr )rrr_tied_weights_keysrurhrmrrrZrrrrrrrr{ classmethodr|rr!r!rr&ras`        8razS RemBERT Model with a `language modeling` head on top for CLM fine-tuning. c"seZdZdgZfddZddZddZe              dd ee j d ee j d ee j d ee j d ee j dee j dee j dee j dee dee j dee dee dee dee deeeffddZZS)RemBertForCausalLMrbcs@t||jstdt|dd|_t||_| dS)NzOIf you want to use `RemBertForCausalLM` as a standalone, add `is_decoder=True.`Frcrdrrr!r&rus    zRemBertForCausalLM.__init__cCrfr rgrKr!r!r&rhriz(RemBertForCausalLM.get_output_embeddingscCrjr rgrkr!r!r&rmrnz(RemBertForCausalLM.set_output_embeddingsNrrrrqrrrr rrorrrrrcKs|dur|n|jj}|j||||||||| | | | |d }|d}||}d}| dur9|j|| fd|jji|}|sO|f|dd}|durM|f|S|St|||j|j|j |j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, RemBertForCausalLM, RemBertConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/rembert") >>> config = RemBertConfig.from_pretrained("google/rembert") >>> config.is_decoder = True >>> model = RemBertForCausalLM.from_pretrained("google/rembert", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```N) rrrqrrrr rrrrrrrwr:)rqrrrrr!r") r^rRr7r+ loss_functionrwrrrr!r")rrrrrqrrrr rrorrrrkwargsrr4r5lm_lossrr!r!r&rsN) zRemBertForCausalLM.forwardr`)rrrr}rurhrmrrrZrrr rrrrrrr!r!rr&rsf        rz RemBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfddZe          ddeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e e ffddZ ZS) RemBertForSequenceClassificationcsJt||j|_t||_t|j|_t |j |j|_ | dSr rtru num_labelsrEr7rrclassifier_dropout_probrrrr9rHrrr!r&rus   z)RemBertForSequenceClassification.__init__Nrrrrqrrrorrrrc  Csr| dur| n|jj} |j|||||||| | d } | d} || } || } d}|dur|jjdurV|jdkr 1` a classification loss is computed (Cross-Entropy). Nrrrqrrrrrr regressionsingle_label_classificationmulti_label_classificationrrr:rp)r^rRr7rr9 problem_typerrrZrrTrsqueezerrrrrr!)rrrrrqrrrorrrrrrrrqrtrr!r!r&rsV     "       z(RemBertForSequenceClassification.forward NNNNNNNNNN)rrrrurrrZrrrrrrrrr!r!rr&rsH      rcr)RemBertForMultipleChoicecs@t|t||_t|j|_t|j d|_ | dS)Nr) rtrurEr7rrrrrrr9rHrrr!r&ruNs   z!RemBertForMultipleChoice.__init__Nrrrrqrrrorrrrc  Csn| dur| n|jj} |dur|jdn|jd} |dur%|d|dnd}|dur4|d|dnd}|durC|d|dnd}|durR|d|dnd}|dure|d|d|dnd}|j|||||||| | d } | d} || } || }|d| }d}|durt}|||}| s|f| dd}|dur|f|S|St ||| j | j dS)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrrrrrr:rp) r^rRrVrrr7rr9rrrr!)rrrrrqrrrorrr num_choicesrrrrreshaped_logitsrqrtrr!r!r&rXsL,    z RemBertForMultipleChoice.forwardr)rrrrurrrZrrrrrrrrr!r!rr&rLsH      rcr)RemBertForTokenClassificationcsNt||j|_t|dd|_t|j|_t |j |j|_ | dSNFrcrrrr!r&rus  z&RemBertForTokenClassification.__init__Nrrrrqrrrorrrrc  Cs| dur| n|jj} |j|||||||| | d } | d} || } || } d}|dursn    (    S7m4G\  hjTgB N