o ߥi6@sddlZddlZddlZddlZddlmZddlmZmZddl Z ddl m m Z ddlZ ddlZddl mZmZmZm Z ddlmZmZddlmZddlmZmZmZmZmZddlmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)m*Z*m+Z+m,Z,dd l-m.Z.dd l/m0Z0dd l1m2Z2dd l3m4Z4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;ddle0?Z@dZAdZBddZCGddde jDZEGddde jDZFGddde jDZGGddde jDZHGddde jDZIGd d!d!e jDZJGd"d#d#e jDZKGd$d%d%e jDZLGd&d'd'e jDZMGd(d)d)e jDZNGd*d+d+e jDZOGd,d-d-e jDZPGd.d/d/e jDZQGd0d1d1e jDZRGd2d3d3e jDZSGd4d5d5e)ZTeGd6d7d7eZUGd8d9d9eTZVGd:d;d;eTZWGdd?d?e jDZYGd@dAdAeTZZGdBdCdCeTZ[GdDdEdEeTZ\GdFdGdGeTZ]GdHdIdIeTZ^GdJdKdKeTZ_GdLdMdMeTZ`GdNdOdOe5e)Zae7jbe;jce2jddPGdQdRdReaZedS)SN) dataclass)OptionalTuple)Tensordevicedtypenn)CrossEntropyLossMSELoss)ACT2FN) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardreplace_return_docstrings) )BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModelapply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) BertConfig)logging)Models)Model TorchModel)MODELS)AttentionBackboneModelOutput)Tasks)parse_labels_in_orderr BertTokenizerc Cszddl}ddl}ddl}Wn tytdwtj|}t d ||j |}g}g} |D]\} } t d | | |j || } || | | q6t|| D]\} } | d} tdd| Drxt d d| qZ|} | D]|}|d |r|d |}n|g}|dd ks|dd krt| d} nH|ddks|ddkrt| d} n6|ddkrt| d} n*|ddkrt| d} nz t| |d} Wntyt d d| Yq|wt|dkrt|d}| |} q||dddkrt| d} n |d kr|| } z| j| jks'Jd| jd| jdWntyA}z |j| j| jf7_d}~wwt d | t| | _qZ|S)z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z(Converting TensorFlow checkpoint from {}z"Loading TF weight {} with shape {}/css|]}|dvVqdS))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0nr.r._/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/multi_modal/mgeo/backbone.py Msz*load_tf_weights_in_bert..z Skipping {}z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifieri _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight {})renumpy tensorflow ImportErrorloggererrorospathabspathinfoformattrainlist_variables load_variableappendzipsplitanyjoin fullmatchgetattrAttributeErrorlenint transposeshapeAssertionErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathr?nptftf_path init_varsnamesarraysnamerXarraypointerm_name scope_namesnumer.r.r1load_tf_weights_in_bert0s                     rncs@eZdZdZfddZ           dddZZS) GisEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csVttj|j|jdd|_t|j|j|_tj|j |jdd|_ tj|j |jdd|_ tj|j |jdd|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_t|drtj|j|jdd|_tj|j|jdd|_tj|j|jdd|_tj|j|jd|_t|j|_ |!dt"#|j$dt%|dd|_&||_'dS) Nr padding_idxprov_vocab_sizeeps position_idsr=position_embedding_typeabsolute)(super__init__r Embedding vocab_size hidden_sizeword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddingsrel_type_vocab_sizerel_type_embeddingsabsolute_x_vocab_sizeabsolute_x_embeddingsabsolute_y_vocab_sizeabsolute_y_embeddingsrelative_x_vocab_sizerelative_x_embeddingsrelative_y_vocab_sizerelative_y_embeddingshasattrrrprov_embeddingscity_vocab_sizecity_embeddingsdist_vocab_sizedist_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferr[arangeexpandrSrxr_selfr_ __class__r.r1r{}s`             zGisEmbeddings.__init__Nrc Cs|dur |} n|dd} | d} |dur&|jdd|| |f}|dur5tj| tj|jjd}|dur>||}||}||}|jdkrU| |}||7}|| |7}|| |dddddf7}|| |dddddf7}|| |dddddf7}|| |dddddf7}|| |dddddf7}|||dddddf7}|| |dddddf7}|||dddddf7}| dur||| 7}||| 7}||| 7}||}||}|S)Nrwr=rrryrr<)sizerur[zeroslongrrrrxrrrrrrrrrrr)r input_idstoken_type_idsru inputs_embedspast_key_values_length rel_type_idsabsolute_position_idsrelative_position_idsprov_idscity_idsdist_ids input_shape seq_lengthr embeddingsrr.r.r1forwardsj                 zGisEmbeddings.forward) NNNNrNNNNNN__name__ __module__ __qualname____doc__r{r __classcell__r.r.rr1rozs +rocs:eZdZdZfddZ        dddZZS) BertEmbeddingszKConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|dt|jdt|dd|_||_dS)Nrprsrurvrxry)rzr{rr|r}r~ pad_token_idrrrrrrrrrrrr[rrrSrxr_rrr.r1r{s2  zBertEmbeddings.__init__Nrc Cs|dur |} n|dd} | d} |dur&|jdd|| |f}|dur5tj| tj|jjd}|dur>||}||} || } |jdkrU| |} | | 7} | | } | | } | S)Nrwr=rry) rrur[rrrrrrxrrr)rrrrurrrrrrrrrrr.r.r1rs0         zBertEmbeddings.forward)NNNNrNNNrr.r.rr1rs rcsZeZdZfddZddZddZddZd d Zd d Z dddZ Z S)BertSelfAttentioncs"t||_|j|jdkrt|dstd|j|jf|j|_t|j|j|_|j|j|_ t |j|j |_ |rQt |j |j |_t |j |j |_nt |j|j |_t |j|j |_t |j|_t|dd|_|jdks{|jdkr|j|_t d|jd |j|_d |_dS) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)rxry relative_keyrelative_key_queryr<r=F)rzr{r_r~num_attention_headsr ValueErrorrVattention_head_size all_head_sizerLinearquery encoder_widthkeyvaluerattention_probs_dropout_probrrSrxrr|distance_embeddingsave_attentionrr_is_cross_attentionrr.r1r{/sB    zBertSelfAttention.__init__cC ||_dSNattn_gradients)rrr.r.r1save_attn_gradientsQ z%BertSelfAttention.save_attn_gradientscC|jSrrrr.r.r1get_attn_gradientsTz$BertSelfAttention.get_attn_gradientscCrr attention_map)rrr.r.r1save_attention_mapWrz$BertSelfAttention.save_attention_mapcCrrrrr.r.r1get_attention_mapZrz#BertSelfAttention.get_attention_mapcCs6|dd|j|jf}|j|}|ddddS)Nrwrr<r=r)rrrviewpermute)rx new_x_shaper.r.r1transpose_for_scores]s  z&BertSelfAttention.transpose_for_scoresNFcCs~||}|du} | r|||} |||} |}n;|durI|||} |||} tj|d| gdd} tj|d| gdd} n|||} |||} ||} | | f}t| | dd} |jdksv|jdkr| d}tj |tj |j d  dd}tj |tj |j d  dd}||}|||jd}|j| jd }|jdkrtd | |}| |} n|jdkrtd | |}td | |}| ||} | t|j} |dur| |} tjdd| }| r|jr||||j||}|dur ||}t|| }|dddd }| dd|jf}|j |}|r5||fn|f}||f}|S)Nrr<dimr=rwrrrrzbhld,lrd->bhlrzbhrd,lrd->bhlrr) rrrrr[catmatmulrWrxrrrrrrrtoreinsummathsqrtrrSoftmaxrr register_hookrrr contiguousr)r hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsmixed_query_layerr key_layer value_layer query_layerattention_scoresrposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probsattention_probs_dropped context_layernew_context_layer_shapeoutputsr.r.r1rcs                 zBertSelfAttention.forwardNNNNNF) rrrr{rrrrrrrr.r.rr1r-s " rc$eZdZfddZddZZS)BertSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNrs) rzr{rrr~denserrrrrrrr.r1r{ zBertSelfOutput.__init__cC&||}||}|||}|Srr rrrr input_tensorr.r.r1r  zBertSelfOutput.forwardrrrr{rrr.r.rr1r rcs<eZdZd fdd ZddZ      d ddZZS) BertAttentionFcs,tt|||_t||_t|_dSr)rzr{rrroutputset pruned_headsrrr.r1r{s    zBertAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrr=r)rUrrrrrrrrrrr runion)rheadsindexr.r.r1 prune_headss   zBertAttention.prune_headsNc Cs<||||||||}||d|} | f|dd} | S)Nrr=)rr) rrrrrrrr self_outputsattention_outputrr.r.r1rs   zBertAttention.forwardFr)rrrr{rrrr.r.rr1rsrcr)BertIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) rzr{rrr~intermediate_sizer  isinstance hidden_actstrr intermediate_act_fnrrr.r1r{ s   zBertIntermediate.__init__cC||}||}|Sr)r r"rrr.r.r1r  zBertIntermediate.forwardrr.r.rr1r s rcr) BertOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rzr{rrrr~r rrrrrrrr.r1r{r zBertOutput.__init__cCr rr r r.r.r1r$rzBertOutput.forwardrr.r.rr1r&rr&cs:eZdZfddZ      d ddZddZZS) BertLayercsft||_|j|_d|_t||_||jk|_|jr'||_ t|dd|_ t ||_ t ||_dS)Nr=T)r)rzr{r_chunk_size_feed_forward seq_len_dimr attention fusion_layerhas_cross_attention layer_numcrossattentionr intermediater&r)rr_r-rr.r1r{-s    zBertLayer.__init__NFc Cs |dur |ddnd}|j|||||d} | d} | dd} | d} |jrz|dus/Jdt|tkrc|j| ||||j|jjt|||j|jjt||d} | d} | | dd} n|j| |||||d} | d} | | dd} t |j |j |j | }|f| } | | f} | S)Nr<rrrr=rwz>encoder_hidden_states must be given for cross-attention layers)r) r*r,typelistr.r-r_r+rUrfeed_forward_chunkr(r))rrrrrrrrself_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attention_outputs layer_outputr.r.r1r<sv      zBertLayer.forwardcCs||}|||}|Sr)r/r)rrintermediate_outputr8r.r.r1r3s  zBertLayer.feed_forward_chunkr)rrrr{rr3rr.r.rr1r'+s  Dr'cs:eZdZfddZ          d ddZZS) BertEncodercs4t|_tfddtjD|_dS)Ncsg|]}t|qSr.)r')r/ir_r.r1 z(BertEncoder.__init__..)rzr{r_r ModuleListrangenum_hidden_layerslayerrrr<r1r{s  zBertEncoder.__init__NFT multi_modalc  s| rdnd} r dnd} r|jjrdnd}|rdnd}| dkr'd}|jj}n"| dkr2d}|jj}n| dkr?|jj}|jj}n | dkrId}|jj}t||D]k}|j|}| r\| |f} |durd||nd}|durn||ndt|jddr|jr|rt d d}fd d }t j j |||||||}n ||||||}|d}|r||d f7}r| |d f} qN| r| |f} | st dd||| | |fDSt||| | |dS)Nr.textrrfusionrCgradient_checkpointingFzh`use_cache=True` is incompatible with `config.gradient_checkpointing=True`. Setting `use_cache=False`...csfdd}|S)Ncsg|RSrr.)inputs)modulerrr.r1custom_forwards zJBertEncoder.forward..create_custom_forward..custom_forwardr.)rHrIr0)rHr1create_custom_forwardsz2BertEncoder.forward..create_custom_forwardrwr=css|] }|dur|VqdSrr.)r/vr.r.r1r2s z&BertEncoder.forward..)last_hidden_statepast_key_valuesr attentionscross_attentions)r_add_cross_attentionr+rAr@rBrStrainingrCwarnr[utils checkpointtupler)rrrrrrrM use_cacheroutput_hidden_states return_dictmodeall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cache start_layer output_layerr; layer_modulelayer_head_maskrJ layer_outputsr.r0r1rs            zBertEncoder.forward) NNNNNNFFTrCrr.r.rr1r:s  r:cr) BertPoolercs*tt|j|j|_t|_dSr)rzr{rrr~r Tanh activationrrr.r1r{s zBertPooler.__init__cCs(|dddf}||}||}|SNr)r re)rrfirst_token_tensor pooled_outputr.r.r1rs  zBertPooler.forwardrr.r.rr1rc rccr)BertPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dSr) rzr{rrr~r rr r!r transform_act_fnrrrrr.r1r{ s   z$BertPredictionHeadTransform.__init__cCs"||}||}||}|Sr)r rkrr$r.r.r1rs   z#BertPredictionHeadTransform.forwardrr.r.rr1rj s  rjcr)BertLMPredictionHeadcsLtt||_tj|j|jdd|_t t |j|_ |j |j_ dSNF)r8) rzr{rj transformrrr~r}decoder Parameterr[rr8rrr.r1r{ s   zBertLMPredictionHead.__init__cCr#rrnror$r.r.r1r.r%zBertLMPredictionHead.forwardrr.r.rr1rls rlcr)BertOnlyMLMHeadcstt||_dSr)rzr{rl predictionsrrr.r1r{6s zBertOnlyMLMHead.__init__cC||}|Sr)rs)rsequence_outputprediction_scoresr.r.r1r: zBertOnlyMLMHead.forwardrr.r.rr1rr4 rrcr)BertOnlyNSPHeadcstt|jd|_dSNr<)rzr{rrr~seq_relationshiprrr.r1r{As zBertOnlyNSPHead.__init__cCrtr)r{)rrhseq_relationship_scorer.r.r1rErwzBertOnlyNSPHead.forwardrr.r.rr1ry?rxrycr)BertPreTrainingHeadscs(tt||_t|jd|_dSrz)rzr{rlrsrrr~r{rrr.r1r{Ls  zBertPreTrainingHeads.__init__cCs||}||}||fSr)rsr{)rrurhrvr|r.r.r1rQs  zBertPreTrainingHeads.forwardrr.r.rr1r}Jrir}c@s*eZdZdZeZeZdZdgZ ddZ dS)BertPreTrainedModel An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. bertrucCs~t|tjtjfr|jjjd|jjdnt|tj r(|j j |jj dt|tjr;|j dur=|j j dSdSdS)z Initialize the weights meanstd?N) rrrr|r5r]normal_r_initializer_rangerr8zero_fill_rrHr.r.r1 _init_weightsbs  z!BertPreTrainedModel._init_weightsN) rrrrr config_classrnload_tf_weightsbase_model_prefix_keys_to_ignore_on_load_missingrr.r.r.r1r~Ws r~c@sleZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ee ejed<dZ ee ejed<dS)BertForPreTrainingOutputa Output type of :class:`~transformers.BertForPreTraining`. Args: loss (`optional`, returned when ``labels`` is provided, ``torch.FloatTensor`` of shape :obj:`(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``output_hidden_states=True`` is passed or when ``config.output_hidden_states=True``): Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape :obj:`(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``output_attentions=True`` is passed or when ``config.output_attentions=True``): Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossprediction_logitsseq_relationship_logitsrrN)rrrrrrr[ FloatTensor__annotations__rrrrrNr.r.r.r1rps rc seZdZdZdfdd ZddZddZd d Zd ed e e d e de def ddZ                      dddZZS) BertModelaS Noted that the bert model here is slightly updated from original bert, so we maintian the code here independently. The Bert Model transformer outputting raw hidden-states without any specific head on top. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`BertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. TcsZt|||_|jdkrt||_nt||_t||_|r$t |nd|_ | dSrf) rzr{r_ gis_embeddingrrror:encoderrcpooler init_weights)rr_add_pooling_layerrr.r1r{s      zBertModel.__init__cCs|jjSrrrrr.r.r1get_input_embeddingszBertModel.get_input_embeddingscCs ||j_dSrr)rrr.r.r1set_input_embeddingss zBertModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrBr*r)rheads_to_prunerBrr.r.r1 _prune_headsszBertModel._prune_headsrrr is_decoderreturnc CsX|dkr|dddddddf}n|dkr|r|\}}tj||d}|ddddf||d|ddddfk} | |j} | jd|jdkrl|jd| jd} tjtj||| f|| jd| gdd} | dddddddf|ddddddf}n|ddddddf}n t d ||j|j|jd }d |d }|S) a{ Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (:obj:`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (:obj:`Tuple[int]`): The shape of the input to the model. device: (:obj:`torch.device`): The device of the input to the model. Returns: :obj:`torch.Tensor` The extended attention mask, with a the same dtype as :obj:`attention_mask.dtype`. rNr<rr=)rrrw)axiszAWrong shape for input_ids (shape {}) or attention_mask (shape {})rrg) rr[rrepeatrrrXronesrrI) rrrrrextended_attention_mask batch_sizerseq_ids causal_maskprefix_seq_lenr.r.r1get_extended_attention_masks\       z%BertModel.get_extended_attention_maskNFrCc& s| dur| njj} | dur| njj} |dur|njj}|r+| dur&| njj} nd} |dur9|dur9td|durI|}|\}}|j}n,|dur]|dd}|\}}|j}n|durq|dd}|\}}|j}ntd| dur| ddjdnd}|durt j |||f|d}|durt j |t j |d } ||||}|durt|tkr|d\}}}n|\}}}||f} t| tkrׇfd d | D}!n| durt j | |d} | }!n| }!nd}!|jj}|durj|||||||||||d }"n|}"j|"||||!| | | | ||d }#|#d}$jdur/|$nd}%|s>|$|%f|#ddSt|$|%|#j|#j|#j|#jdS)a Args: input_ids (`torch.LongTensor` of shape `((batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`BertTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `((batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `((batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `((batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `((batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~file_utils.ModelOutput`] instead of a plain tuple. encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Others (**kwargs) some additional parameters might passed in from upstream pipeline, which not influence the results. NFzDYou cannot specify both input_ids and inputs_embeds at the same timerwzGYou have to specify either input_ids or inputs_embeds or encoder_embedsrr<rrcg|]}|qSr.)invert_attention_mask)r/maskrr.r1r=sz%BertModel.forward..) rrurrrrrrrrr) rrrrrMrVrrWrXrYr=rL pooler_outputrMrrNrO)r_rrWuse_return_dictrVrrrrXr[rrrrr1r2r get_head_maskrArrrrrMrrNrO)&rrrrrurrencoder_embedsrrrMrVrrWrXrrYrrrrrrrrrrrrencoder_batch_sizeencoder_sequence_length_encoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputsrurhr.rr1r)so         zBertModel.forward)T)NNNNNNNNNNNNNNFrCNNNNNN)rrrrr{rrrrrrVrboolrrrr.r.rr1rsL' NrcsLeZdZfddZddZddZ           d dd ZZS) BertForPreTrainingc,t|t||_t||_|dSr)rzr{rrr}clsrrrr.r1r{     zBertForPreTraining.__init__cC |jjjSrrrsrorr.r.r1get_output_embeddings(rz(BertForPreTraining.get_output_embeddingscC||jj_dSrrrnew_embeddingsr.r.r1set_output_embeddings+z(BertForPreTraining.set_output_embeddingsNc  Cs| dur| n|jj} |j||||||| | | d } | dd\} }|| |\}}d}|durS|durSt}||d|jj|d}||dd|d}||}| sj||f| dd}|durh|f|S|St|||| j| j dS)a labels (:obj:`torch.LongTensor` of shape ``(batch_size, sequence_length)``, `optional`): Labels for computing the masked language modeling loss. Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`` next_sentence_label (``torch.LongTensor`` of shape ``(batch_size,)``, `optional`): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see :obj:`input_ids` docstring) Indices should be in ``[0, 1]``: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. kwargs (:obj:`Dict[str, any]`, optional, defaults to `{}`): Used to hide legacy arguments that have been deprecated. Returns: Example: >>> from transformers import BertTokenizer, BertForPreTraining >>> import torch >>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') >>> model = BertForPreTraining.from_pretrained('bert-base-uncased') >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits NrrrurrrrWrXr<rw)rrrrrN) r_rrrr rr}rrrN)rrrrrurrlabelsnext_sentence_labelrrWrXrrurhrvr| total_lossloss_fctmasked_lm_lossnext_sentence_lossrr.r.r1r.sV%  zBertForPreTraining.forward NNNNNNNNNNN)rrrr{rrrrr.r.rr1rs  rcseZdZdgZddgZfddZddZdd Z  dddZ dddZ ddZ Z S)BertLMHeadModelrrupredictions.decoder.biasc0t|t|dd|_t||_|dSNFrrzr{rrrrrrrrr.r1r{   zBertLMHeadModel.__init__cCrrrrr.r.r1rrz%BertLMHeadModel.get_output_embeddingscCrrrrr.r.r1rrz%BertLMHeadModel.set_output_embeddingsNTrrCrFcCs|dur|n|jj}| durd} |j||||||||| | | | |||d}|d}||}|r>|ddddddfSd}| dur~|ddddddf}| ddddf} t|d}||d|jj| d}||dd d}|durt j t j |dd|dd }|| d k d}d||||}|s|f|d d}|dur|f|S|St |||j|j|j|jd S) a encoder_hidden_states (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in ``[0, 1]``: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels n ``[0, ..., config.vocab_size]`` past_key_values (:obj:`tuple(tuple(torch.FloatTensor))` of length :obj:`config.n_layers` with each tuple having 4 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If :obj:`past_key_values` are used, the user can optionally input only the last :obj:`decoder_input_ids` (those that don't have their past key value states given to this model) of shape :obj:`(batch_size, 1)` instead of all :obj:`decoder_input_ids` of shape :obj:`(batch_size, sequence_length)`. use_cache (:obj:`bool`, `optional`): If set to :obj:`True`, :obj:`past_key_values` key value states are returned and can be used to speed up decoding (see :obj:`past_key_values`). Returns: Example: >>> from transformers import BertTokenizer, BertLMHeadModel, BertConfig >>> import torch >>> tokenizer = BertTokenizer.from_pretrained('bert-base-cased') >>> config = BertConfig.from_pretrained("bert-base-cased") >>> model = BertLMHeadModel.from_pretrained('bert-base-cased', config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits NF)rrrurrrrrMrVrrWrXrrYrrwr=) reductionrr<)rlogitsrMrrNrO)r_rrrrr rr}rsumr[F log_softmaxrrMrrNrO)rrrrrurrrrrrMrVrrWrXrrrY soft_labelsalpha return_logitsrrurvlm_lossshifted_prediction_scoresr loss_distillrr.r.r1rstH   zBertLMHeadModel.forwardcKsV|j}|dur ||}|dur|ddddf}||||dd|ddddS)NrwrrT)rrrMrrr)rXnew_onesget)rrpastr model_kwargsrr.r.r1prepare_inputs_for_generations   z-BertLMHeadModel.prepare_inputs_for_generationcs.d}|D]}|tfdd|Df7}q|S)Nr.c3s|] }|dVqdS)rN) index_select)r/ past_statebeam_idxr.r1r27s  z1BertLMHeadModel._reorder_cache..)rU)rrrreordered_past layer_pastr.rr1_reorder_cache4s zBertLMHeadModel._reorder_cache)NNNNNNNNNNNNNNTrrCNrF)NN) rrr"_keys_to_ignore_on_load_unexpectedrr{rrrrrrr.r.rr1r~sB   rcr)GisBertLMPredictionHeadcsHtt||_tj|j|dd|_tt ||_ |j |j_ dSrm) rzr{rjrnrrr~rorpr[rr8)rr_r}rr.r1r{?s  z GisBertLMPredictionHead.__init__cCr#rrqr$r.r.r1rLr%zGisBertLMPredictionHead.forwardrr.r.rr1r=s  rcsreZdZdgZddgZfddZ                        dd d Z dd d ZZS)BertForGisMaskedLMrrurcst|t|dd|_t||j|_t||j|_t||j |_ t||j |_ t||j |_ t||j|_t||j|_t||j|_t||j|_t||j|_t||j|_||_|dSr)rzr{rrrr} cls_geom_idr cls_geom_typer cls_rel_typercls_absolute_position_x1cls_absolute_position_x2rcls_absolute_position_y1cls_absolute_position_y2rcls_relative_position_x1cls_relative_position_x2rcls_relative_position_y1cls_relative_position_y2r_rrrr.r1r{YsF  zBertForGisMaskedLM.__init__NFrCrc' CsT| dur| n|jj} |j|fid|d|d|d|d|d|d|d | d | d | d | d |d|d|d|d|}|d}||}t}||d|jj| d}|j|j|j |j |j |j |j |j|j|jg }|||dddddf|dddddf|dddddf|dddddf|dddddf|dddddf|dddddf|dddddfg } |jj|jj|jj|jj|jj|jj|jj|jj|jj|jjg }!t|| |!D]\}"}#}$|#dur|"|}%|||%d|$|#d7}q| s |f|dd}&|dur|f|&S|&St|||j|jdS)a labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Labels for computing the masked language modeling loss. Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`` NrrrurrrrrrrWrXrrYrrrrrwr<r=rrrrrN)r_rrrr rr}rrrrrrrrrrrrrrrrrNrrrN)'rrrrrurrrrrrrrWrXrrYrrrrrrtoken_type_ids_labelrel_type_ids_labelabsolute_position_ids_labelrelative_position_ids_labelrrurvrr positions_clspositions_labelpositions_sizemyclsmylabelsmysizemyprediction_scoresrr.r.r1rus&          zBertForGisMaskedLM.forwardcK|j}|d}|jjdusJd||jddf}tj||gdd}tj|df|jjtj|jd}tj||gdd}||dSNrz.The PAD token should be defined for generationr=rwrr)rr rXr_r new_zerosr[rfullrrrrrrreffective_batch_size padding_mask dummy_tokenr.r.r1rs  z0BertForGisMaskedLM.prepare_inputs_for_generation)NNNNNNNNNNNNNFrCNrFNNNNNNNr) rrrrrr{rrrr.r.rr1rRsB  zrcszeZdZdgZddgZfddZddZdd Z dddZ dddZ Z S)BertForMaskedLMrrurcrrrrrr.r1r{rzBertForMaskedLM.__init__cCrrrrr.r.r1rrz%BertForMaskedLM.get_output_embeddingscCrrrrr.r.r1rrz%BertForMaskedLM.set_output_embeddingsNFrCrcCsT| dur| n|jj} |j|fid|d|d|d|d|d|d|d | d | d | d | d |d|d|d|d|}|d}||}|rN|Sd}| duret}||d|jj| d}|durtjt j |dd|dd }|| dk }d||||}| s|f|dd}|dur|f|S|St |||j |jdS)a labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Labels for computing the masked language modeling loss. Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`` NrrrurrrrrrrWrXrrYrrrrrwrrr=r<r)r_rrrr rr}r[rrrrrrrN)rrrrrurrrrrrrrWrXrrYrrrrrrrrurvrrrrr.r.r1rs      zBertForMaskedLM.forwardcKr r rrr.r.r1rks  z-BertForMaskedLM.prepare_inputs_for_generation)NNNNNNNNNNNNNFrCNrFNNNr) rrrrrr{rrrrrr.r.rr1rs>  Wrc:eZdZfddZ          dddZZS)BertForNextSentencePredictioncrr)rzr{rrryrrrrr.r1r{rz&BertForNextSentencePrediction.__init__Nc  Ks| dur| n|jj} |j|||||||| | d } | d} || }d}|dur6t}||dd|d}| sL|f| dd}|durJ|f|S|St||| j| jdS)u labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`): Labels for computing the next sequenåce prediction (classification) loss. Input should be a sequence pair (see ``input_ids`` docstring). Indices should be in ``[0, 1]``: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Returns: Example: >>> from transformers import BertTokenizer, BertForNextSentencePrediction >>> import torch >>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') >>> model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased') >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors='pt') >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random Nrr=rwr<r) r_rrrr rrrrN)rrrrrurrrrrWrXkwargsrrhseq_relationship_scoresrrrr.r.r1rsD! z%BertForNextSentencePrediction.forward NNNNNNNNNNrr.r.rr1rs  rcr)BertForSequenceClassificationcsJt||j|_t||_t|j|_t |j |j|_ | dSr rzr{ num_labelsrrrrrrrr~r;rrrr.r1r{s   z&BertForSequenceClassification.__init__Nc  Cs| dur| n|jj} |j|||||||| | d } | d} || } || } d}|durP|jdkr@t}|| d|d}nt}|| d|j|d}| sf| f| dd}|durd|f|S|St || | j | j dS)a labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`): Labels for computing the sequence classification/regression loss. Indices should be in :obj:`[0, ..., config.num_labels - 1]`. If :obj:`config.num_labels == 1` a regression loss is computed (Mean-Square loss), If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr=rwr<r) r_rrrr;rr rr rrrN)rrrrrurrrrrWrXrrhrrrrr.r.r1rsB   z%BertForSequenceClassification.forwardrrr.r.rr1rs  rcr)BertForMultipleChoicecs@t|t||_t|j|_t|j d|_ | dS)Nr=) rzr{rrrrrrrr~r;rrrr.r1r{s   zBertForMultipleChoice.__init__Nc  Csn| dur| n|jj} |dur|jdn|jd} |dur%|d|dnd}|dur4|d|dnd}|durC|d|dnd}|durR|d|dnd}|dure|d|d|dnd}|j|||||||| | d } | d} || } || }|d| }d}|durt}|||}| s|f| dd}|dur|f|S|St ||| j | j dS)at labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`): Labels for computing the multiple choice classification loss. Indices should be in ``[0, ..., num_choices-1]`` where :obj:`num_choices` is the size of the second dimension of the input tensors. (See :obj:`input_ids` above) Nr=rwrrr<r) r_rrXrrrrr;r rrrN)rrrrrurrrrrWrX num_choicesrrhrreshaped_logitsrrrr.r.r1r$s      zBertForMultipleChoice.forwardrrr.r.rr1rs  rcs@eZdZdgZfddZ          dddZZS)BertForTokenClassificationrcsNt||j|_t|dd|_t|j|_t |j |j|_ | dSrrrrr.r1r{ss  z#BertForTokenClassification.__init__Nc  Cs | dur| n|jj} |j|||||||| | d } | d} || } || } d}|duret}|durX|ddk}| d|j}t ||dt |j  |}|||}n || d|j|d}| s{| f| dd}|dury|f|S|St || | j| jdS)z labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Labels for computing the token classification loss. Indices should be in ``[0, ..., config.num_labels - 1]``. Nrrrwr=r<r)r_rrrr;r rrr[wheretensor ignore_indextype_asrrrN)rrrrrurrrrrWrXrrurrr active_loss active_logits active_labelsrr.r.r1r}sL    z"BertForTokenClassification.forwardrrrrrr{rrr.r.rr1r!os  r!csBeZdZdgZfddZ           dddZZS)BertForQuestionAnsweringrcs@t||j|_t|dd|_t|j|j|_| dSr) rzr{rrrrrr~ qa_outputsrrrr.r1r{s  z!BertForQuestionAnswering.__init__Nc  Cs@| dur| n|jj} |j||||||| | | d } | d} || }|jddd\}}|d}|d}d}|dur~|dur~t|dkrK|d}t|dkrX|d}|d}|d||d|t |d}|||}|||}||d}| s||f| dd}|dur|f|S|St |||| j | j d S) aa start_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (:obj:`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (:obj:`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrr=rwr)r$r<)r start_logits end_logitsrrN) r_rrr+rOsqueezerUrclamp_r rrrN)rrrrrurrstart_positions end_positionsrrWrXrrurr,r-r ignored_indexr start_lossend_lossrr.r.r1rsZ            z BertForQuestionAnswering.forwardrr)r.r.rr1r*s  r*csTeZdZdZeZdZdZdgZfddZ ddZ dd d Z e fd d Z ZS)MGeoPreTrainedModelrrTruc s*tj|jfi|tt||dSr)rzr{ name_or_pathr!)rr_rrr.r1r{( szMGeoPreTrainedModel.__init__cCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsrrNr)rrrr5r]rr_rr8rr|rqrrrr.r.r1r, s$      z!MGeoPreTrainedModel._init_weightsFcCst|tr ||_dSdSr)rr:rF)rrHrr.r.r1_set_gradient_checkpointing> s  z/MGeoPreTrainedModel._set_gradient_checkpointingc sn|dd}|dd}t||fi|}|dur%tdi|}||}n tt|jdd|i|}||_|S)aInstantiate the model. Args: kwargs: Input args. model_dir: The model dir used to load the checkpoint and the label information. num_labels: An optional arg to tell the model how many classes to initialize. Method will call utils.parse_label_mapping if num_labels not supplied. If num_labels is not found, the model will use the default setting (2 classes). Returns: The loaded model, which is initialized by transformers.PreTrainedModel.from_pretrained model_dirNcfgpretrained_model_name_or_pathr.)popr&rrzr!from_pretrainedr8)rrr8r9 model_argsr_r^rr.r1 _instantiateB s   z MGeoPreTrainedModel._instantiater)rrrrrrrsupports_gradient_checkpointingrr{rr7 classmethodr>rr.r.rr1r5 s  r5) module_namecsreZdZ   ddededeffdd Z   dd d Zd d ZddZ Z S)MGeo single-modalr=Fr_ finetune_modegis_numc st|||_||_t||d|_|jdkrBt|j}t|dd|_|j D]}d|_ q*t |j |jj |_t ||j |_|dS)Nr multi-modalF)rzr{rDr_r text_encoderr from_dict gis_encoder parameters requires_gradrrr~gis2textr|gis_typer)rr_rDrErr gis_configparamrr.r1r{e s    z MGeo.__init__Nc!sXjdkr.|dur.t|dkr.g}g}|D]}|jdddd|j||dq|r:jj|||d}njj|d}jdkr|durt|dkr|g}|g}fd d |D}t|||D]\}}}|||||qet j |d d }t j |d d }n|}|}j||| | |dd} |s| St | j| j | j | j| j| jdS)NrFrTrD)rXrYr)rrur)rcrr.)rM)r/gtprr.r1r= r>z MGeo.forward..r=rrw)rrrrWrXrYrr.)rDrUrMrIrLrGrrNrLr[rr$rrMrrNrOr)!rrrrrurrrrrrMrVrrWrXrrYgis_listgis_tpuse_token_typegis_embsgis_attsgisrembsattstp_embgegagt merge_embmerge_attentionrr.rr1r~ sr z MGeo.forwardcC|dS)NrLr.rrr.r.r1extract_sequence_outputs rzMGeo.extract_sequence_outputscCr_)Nrr.r`r.r.r1extract_pooled_outputs rzMGeo.extract_pooled_outputs)rCr=F)NNNNNNNNNNNNNNFrCNNF) rrrrr!rVr{rrarbrr.r.rr1rBb sB IrB)frrErandomwarnings dataclassesrtypingrrr[torch.nn.functionalr functionalrtorch.utils.checkpoint transformersrrrtorch.nnr r transformers.activationsr transformers.file_utilsr r rrrtransformers.modeling_outputsrrrrrrrrrtransformers.modeling_utilsrrrr+transformers.models.bert.configuration_bertrtransformers.utilsrmodelscope.metainfor modelscope.modelsr!r"modelscope.models.builderr#modelscope.outputsr$modelscope.utils.constantr%modelscope.utils.nlp.utilsr&set_verbosity_error get_loggerrC_CONFIG_FOR_DOC_TOKENIZER_FOR_DOCrnModulerorrrrrr&r'r:rcrjrlrrryr}r~rrrrrrrrrrr!r*r5register_modulebackbonemgeorBr.r.r.r1s  ,        JmF4[u   ' `@0PIVO_E