o ߥi@s<dZddlZddlZddlmZddlZddlmmZ ddl Zddl Z ddlm Z m Z mZddlmZddl mZmZddlmZddlmZmZmZmZdd lmZmZmZdd lmZmZm Z m!Z!dd l"m#Z#dd l$m%Z%m&Z&dd l'm(Z(m)Z)ddl*m+Z+ddl,m-Z-e j#.e#/Z0dZ1dZ2dZ3ddZ4ddZ5Gdddej6Z7Gdddej6Z8Gdddej6Z9Gdddej6Z:Gdd d ej6Z;Gd!d"d"ej6ZGd'd(d(ej6Z?Gd)d*d*ej6Z@Gd+d,d,ej6ZAGd-d.d.ej6ZBGd/d0d0ej6ZCGd1d2d2ej6ZDGd3d4d4ej6ZEGd5d6d6ej6ZFGd7d8d8eZGd9ZHd:ZIed;eHGdd?d?eGZKed@eHGdAdBdBeGZLGdCdDdDeGZMGdEdFdFeZNGdGdHdHeNZOGdIdJdJeNZPGdKdLdLeNZQGdMdNdNeZRGdOdPdPeRZSGdQdRdReRZTdS)SzPyTorch MPLUG model. N)Tuple)Tensordevicenn)CrossEntropyLoss) BertConfig BertTokenizer)ACT2FN)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardreplace_return_docstrings))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModelapply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)logging) HiTeAConfig MPlugConfig)MViTv2MViTv2_Base_config) TextGenerator) ModelFilez config.yamlrrc Cszddl}ddl}ddl}Wn tytdwtj|}t d ||j |}g}g} |D]\} } t d | | |j || } || | | q6t|| D]\} } | d} tdd| Drxt d d| qZ|} | D]|}|d |r|d |}n|g}|dd ks|dd krt| d} nH|ddks|ddkrt| d} n6|ddkrt| d} n*|ddkrt| d} nz t| |d} Wntyt d d| Yq|wt|dkrt|d}| |} q||dddkrt| d} n |d kr|| } z| j| jks'Jd| jd| jdWntyA}z |j| j| jf7_d}~wwt d | t| | _qZ|S)z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z(Converting TensorFlow checkpoint from {}z"Loading TF weight {} with shape {}/css|]}|dvVqdS))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0nr"r"f/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/multi_modal/mplug/modeling_mplug.py Xsz*load_tf_weights_in_bert..z Skipping {}z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifieri _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight {})renumpy tensorflow ImportErrorloggererrorospathabspathinfoformattrainlist_variables load_variableappendzipsplitanyjoin fullmatchgetattrAttributeErrorlenint transposeshapeAssertionErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathr3nptftf_path init_varsnamesarraysnamerLarraypointerm_name scope_namesnumer"r"r%load_tf_weights_in_bert;s                     rbcCsB|jtjkrt|rt|jjd}tj|| |d}|S)Ni)minmax)dtyperOfloat16isinfrDfinfordclamp)tensor clamp_valuer"r"r% clamp_infsrlcs4eZdZdZfddZ     dddZZS) BertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|dt|jdt|dd|_||_dS)N) padding_idxeps position_ids)r1position_embedding_typeabsolute)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferrOarangeexpandrGrsrSselfrS __class__r"r%rvs2  zBertEmbeddings.__init__Nrc Cs|dur |}n|dd}|d}|dur&|jdd|||f}|dur5tj|tj|jjd}|dur>||}||}||} |jdkrU| |} | | 7} | | } | | } | S)Nrrr1rerrt) sizerqrOzeroslongrr{rrsr}rr) r input_idstoken_type_idsrq inputs_embedspast_key_values_length input_shape seq_lengthr embeddingsr}r"r"r%forwards0         zBertEmbeddings.forward)NNNNr)__name__ __module__ __qualname____doc__rvr __classcell__r"r"rr%rms rmcsZeZdZfddZddZddZddZd d Zd d Z dddZ Z S)BertSelfAttentioncs"t||_|j|jdkrt|dstd|j|jf|j|_t|j|j|_|j|j|_ t |j|j |_ |rQt |j |j |_t |j |j |_nt |j|j |_t |j|j |_t |j|_t|dd|_|jdks{|jdkr|j|_t d|jd |j|_d |_dS) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)rsrt relative_keyrelative_key_queryr0r1F)rurvrSrynum_attention_headshasattr ValueErrorrJattention_head_size all_head_sizerLinearquery encoder_widthkeyvaluerattention_probs_dropout_probrrGrsr|rwdistance_embeddingsave_attentionrrSis_cross_attentionrr"r%rvsB    zBertSelfAttention.__init__cC ||_dSNattn_gradients)rrr"r"r%save_attn_gradients z%BertSelfAttention.save_attn_gradientscC|jSrrrr"r"r%get_attn_gradientsz$BertSelfAttention.get_attn_gradientscCrr attention_map)rrr"r"r%save_attention_maprz$BertSelfAttention.save_attention_mapcCrrrrr"r"r%get_attention_maprz#BertSelfAttention.get_attention_mapcCs6|dd|j|jf}|j|}|ddddS)Nrrrr0r1)rrrviewpermute)rx new_x_shaper"r"r%transpose_for_scoress  z&BertSelfAttention.transpose_for_scoresNFcCs||}|du} | r|||} |||} |}n;|durI|||} |||} tj|d| gdd} tj|d| gdd} n|||} |||} ||} | | f}t| | dd} t| } |j dksz|j dkr| d}tj |tj |j d dd}tj |tj |j d dd}||}|||jd}|j| jd }|j dkrtd | |}| |} n|j dkrtd | |}td | |}| ||} | t|j} |dur| |} tjdd| }| r|jr||||j||}|dur||}t|| }|dddd }| dd|j f}|j|}|r;||fn|f}||f}|S)Nrr0dimr1rrrrrrezbhld,lrd->bhlrzbhrd,lrd->bhlrr)!rrrrrOcatmatmulrKrlrsrrrrrrr|toreeinsummathsqrtrrSoftmaxrr register_hookrrr contiguousr)r hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsmixed_query_layerr key_layer value_layer query_layerattention_scoresrposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probsattention_probs_dropped context_layernew_context_layer_shapeoutputsr"r"r%rs                zBertSelfAttention.forwardNNNNNF) rrrrvrrrrrrrr"r"rr%rs " rc$eZdZfddZddZZS)BertSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNro) rurvrrrydenserrrrrrrr"r%rvj zBertSelfOutput.__init__cCs&||}||}|||}|Sr)rrrrr input_tensorr"r"r%rqs  zBertSelfOutput.forwardrrrrvrrr"r"rr%rh rcs<eZdZd fdd ZddZ      d ddZZS) BertAttentionFcs,tt|||_t||_t|_dSr)rurvrrroutputset pruned_headsrrr"r%rvzs    zBertAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrr1r)rIrrrrrrrrrrrrunion)rheadsindexr"r"r% prune_headss   zBertAttention.prune_headsNc Cs<||||||||}||d|} | f|dd} | S)Nrr1)rr) rrrrrrrr self_outputsattention_outputrr"r"r%rs   zBertAttention.forward)Fr)rrrrvrrrr"r"rr%rxsrcr)BertIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) rurvrrryintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrrr"r%rvs   zBertIntermediate.__init__cC||}||}|Sr)rrrrr"r"r%r  zBertIntermediate.forwardrr"r"rr%rs rcr) BertOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rurvrrrryrrrrrrrrr"r%rvrzBertOutput.__init__cCs6||}t|}||}t|}|||}|Sr)rrlrrrr"r"r%rs  zBertOutput.forwardrr"r"rr%rrrcs<eZdZfddZ       d ddZddZZS) FusionLayercs^t||_t|jdd|_|j|_d|_t||_t|dd|_ t ||_ t ||_ dS)N stride_layerdr1Tr)rurvrSrGrchunk_size_feed_forward seq_len_dimr attentioncrossattentionr intermediaterrrrS layer_numrr"r%rvs   zFusionLayer.__init__NFc Cs2|dur |ddnd} |dks||jdkrO|j||||| d} | d} | dd} | d} |dus7Jd|j| |||||d}|d} | |dd} n/|dkr~||jdkr~|jt||gdt||gd||| d} | d} | dd} | d} t|j|j|j| }|f| } | | d| df} | S) Nr0rrrr1rr>encoder_hidden_states must be given for cross-attention layersrr) rr rrOrrfeed_forward_chunkr r )rrrrrr layer_numsrrself_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attention_outputs layer_outputr"r"r%rsb    zFusionLayer.forwardcC||}|||}|Srrrrrintermediate_outputrr"r"r%r  zFusionLayer.feed_forward_chunk)NNNNNNFrrrrvrrrr"r"rr%rs  =rcs:eZdZfddZ      d ddZddZZS) BertLayercsdt||_|j|_d|_t||_t|jdd|_|jr&t|dd|_ t ||_ t ||_ dS)Nr1add_cross_attentionFTr )rurvrSr r rr rGhas_cross_attentionrrrrrrrr"r%rv!s   zBertLayer.__init__NFc Cs*|dur |ddnd}|j|||||d} | d} | dd} | d} |jrz|dus/Jdt|tkrc|j| ||||j|jjt|||j|jjt||d} | d} | | dd} n|j| |||||d} | d} | | dd} t |j |j |j | }|f| } | | d| df} | S)Nr0rrr1rrrr) r r$typelistrrrS fusion_layerrIrrr r )rrrrrrrrrrrrrrrr"r"r%r0sv     zBertLayer.forwardcCrrrrr"r"r%rsr zBertLayer.feed_forward_chunkrr!r"r"rr%r"s  Cr"c8eZdZfddZ         dddZZS) FusionEncodercsHt|_tfddtjD|_tdjj |_ dS)Ncg|]}t|qSr")rr#irSr"r% z*FusionEncoder.__init__..r) rurvrSr ModuleListrangenum_hidden_layerslayerrd fusion_layers start_layerrrr-r%rv{s   zFusionEncoder.__init__NFTc  s| rdnd} r dnd} |rdnd} t|jdd|_|jd}|jd}t|jt|jD]}|j|}| r;| |f} |durC||nd}|durM||ndt|jddr||jr||rbt dd}fdd }t j j |||||||||j}n||||||||j}|d }|r| |d f7} r| |df} |jd||krt |||fd\}}||7}q-| r| |f} ||gS) Nr"rr r1gradient_checkpointingFh`use_cache=True` is incompatible with `config.gradient_checkpointing=True`. Setting `use_cache=False`...cfdd}|S)Nctg|RSrtupleinputsmodulerrr"r%custom_forwardzLFusionEncoder.forward..create_custom_forward..custom_forwardr"r?r@rr?r%create_custom_forwardz4FusionEncoder.forward..create_custom_forwardrrr)rGrSrrLr1r5rIr3trainingr7warningrOutils checkpointrC)rrrrrrpast_key_values use_cacheroutput_hidden_states return_dictall_hidden_statesall_self_attentionsnext_decoder_cache image_length text_lengthr, layer_modulelayer_head_maskrD layer_outputsencoder_hidden_states_newr"rr%rs~          zFusionEncoder.forward NNNNNNFFTrr"r"rr%r)ys  r)cr() BertEncodercs4t|_tfddtjD|_dS)Ncr*r")r"r+r-r"r%r.r/z(BertEncoder.__init__..)rurvrSrr0r1r2r3rrr-r%rvs  zBertEncoder.__init__NFTc  s^| rdnd} r dnd} r|jjrdnd} |rdnd}tt|jD]k}|j|}| r1| |f} |dur9||nd}|durC||ndt|jddrn|jrn|rXtdd}fdd}t j j |||||||}n ||||||}|d}|r||df7}r| |d f} q#| r| |f} | st d d ||| | | fDSt ||| | | d S) Nr"r6Fr7cr8)Ncr9rr:r<r>r"r%r@rAzJBertEncoder.forward..create_custom_forward..custom_forwardr"rBrrCr%rDrEz2BertEncoder.forward..create_custom_forwardrrrr1css|] }|dur|VqdSrr")r#vr"r"r%r&!s z&BertEncoder.forward..)last_hidden_staterJr attentionscross_attentions)rSr#r1rIr3rGrFr7rGrOrHrIr;r)rrrrrrrJrKrrLrMrNrOall_cross_attentionsrPr,rSrTrDrUr"rr%rs       zBertEncoder.forwardrWrr"r"rr%rXs rXcr) BertPoolercs*tt|j|j|_t|_dSr)rurvrrryrTanh activationrrr"r%rv3s zBertPooler.__init__cCs(|dddf}||}||}|S)Nr)rr`)rrfirst_token_tensor pooled_outputr"r"r%r8s  zBertPooler.forwardrr"r"rr%r^1 r^cr)BertPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dSr) rurvrrryrrrrr transform_act_fnrrrrr"r%rvCs   z$BertPredictionHeadTransform.__init__cCs"||}||}||}|Sr)rrerrr"r"r%rMs   z#BertPredictionHeadTransform.forwardrr"r"rr%rdAs  rdcr)BertLMPredictionHeadcsLtt||_tj|j|jdd|_t t |j|_ |j |j_ dS)NF)r,) rurvrd transformrrryrxdecoder ParameterrOrr,rrr"r%rvVs   zBertLMPredictionHead.__init__cCrr)rgrhrr"r"r%rdrzBertLMPredictionHead.forwardrr"r"rr%rfTs rfcr)BertOnlyMLMHeadcstt||_dSr)rurvrf predictionsrrr"r%rvls zBertOnlyMLMHead.__init__cC||}|Sr)rk)rsequence_outputprediction_scoresr"r"r%rp zBertOnlyMLMHead.forwardrr"r"rr%rjj rjcr)BertOnlyNSPHeadcstt|jd|_dSNr0)rurvrrryseq_relationshiprrr"r%rvws zBertOnlyNSPHead.__init__cCrlr)rs)rrbseq_relationship_scorer"r"r%r{rozBertOnlyNSPHead.forwardrr"r"rr%rqurprqcr)BertPreTrainingHeadscs(tt||_t|jd|_dSrr)rurvrfrkrrryrsrrr"r%rvs  zBertPreTrainingHeads.__init__cCs||}||}||fSr)rkrs)rrmrbrnrtr"r"r%rs  zBertPreTrainingHeads.forwardrr"r"rr%rurcruc@s*eZdZdZeZeZdZdgZ ddZ dS)BertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. bertrqcCs~t|tjtjfr|jjjd|jjdnt|tj r(|j j |jj dt|tjr;|j dur=|j j dSdSdS)z Initialize the weights g)meanstd?N) rrrrwr)rQnormal_rSinitializer_rangerr,zero_fill_)rr?r"r"r% _init_weightss  z!BertPreTrainedModel._init_weightsN) rrrrr config_classrbload_tf_weightsbase_model_prefix_keys_to_ignore_on_load_missingrr"r"r"r%rvs rva This model inherits from :class:`~transformers.PreTrainedModel`. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch `torch.nn.Module `__ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (:class:`~transformers.BertConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. a Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`~transformers.BertTokenizer`. See :meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for details. `What are input IDs? <../glossary.html#input-ids>`__ attention_mask (:obj:`torch.FloatTensor` of shape :obj:`({0})`, `optional`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. `What are attention masks? <../glossary.html#attention-mask>`__ token_type_ids (:obj:`torch.LongTensor` of shape :obj:`({0})`, `optional`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0, 1]``: - 0 corresponds to a `sentence A` token, - 1 corresponds to a `sentence B` token. `What are token type IDs? <../glossary.html#token-type-ids>`_ position_ids (:obj:`torch.LongTensor` of shape :obj:`({0})`, `optional`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0, config.max_position_embeddings - 1]``. `What are position IDs? <../glossary.html#position-ids>`_ head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`({0}, hidden_size)`, `optional`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert :obj:`input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (:obj:`bool`, `optional`): Whether or not to return the attentions tensors of all attention layers. See ``attentions`` under returned tensors for more detail. output_hidden_states (:obj:`bool`, `optional`): Whether or not to return the hidden states of all layers. See ``hidden_states`` under returned tensors for more detail. return_dict (:obj:`bool`, `optional`): Whether or not to return a :class:`~transformers.file_utils.ModelOutput` instead of a plain tuple. z^The bare Bert Model transformer outputting raw hidden-states without any specific head on top.c eZdZdZdfdd ZddZddZd d Zee d e e d e ed dedeedededef ddZ               dddZZS) BertModel= The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in `Attention is all you need `__ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. argument and :obj:`add_cross_attention` set to :obj:`True`; an :obj:`encoder_hidden_states` is then expected as an input to the forward pass. TcsDt|||_t||_t||_|rt|nd|_| dSr) rurvrSrmrrXencoderr^pooler init_weightsrrSadd_pooling_layerrr"r%rvs    zBertModel.__init__cC|jjSrrr{rr"r"r%get_input_embeddingszBertModel.get_input_embeddingscC ||j_dSrrrrr"r"r%set_input_embeddings zBertModel.set_input_embeddingscC*|D]\}}|jj|j|qdSz Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel Nitemsrr3r rrheads_to_pruner3rr"r"r% _prune_headszBertModel._prune_headsbatch_size, sequence_lengthbert-base-uncasedprocessor_classrI output_typerrrr is_decoderreturnc CX|dkr|dddddddf}n|dkr|r|\}}tj||d}|ddddf||d|ddddfk} | |j} | jd|jdkrl|jd| jd} tjtj||| f|| jd| gdd} | dddddddf|ddddddf}n|ddddddf}n t d ||j|j|jd }d |d }|S aW Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (:obj:`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (:obj:`Tuple[int]`): The shape of the input to the model. device: (:obj:`torch.device`): The device of the input to the model. Returns: :obj:`torch.Tensor` The extended attention mask, with a the same dtype as :obj:`attention_mask.dtype`. rNr0rr1)rrerr)axiszAWrong shape for input_ids (shape {}) or attention_mask (shape {})rrzg rrOrrepeatrrerLronesrr= rrrrrextended_attention_mask batch_sizerseq_ids causal_maskprefix_seq_lenr"r"r%get_extended_attention_masks\       z%BertModel.get_extended_attention_maskNFc s| dur| njj} | dur| njj} |dur|njj}|r+| dur&| njj} nd} |dur9|dur9td|durI|}|\}}|j}n,|dur]|dd}|\}}|j}n|durq|dd}|\}}|j}ntd| dur| ddjdnd}|durt j |||f|d}|durt j |t j |d } ||||}|durt|tkr|d\}}}n|\}}}||f}t| tkrׇfd d | D}n| durt j ||d} | }n| }nd}|jj}|durj|||||d }n|}j|||||| | | | |d }|d}jdur(|nd}|s7||f|ddSt|||j|j|j|jdS)~ encoder_hidden_states (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in ``[0, 1]``: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (:obj:`tuple(tuple(torch.FloatTensor))` of length :obj:`config.n_layers` with each tuple having 4 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If :obj:`past_key_values` are used, the user can optionally input only the last :obj:`decoder_input_ids` (those that don't have their past key value states given to this model) of shape :obj:`(batch_size, 1)` instead of all :obj:`decoder_input_ids` of shape :obj:`(batch_size, sequence_length)`. use_cache (:obj:`bool`, `optional`): If set to :obj:`True`, :obj:`past_key_values` key value states are returned and can be used to speed up decoding (see :obj:`past_key_values`). NFDYou cannot specify both input_ids and inputs_embeds at the same timerrGYou have to specify either input_ids or inputs_embeds or encoder_embedsrr0rrcg|]}|qSr"invert_attention_maskr#maskrr"r%r.z%BertModel.forward..rrqrrr rrrrrJrKrrLrMr1rZ pooler_outputrJrr[r\rSrrLuse_return_dictrKrrrrLrOrrrrr%r&r get_head_maskr2rrrrrJrr[r\rrrrrqrrencoder_embedsrrrJrKrrLrMrrrrrrrencoder_batch_sizeencoder_sequence_length_encoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputsrmrbr"rr%rTs(         zBertModel.forwardTNNNNNNNNNNNNNNFrrrrrvrrrr BERT_INPUTS_DOCSTRINGr=r _TOKENIZER_FOR_DOCr_CONFIG_FOR_DOCrrrJrboolrrrr"r"rr%rsP  Hrc r) FusionModelrTcs:t|||_t||_|rt|nd|_|dSr)rurvrSr)rr^rrrrr"r%rvs   zFusionModel.__init__cCrrrrr"r"r%r rz FusionModel.get_input_embeddingscCrrrrr"r"r%r rz FusionModel.set_input_embeddingscCrrrrr"r"r%rrzFusionModel._prune_headsrrrrrrrrc Crrrrr"r"r%rs\       z'FusionModel.get_extended_attention_maskNFc s~| dur| njj} | dur| njj} |dur|njj}|r+| dur&| njj} nd} |dur9|dur9td|durI|}|\}}|j}n,|dur]|dd}|\}}|j}n|durq|dd}|\}}|j}ntd| dur| ddjdnd}|durt j |||f|d}|durt j |t j |d } ||||}|durt|tkr|d\}}}n|\}}}||f}t| tkrׇfd d | D}n| durt j ||d} | }n| }nd}|jj}|durj|||||d }n|}j|||||| | | | |d }|\}}jdur(|nd}|s1||gSt|||j|j|j|jdS)rNFrrrrrr0rrcrr"rrrr"r%r.rz'FusionModel.forward..rrrrrr"rr%rgs&         zFusionModel.forwardrrrr"r"rr%rsP GrzGBert Model with a `language modeling` head on top for CLM fine-tuning. cseZdZdgZddgZfddZddZdd Zee d e e e d    dddZ dddZddZZS)BertLMHeadModelrrqpredictions.decoder.biasc0t|t|dd|_t||_|dSNFrrurvrrwrjclsrrrr"r%rv   zBertLMHeadModel.__init__cC |jjjSrrrkrhrr"r"r%get_output_embeddingsrz%BertLMHeadModel.get_output_embeddingscC||jj_dSrrrnew_embeddingsr"r"r%set_output_embeddingsz%BertLMHeadModel.set_output_embeddingsr)rrNTrxrFcCs|dur|n|jj}| durd} |j||||||||| | | | ||d}|d}||}|r=|ddddddfSd}| dur}|ddddddf}| ddddf} t|d}||d|jj| d}||dd d}|durt j t j |dd|dd }|| d k d}d||||}|s|f|d d}|dur|f|S|St |||j|j|j|jd S) av encoder_hidden_states (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in ``[0, 1]``: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels n ``[0, ..., config.vocab_size]`` past_key_values (:obj:`tuple(tuple(torch.FloatTensor))` of length :obj:`config.n_layers` with each tuple having 4 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If :obj:`past_key_values` are used, the user can optionally input only the last :obj:`decoder_input_ids` (those that don't have their past key value states given to this model) of shape :obj:`(batch_size, 1)` instead of all :obj:`decoder_input_ids` of shape :obj:`(batch_size, sequence_length)`. use_cache (:obj:`bool`, `optional`): If set to :obj:`True`, :obj:`past_key_values` key value states are returned and can be used to speed up decoding (see :obj:`past_key_values`). Returns: Example: >>> from transformers import BertTokenizer, BertLMHeadModel, BertConfig >>> import torch >>> tokenizer = BertTokenizer.from_pretrained('bert-base-cased') >>> config = BertConfig.from_pretrained("bert-base-cased") >>> model = BertLMHeadModel.from_pretrained('bert-base-cased', config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits NF rrrqrrrrrJrKrrLrMrrrrr1) reductionrr0losslogitsrJrr[r\)rSrrwrrrrrxrsumrOF log_softmaxrrJrr[r\rrrrrqrrrrlabelsrJrKrrLrMrr soft_labelsalpha return_logitsrrmrnlm_lossshifted_prediction_scoresloss_fct loss_distillrr"r"r%r"sr@   zBertLMHeadModel.forwardcKsV|j}|dur ||}|dur|ddddf}||||dd|ddddS)NrrrrT)rrrJrrr)rLnew_onesget)rrpastr model_kwargsrr"r"r%prepare_inputs_for_generations   z-BertLMHeadModel.prepare_inputs_for_generationcs.d}|D]}|tfdd|Df7}q|S)Nr"c3s|] }|dVqdS)rN) index_select)r# past_statebeam_idxr"r%r&s  z1BertLMHeadModel._reorder_cache..r:)rrrreordered_past layer_pastr"rr%_reorder_caches zBertLMHeadModel._reorder_cacheNNNNNNNNNNNNNNTrxNrF)NN)rrr"_keys_to_ignore_on_load_unexpectedrrvrrr rr=r rrrrrrr"r"rr%r sL x rcseZdZdgZddgZfddZddZdd Zee d e e d e ed     dddZZS)BertPrefixModelrrqrcrrrrrr"r%rvrzBertPrefixModel.__init__cCrrrrr"r"r%rrz%BertPrefixModel.get_output_embeddingscCrrrrr"r"r%rrz%BertPrefixModel.set_output_embeddingsrrrNTrxrFcCsp|dur|n|jj}| durd} |j||||||||| | | | ||d}|d}||}|r=|ddddddfSd}| duro|ddddddf}| ddddf} t}||d|jj| d}|durtj t j |dd|dd }|| dk }d||||}|s|f|dd}|dur|f|S|St |||j|j|j|jd S) NFrrrrr1rrr0r)rSrrwrrrrrxrOrrrrxrrJrr[r\rr"r"r%rsp  zBertPrefixModel.forwardr)rrrrrrvrrr rr=r rrrrrr"r"rr%rsH rcseZdZeZfddZedddZeddd Z d d Z d d Z ddZ e ddZe ddZdddZeddZZS)MPlugcsft|||_ttj|jt j |_ | || ||_t|jdd|_t|jdd|_dSr)rurvrSrfrom_pretrainedr9r:rE model_dirr VOCAB_FILE tokenizermodule_setting_initialize_clipvisual_encoderrconfig_encoder text_encoderr config_fusionfusion_encoderrrr"r%rv3s    zMPlug.__init__NTc Csddlm}|jt|jt|jti}|j t j |t }||_|dur&|j}|||}|rgt j |tj}tj|dd} d| vrE| d} d| vrM| d} dd| D} |j| d d } td |t| |S) NrTaskscpu map_locationrRr?cSi|] \}}|dd|qSzmodel.replacer#krYr"r"r% V z)MPlug.from_pretrained..Fstrictzload checkpoint from %s)modelscope.utils.constantrvisual_question_answeringMPlugForVisualQuestionAnsweringimage_captioningMPlugForImageCaptionimage_text_retrievalMPlugForImageTextRetrievalrfrom_yaml_filer9r:rE CONFIG_NAMErtaskrTORCH_MODEL_BIN_FILErOloadrload_state_dictprint) rrr(load_checkpointr task_mappingrSrRcheckpoint_pathrImsgr"r"r%r?s8    zMPlug.from_pretrainedcCsdd}ddlm}||}d|jvr,t|j|jd}tt|dd }nt|j|jd}tt|dd  }||j j d | d |_ ||j _ |S) NcSs|jd}|ddddf|dddf}}|d8}ttt|}tt|}|d||ddddd}|j}tj | ||fdd}| |}|ddddd||d}t j ||gdd}|S) Nr1rrrrr0bilinear)rmoder)rLrJrrrIreshaperrer interpolatefloatrrOr)posemb posemb_newntok_new posemb_tok posemb_gridgs_oldgs_neworigr"r"r%resize_pos_embedcs( &    z0MPlug._initialize_clip..resize_pos_embedr1)clipzViT-B-16iir)r@load_from_config clip_namerJ image_resrrirOrr6visualr unsqueezer))rS num_patchesr?r@ clip_model pos_embedr"r"r%r `s     zMPlug._initialize_clipcCs|j|_|jrq|||_t|jdd|_t|jdd|_t |j |_ |j |jg|j |jg|j|j gg|_|jj|jkrht|j|jj|_tj|jjdd|_t|jj|_|j|j|jg|j|jgg|d|_dSdSNFr-q=roףp= ?)distillr visual_encoder_mrr text_encoder_mrr fusion_encoder_mrconfig_decodertext_decoder_mr r  text_decoder model_pairsry vision_widthrr visn_fc_mrvisn_layer_norm_mrr dropout_mextendvisn_fcvisn_layer_norm copy_paramsmomentumrr"r"r% init_distills@        zMPlug.init_distillcOtrNotImplementedErrorrrNkwargsr"r"r%rz MPlug.forwardcCstj|j|j}t||_|jj|j_ t||_ t||_ d|j _ |j j |j _ d|_|jj|jkrVt|j|jj|_tj|jjdd|_t|jj|_d|_dSdS)NTFrLro)r9r:rEr bert_configrfrom_json_filer text_encoder_layersr2r rRr#text_decode_layerslargeryrVrrr[rr\rrrrrSbert_config_pathr"r"r%rs$      zMPlug.module_settingcCH|jD]}t|d|dD]\}}|j|jd|_qqdSNrr1FrUrB parametersrQcopy_ requires_gradr model_pairparamparam_mr"r"r%r]    zMPlug.copy_paramscCR|jD]#}t|d|dD]\}}|j|j|jd|j|_qqdSNrr1rzrUrBrprQr^rsr"r"r%_momentum_update     zMPlug._momentum_updater1cC$||g}|jj||d\}}||fSNout_sizebeam_generatortranslate_batchrquestion_states question_attsrencoder_inputstopk_ids topk_scoresr"r"r% generation  zMPlug.generationcnddl||dg|}||<|j|}tfddtD}t||| |j S)Nrr1cg|] }|qSr"rr+init_dimn_tilerUr"r%r.zMPlug._tile.. r4rrrrO LongTensor concatenater1rrrrrr repeat_idx order_indexr"rr%_tile  z MPlug._tileNT)r1r1)rrrrrrv classmethodr staticmethodr r_rrrOno_gradr]r{rrrr"r"rr%r0s"   '   rc0eZdZfddZ     dddZZS) r!c4t|t|j|_t||j|_||dSrrurvrrRrTrrr_rrr"r%rv  z(MPlugForVisualQuestionAnswering.__init__NrTc! Cs.|jt|jd}|jj|dd}|jr!||| |}t j | ddt j d|j} |r^ |j|j|jjkd} |j|j|jdd} | j} |j| |j|| dd} | \}}t ||gd }t | |jgd }|dur|d g|jd }g}g}t|D]\}}|||g|7}|||g|7}qt |d }t |d }|jr9t l||jj|dd}|jr|| |!|}|j"|j|jdd}|j}|j#||j|| dd}|\}}t ||gd }g}t|D] \}}|||g|7}qt |d }|j$|j|j||dd }Wdn 1swY|j%|j|j||| dt&j'|dd d d}n|j%|j|j||| dd d}|durNd }||j(}|)| d }|S|j|j|jdd} | j} |j| |j|| dd} | \}}t ||gd }t | |jgd }|*||\}} || fS)NrTskip_last_layerrrrrrMFrrrrrMr1rrrrrrnonerrrrrMrrrrrrrMr)+rnextrprer rFrjrr\r[rOrrrrr masked_fillrrzr rrZrrrL enumeratestackrNrr{rOrYrXrWrPrQrSrTrsoftmaxrrr)!rimagequestionanswerrrweightsr> image_embeds image_attsanswer_targets text_output text_embeds fusion_output image_outputquestion_outputmerge_text_attentionrrbr$image_embeds_m text_output_m text_embeds_mfusion_output_mimage_output_mquestion_output_mquestion_states_mlogits_m answer_outputrr topk_probsr"r"r%rs     %     z'MPlugForVisualQuestionAnswering.forwardNrNNTrr"r"rr%r!  r!c>eZdZfddZ   d ddZ    d d d ZZS) r#c*t|t|j|_t||j|_dSrrurvrrRrTrrrrr"r%rvm  zMPlugForImageCaption.__init__NTcCs|jj|dd}|jr||||}tj|ddtj d |j }|j |j |jdd}|j} |j| |j||dd} | \} } t| | gd} t||jgd} |j| | |d \}}||fS) NTrrrrrFrr1r)r rFrjrr\r[rOrrrrrr rrrZrrr)rrrrr>rrrrrrrrrrrr"r"r% beam_searchrs@  z MPlugForImageCaption.beam_searchFc Cs|r |j|||d|dS|jt|jd}|jj|dd}|jr-|| | |}t j | ddt jd|j}|r`|j|j|jjkd} |j|j|j||| ddd} | j} | S|||\} } | | fS) NTr>rrrrrrrr)rrrrprer rFrjrr\r[rOrrrrrrrrzrTrrr)rrrrr>rscstrrrrrrrr"r"r%rs@ zMPlugForImageCaption.forwardNTrNTrFrrrrvrrrr"r"rr%r#ks  "r#cs>eZdZfddZddZeddZd d d ZZ S) r%cs&t||j|_ttg|j|_|j|_|j |_ |j |_ |j|_|j j |_ |j|_t|j |j|_t|j |j|_t|j d|_|dt|j|j|dt|j|j|dtd|jfd|dtjdtjdtj|jd d |_tj|jd d |_||dS) Nr0 image_queue text_queue idx_queuer1r queue_ptrrrr)rurv embed_dimrrirOrtemp queue_sizer^rr ry text_widthr vision_proj text_projitm_headrrandnfullrrr normalizerrr_rrr"r%rvs2  z#MPlugForImageTextRetrieval.__init__cCs|j|_|jr|||_t|jdd|_t|jdd|_t |j |j |_ t |j |j |_|j|jg|j|jg|j|jg|j|j gg|_|jj|jkryt |j|jj|_t j|jjdd|_t |jj|_|j|j|jg|j|jgg|d|_ dSdSrK)!rNr rOrr rPrr rQrrrr vision_proj_m text_proj_mr r rrrUryrVrWrrXrrrYrZr[r\r]r^rr"r"r%r_sD        z'MPlugForImageTextRetrieval.init_distillc Csdd}||}||}||}|jd}t|j} |j|jdd| | |f<|j|jdd| | |f<|j|jdd| | |f<| ||j} | |jd<dS)NcsNtjsSfddttjD}tjj|ddtj|dd}|S)z Performs all_gather operation on the provided tensors. *** Warning ***: torch.distributed.all_gather has no gradient. csg|]}tqSr")rO ones_like)r#rrjr"r%r.rz^MPlugForImageTextRetrieval._dequeue_and_enqueue..concat_all_gather..F)async_oprr)rO distributedis_initializedr1get_world_size all_gatherr)rjtensors_gatherrr"rr%concat_all_gathers   zJMPlugForImageTextRetrieval._dequeue_and_enqueue..concat_all_gatherr)rLrJrTrrrr) r image_feat text_featidxr image_feats text_featsidxsrptrr"r"r%_dequeue_and_enqueues  z/MPlugForImageTextRetrieval._dequeue_and_enqueueNTc6 Cs(|r|jj|dd}|jr||||}tj|ddtj d |j }t j ||dddddfdd}|j|j|jdd}|j} t j || dddddfdd} |dd}tj||jgdd} t|| } | | jddd } t||jj|dd}|jr| |!|"|}t j |#|dddddfdd}tj||j$gdd}|j%|j|jdd}t j |&|jdddddfdd}tj||j'gdd}|j(r)|||j)}|||j)}|j*t j+|ddd|j*| }|j*t j+|ddd|j*| }Wdn 1s4wY|||j)}| ||j)}|j(rntjt j,|dd|dd- }tjt j,|dd|dd- }n"tjt j,|dd| dd- }tjt j,|dd| dd- }||d }|.||||j/| |j||d d \}}t>|d}t j+|ddd|fdd} t j+|ddd|fdd}!t||j0}"| 1|"d|!1|"dWdn 1swYg}#t2|D]}$t3|!|$d4}%|#5||%qtj6|#dd}#g}&g}'t2|D]}$t3| |$d4}%|&5| |%|'5|j|%qtj6|&dd}&tj6|'dd}'tj| |&gdd}(tj|j|'gdd})tj|#|gdd}*tj||gdd}+|j/|(|)|*|+d d \}},tj|dddddf|,dddddfgdd}-|7|-}.tj|tj d}/tj8d |tj d}0tj|/|0gdd |j }1t 9|.|1}2||2S|j|j|jd }|j} |jj|dd}|||}tj|ddtj |j d}3|j/| |j||3d d \}}4|7|4dddddf}5t j+|5dd}5|5S)NTrrrrrrrr1)keepdimr0Fr)rr):r rFrjrr\r[rOrrrrrrrrr rrrZrrrtrclonedetacheqr6rrr{rOrYrXrWrrrPrrrNrrrrrxrrr masked_fill_r1 multinomialitemrArrr cross_entropy)6rrtextrr>rrrrrridx_allpos_idx sim_targetsr image_feat_mimage_feat_allr text_feat_m text_feat_all sim_i2t_m sim_t2i_msim_i2t_targetssim_t2i_targetssim_i2tsim_t2iloss_i2tloss_t2iloss_itar output_posbs weights_i2t weights_t2irimage_embeds_negrneg_idxtext_embeds_neg text_atts_negtext_embeds_all text_atts_allimage_embeds_allimage_atts_all output_neg vl_embeddings vl_outputones_tmp zeros_tmp itm_labelsloss_itm image_attrscoresr"r"r%r sr      !         ,    z"MPlugForImageTextRetrieval.forwardr) rrrrvr_rOrrrrr"r"rr%r%s   %r%cs|eZdZeZfddZedddZddZdd Z d d Z e d d Z e ddZdddZeddZZS)HiTeAcsnt|||_ttj|jt j |_ | |t |jt|jd|_t|jdd|_t|jdd|_dS)Nimg_sizerS num_framesFr)rurvrSrrr9r:rErrrrrrrErr*r rr r rr rrrr"r%rv s"   zHiTeA.__init__Tc Csddlm}|jt|jti}|jtj |t }||_ ||j |}|rTtj |tj}tj|dd}d|vr<|d}d|vrD|d}dd|D}|j|d d |S) NrrrrrRr?cSrrrrr"r"r%r rz)HiTeA.from_pretrained..Fr)rrvideo_question_answeringHiTeAForVideoQuestionAnsweringvideo_captioningHiTeAForVideoCaptionrr&r9r:rEr'rr(rr)rOr*rr+) rrr-rr.rSrRr/rIr"r"r%r s.  zHiTeA.from_pretrainedcCs|j|_|jrBt|jt|jd|_t|jdd|_t |j dd|_ t |j |_|j|jg|j|jg|j|jgg|_|d|_dSdS)Nr(FrrM)rNrrErr*rOrr rPrr rQrrRrSr r rTrUr]r^rr"r"r%r_ s*     zHiTeA.init_distillcOr`rrarcr"r"r%r rez HiTeA.forwardcCsZtj|j|j}t||_|jj|j_ t||_ t||_ d|j _ |j j |j _ dSr)r9r:rErrfrrgr rhr2r rRr#rirkr"r"r%r s    zHiTeA.module_settingcCrmrnrorsr"r"r%r] rwzHiTeA.copy_paramscCrxryrzrsr"r"r%r{ r|zHiTeA._momentum_updater1cCr}r~rrr"r"r%r rzHiTeA.generationcr)Nrr1crr"rr+rr"r%r.( rzHiTeA._tile..rrr"rr%r rz HiTeA._tilerr)rrrrrrvrrr_rrrOrr]r{rrrrr"r"rr%r' s     r'cr) r,crrrrrr"r%rv. rz'HiTeAForVideoQuestionAnswering.__init__NrTc! Cs|jt|jd}||}tj|ddtjd|j } |r; |j |j |j j kd} |j|j |jdd} | j} |j| |j|| dd} | \}}t||gd}t| |jgd}|durkdg|jd }g}g}t|D]\}}|||g|7}|||g|7}qst|d }t|d }|jrt[|||}|j|j |jdd}|j}|j||j|| dd}|\}}t||gd}g}t|D] \}}|||g|7}qt|d }|j|j |j||dd }Wdn1swY|j|j |j||| dtj|dd d d }n|j|j |j||| dd d}|dur+d}||j }|!|d }|S|j|j |jdd} | j} |j| |j|| dd} | \}}t||gd}t| |jgd}|"||\}} || fS)NrrrrTrFrr1rrrrrr)#rrrprer rOrrrrrrrrzr rrZrrrLrrrNrr{rOrPrQrSrTrrrrr)!rvideorrrrrr> video_embeds video_attsrrrr video_outputrrrrrr$video_embeds_mrrrrrrrrrrrr"r"r%r4 s             z&HiTeAForVideoQuestionAnswering.forwardrrr"r"rr%r,, rr,cr) r.crrrrrr"r%rv rzHiTeAForVideoCaption.__init__NTrcCs||}tj|ddtjd|j}|j|j|j dd}|j } |j | |j ||dd} | \} } t | | gd} t ||j gd} |j | | |d\}}||fS) NrrrTrFrr1r)r rOrrrrrr rrrZrrr)rr/rrr>rr0r1rrrr2rrrrr"r"r%r s8   z HiTeAForVideoCaption.beam_searchFc Cs|r |j|||d|dS|jt|jd}||}tj|ddtj d|j }|rO|j |j |j jkd} |j|j |j||| ddd} | j} | S|||\} } | | fS)NTrrrrrrr)rrrrprer rOrrrrrrrrzrTrrr)rr/rrr>rrr0r1rrrrrr"r"r%r s8  zHiTeAForVideoCaption.forwardrrrr"r"rr%r. s  r.)Urrr9typingrrOtorch.nn.functionalr functionalrtorch.utils.checkpoint transformersrrtorch.nnrrrtransformers.activationsr transformers.file_utilsr r r r transformers.modeling_outputsrrrtransformers.modeling_utilsrrrrtransformers.utilsr7modelscope.models.multi_modal.mplug.configuration_mplugrr(modelscope.models.multi_modal.mplug.mvitrr-modelscope.models.multi_modal.mplug.predictorrrrset_verbosity_error get_loggerr7r'rrrbrlModulermrrrrrrr"r)rXr^rdrfrjrqrurvBERT_START_DOCSTRINGrrrrrrr!r#r%r'r,r.r"r"r"r%s       JB4QZ\\   *7m+Kq