o ߥi@sdZddlZddlmZddlZddlZddlmZddlmZddl m Z ddl m Z m Z mZmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZddlmZeZeej edkZ!dZ"dZ#d3ddZ$e"e#fddZ%e"e#fddZ&d4ddZ'Gdddej(Z)Gdddej(Z*Gd d!d!ej(Z+Gd"d#d#ej(Z,Gd$d%d%ej(Z-Gd&d'd'ej(Z.Gd(d)d)ej(Z/Gd*d+d+ej(Z0Gd,d-d-ej(Z1Gd.d/d/ee Z2ej3ej4ej5d0Gd1d2d2e2Z6dS)5zPyTorch PoNet model. N) LooseVersion)version)nn)ACT2FN)PreTrainedModelapply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)Models)Model TorchModel)MODELS)AttentionBackboneModelOutput)Tasks) get_logger) PoNetConfigz1.12.0efc CsBtrt|j||dddddf||ddd}nd|dddddf|}|d}g|jdd|d|jdR}|dj |}tj |d|j ddddddfj |}|dj |} | ||kt | |jdd\}} |dj g|jdd |dR} t||| j|jd S) NamaxF)reduce include_selfrdevicedimdtype)is_pytorch_12plustorch zeros_likescatter_reduce expand_asminshapemax unsqueezeexpandarangermasked_scatter full_likeitemsizegathertor ) srcindexroutdummy_scatter_index min_valuedummpy_scatter_shapedummy_scatter_index_expandindex_reconstruct_expand src_expand_dummyr=X/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/ponet/backbone.py segment_max/sF      *r?cCsp||kjtjd||kjtjd}|tjt|ddddf|ddddfgdd}|jdddS)Nrrrrr)r1r"longcatr#cumsum input_idscls_ideos_idmaskr=r=r>get_segment_indexKs2rHcCs||k||kB}|SNr=rCr=r=r>get_token_type_maskSsrJcCs4tj|d|dd}||dddddd}|S)Nrr)stridepaddingr)r MaxPool1dpermute) hidden_states kernel_sizemr4r=r=r> get_win_maxXsrScs4eZdZdZfddZ     dddZZS) PoNetEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|dt|jdttjtdkri|jdtj|jtj|jjd d d dSdS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rrz1.6.0token_type_idsr rF) persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrrXregister_bufferr"r+r*rparse __version__zerosrZr/r@rselfconfig __class__r=r>r_asF  zPoNetEmbeddings.__init__Nrc Cs|dur |}n|dd}|d}|dur&|jdd|||f}|durPt|drE|jddd|f}||d|} | }n tj|tj|jjd}|durY| |}| |} || } |j dkrp| |} | | 7} | | } || } | S)Nrrr[rr\rY)r/rZhasattrr[r*r"rrr@rrdrhrXrfrirm) rtrDr[rZ inputs_embedspast_key_values_length input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedrh embeddingsrfr=r=r>forwards@         zPoNetEmbeddings.forward)NNNNr)__name__ __module__ __qualname____doc__r_r __classcell__r=r=rvr>rT^s "rTc:eZdZfddZddZ      d ddZZS) PoNetSelfAttentioncstt|j|j|_t|j|j|_|j|_t|dd|_ t |j|j|_ |j|j |_ t|j|j |_ t|j|j |_t|j|j |_dS)NclsgsepgT)r^r_rLinearrb dense_local dense_segmentnum_attention_headsrnrintattention_head_size all_head_sizedense_qdense_kdense_orsrvr=r>r_s zPoNetSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)NrrrrrK)r/rrviewrO)rtx new_x_shaper=r=r>transpose_for_scoress  z'PoNetSelfAttention.transpose_for_scoresNFc Cs&|||} |||} | } |||} |dur(|dddk}|durK| |d| jddt |j | j d |djdd}n| j dd}td|| t| jd}|durk||d}|jdd}td| |}||}||}|dur||dd ||dd |jrt| |jddd }t||d }||jddd||jdddn t|}t||d }||}||}|jdd|| |}|d d dd jg|jdd dR}|dur||dd| r||f}|S|f}|S)Nrrrrrzbdh,bdlh -> bdlz bdlh,bdl->bdhi)r3rrrK)rrrrsqueezer) masked_fill_sumr" ones_liker1r masked_fillmeaneinsummathsqrtr'softmaxrrrrSr?rOreshape)rtrP segment_indextoken_type_maskattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionscontext_layer_qcontext_layer_kcontext_layer_vcontext_layer_o_attention_maskqattatt_probvcontext_layer_segmentcontext_layer_local context_layeroutputsr=r=r>rs              zPoNetSelfAttention.forwardNNNNNF)rrrr_rrrr=r=rvr>rs  rc$eZdZfddZddZZS)PoNetSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNrV) r^r_rrrbdenserirjrkrlrmrsrvr=r>r_ zPoNetSelfOutput.__init__cC&||}||}|||}|SrIrrmrirtrP input_tensorr=r=r>r  zPoNetSelfOutput.forwardrrrr_rrr=r=rvr>r rcr)PoNetIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSrI) r^r_rrrbintermediate_sizer isinstance hidden_actstrrintermediate_act_fnrsrvr=r>r_'s   zPoNetIntermediate.__init__cCs||}||}|SrI)rr)rtrPr=r=r>r/s  zPoNetIntermediate.forwardrr=r=rvr>r%s rcr) PoNetOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r^r_rrrrbrrirjrkrlrmrsrvr=r>r_7rzPoNetOutput.__init__cCrrIrrr=r=r>r>rzPoNetOutput.forwardrr=r=rvr>r5rrcr) PoNetAttentioncs*tt||_t||_t|_dSrI)r^r_rrtroutputset pruned_headsrsrvr=r>r_Gs    zPoNetAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)lenrrtrrrr querykeyvaluerrrunion)rtheadsr3r=r=r> prune_headsMs   zPoNetAttention.prune_headsNFc Cs@|||||||||| } || d|} | f| dd} | S)Nrr)rtr) rtrPrrrrrrrr self_outputsattention_outputrr=r=r>r`s    zPoNetAttention.forwardr)rrrr_rrrr=r=rvr>rEs rcs:eZdZfddZ      d ddZddZZS) PoNetLayercstt|j|_d|_t||_d|_|j|_|j|_|jr.|js)J|dt||_t ||_ t ||_ dS)NrFz> should be used as a decoder model if cross attention is added) r^r_chunk_size_feed_forward seq_len_dimr attention is_decoderadd_cross_attentioncrossattentionr intermediaterrrsrvr=r>r_s    zPoNetLayer.__init__NFc  Cs|dur |ddnd} |j|||||| | d} | d} |jr*| dd} | d}n| dd} d}|jrq|durqt|dsFJd|d|durP|d dnd}|| |||||| }|d} | |dd} |d}||}t|j|j|j| }|f| } |jr| |f} | S) Nrrrrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r)rrrxrrfeed_forward_chunkrr)rtrPrrrrrrrrself_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputr=r=r>rsv       zPoNetLayer.forwardcCs||}|||}|SrI)rr)rtrintermediate_outputrr=r=r>rs  zPoNetLayer.feed_forward_chunkr)rrrr_rrrr=r=rvr>r}s  Grcs8eZdZfddZ         dddZZS) PoNetEncodercs4t|_tfddtjD|_dS)Ncsg|]}tqSr=)r).0r;rur=r> sz)PoNetEncoder.__init__..)r^r_rur ModuleListrangenum_hidden_layerslayerrsrvrr>r_s  zPoNetEncoder.__init__NFTc  sr| rdnd} r dnd}r|jjrdnd}| rdnd}t|jD]w\}}| r,| |f} |dur4||nd}|dur>||ndt|jddrk|jrk| rStdd} fdd}tj j |||||||||}n |||||||| }|d}| r||df7}r||d f}|jjr||d f}q!| r| |f} | st d d ||| ||fDSt ||| ||d S)Nr=gradient_checkpointingFzh`use_cache=True` is incompatible with `config.gradient_checkpointing=True`. Setting `use_cache=False`...csfdd}|S)Ncsg|RSrIr=)inputs)modulerrr=r>custom_forward s zKPoNetEncoder.forward..create_custom_forward..custom_forwardr=)rrr)rr>create_custom_forward sz3PoNetEncoder.forward..create_custom_forwardrrrrcss|] }|dur|VqdSrIr=)rrr=r=r> 7s z'PoNetEncoder.forward..)last_hidden_statepast_key_valuesrP attentionscross_attentions) rur enumeraterrntrainingloggerwarningr"utils checkpointtupler)rtrPrrrrrrr use_cacheroutput_hidden_states return_dictall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_maskr layer_outputsr=rr>rs        zPoNetEncoder.forward) NNNNNNFFTrr=r=rvr>rs  rcr) PoNetPoolercs*tt|j|j|_t|_dSrI)r^r_rrrbrTanh activationrsrvr=r>r_Is zPoNetPooler.__init__cCs(|dddf}||}||}|S)Nr)rr)rtrPfirst_token_tensor pooled_outputr=r=r>rNs  zPoNetPooler.forwardrr=r=rvr>rGs rcsFeZdZdZeZdZdgZfddZddZ e fdd Z Z S) PoNetPreTrainedModelzm A base class to handle weights initialization and a simple interface for loading pretrained models. ponetrZc s*tj|jfi|tt||dSrI)r^r_ name_or_pathr )rtrukwargsrvr=r>r_`szPoNetPreTrainedModel.__init__cCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsr)rstdNg?)rrrweightdatanormal_ruinitializer_rangebiaszero_r`rUrifill_)rtrr=r=r> _init_weightsds$      z"PoNetPreTrainedModel._init_weightsc sD|dd}|durtdi|}||}|Stt|j|d}|S)N model_dir)pretrained_model_name_or_pathr=)poprr^r from_pretrained)clsrr! ponet_configmodelrvr=r> _instantiatevs z!PoNetPreTrainedModel._instantiate) rrrrr config_classbase_model_prefix_keys_to_ignore_on_load_missingr_r  classmethodr(rr=r=rvr>rWs r) module_namecs`eZdZdZdfdd ZddZddZd d Z dd d ZZ S) PoNetModela&The bare PoNet Model transformer outputting raw hidden-states without any specific head on top. This model inherits from :class:`~transformers.PreTrainedModel`. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch `torch.nn.Module `__ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (:class:`~modelscope.models.nlp.ponet.PoNetConfig`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the :meth:`~transformers.PreTrainedModel.from_pretrained` method to load the model weights. The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in `Attention is all you need `__ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the :obj:`is_decoder` argument of the configuration set to :obj:`True`. To be used in a Seq2Seq model, the model needs to initialized with both :obj:`is_decoder` argument and :obj:`add_cross_attention` set to :obj:`True`; an :obj:`encoder_hidden_states` is then expected as an input to the forward pass. Tc sLtj|fi|||_t||_t||_|rt|nd|_| dSrI) r^r_rurTrrencoderrpooler init_weights)rtruadd_pooling_layerrrvr=r>r_s    zPoNetModel.__init__cCs|jjSrIrrd)rtr=r=r>get_input_embeddingsszPoNetModel.get_input_embeddingscCs ||j_dSrIr3)rtrr=r=r>set_input_embeddingss zPoNetModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr/rrr)rtheads_to_prunerrr=r=r> _prune_headsszPoNetModel._prune_headsNc Cs6| dur| n|jj} | dur| n|jj} |dur|n|jj}|jjr-| dur(| n|jj} nd} |dur;|dur;td|durH|}|\}}n|durY|dd}|\}}ntd|durd|jn|j}| durt| ddj dnd}|durt j |||f|d}|durt j |t j |d }||||}|jjr|dur|\}}}||f}| durt j ||d} || }nd}|||jj}|j|||||d }|durt|n|}t|}|j|||||||| | | | |d }|d}|jdur||nd}|s ||f|d dSt|||j|j|j|jd S)u Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using :class:`~modelscope.models.nlp.ponet.PoNetTokenizer`. See :meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for details. attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. token_type_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Segment token indices to indicate first and second portions of the inputs. Indices are selected in ``[0,1]``: - 0 corresponds to a `sentence A` token, - 1 corresponds to a `sentence B` token. position_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range ``[0,config.max_position_embeddings - 1]``. head_mask (:obj:`torch.FloatTensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`): Mask to nullify selected heads of the self-attention modules. Mask values selected in ``[0, 1]``: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`): Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert :obj:`input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (:obj:`bool`, `optional`): Whether or not to return the attentions tensors of all attention layers. See ``attentions`` under returned tensors for more detail. output_hidden_states (:obj:`bool`, `optional`): Whether or not to return the hidden states of all layers. See ``hidden_states`` under returned tensors for more detail. return_dict (:obj:`bool`, `optional`): Whether or not to return a :class:`~transformers.file_utils.ModelOutput` instead of a plain tuple. encoder_hidden_states (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in ``[0, 1]``: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (:obj:`tuple(tuple(torch.FloatTensor))` of length :obj:`config.n_layers` with each tuple having 4 tensors of shape :obj: `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If :obj:`past_key_values` are used, the user can optionally input only the last :obj:`decoder_input_ids` (those that don't have their past key value states given to this model) of shape :obj:`(batch_size, 1)` instead of all :obj:`decoder_input_ids` of shape :obj:`(batch_size, sequence_length)`. use_cache (:obj:`bool`, `optional`): If set to :obj:`True`, :obj:`past_key_values` key value states are returned and can be used to speed up decoding (see :obj:`past_key_values`). Returns: Returns `modelscope.outputs.AttentionBackboneModelOutput` Examples: >>> from modelscope.models import Model >>> from modelscope.preprocessors import Preprocessor >>> model = Model.from_pretrained('damo/nlp_ponet_fill-mask_chinese-base', task='backbone') >>> preprocessor = Preprocessor.from_pretrained('damo/nlp_ponet_fill-mask_chinese-base') >>> print(model(**preprocessor('这是个测试'))) NFzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsrrrr\)rDrZr[ryrz) rrrrrrrrrr)r pooler_outputrrPrr)rurruse_return_dictrr ValueErrorr/rr'r"onesrrr@get_extended_attention_maskinvert_attention_mask get_head_maskrrrHrJr/r0rrrPrr) rtrDrr[ segment_idsrZrryrrrrrrrr{ batch_sizer|rrzextended_attention_maskencoder_batch_sizeencoder_sequence_lengthr;encoder_hidden_shapeencoder_extended_attention_maskembedding_outputrrencoder_outputssequence_outputrr=r=r>rs`       zPoNetModel.forward)T)NNNNNNNNNNNNNN) rrrrr_r4r5r8rrr=r=rvr>r.s*  r.)r)rK)7rrdistutils.versionrr"torch.utils.checkpoint packagingrrtransformers.activationsrtransformers.modeling_utilsrrrr modelscope.metainfor modelscope.modelsr r modelscope.models.builderr modelscope.outputsrmodelscope.utils.constantrmodelscope.utils.loggerr configurationrrrqr!CLS_IDEOS_IDr?rHrJrSModulerTrrrrrrrrrregister_modulebackbonerr.r=r=r=r>sH            Oh8_k,