o ߥi~@sTdZddlmZddlmZmZmZddlZddlZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZmZdd lmZdd lmZddlmZddlmZddl m!Z!e"ZGddde j#Z$Gdddej%j&Z'Gddde(Z)ddZ*Gdddej%j&Z+Gddde j#Z,Gddde j#Z-Gd d!d!e j#Z.Gd"d#d#e j#Z/Gd$d%d%e j#Z0Gd&d'd'e j#Z1Gd(d)d)e j#Z2Gd*d+d+e j#Z3d,d-Z4 . .d@d/d0Z5ej6j7d1d2Z8ej6j7d3d4Z9ej6j7d5d6Z:Gd7d8d8e j#Z;Gd9d:d:e j#Zej?ej@d=Gd>d?d?e=ZAdS)Az PyTorch DeBERTa-v2 model.)Sequence)OptionalTupleUnionN)nn) LayerNorm)ACT2FN)PreTrainedModel)softmax_backward_data)Models)Model TorchModel)MODELS)AttentionBackboneModelOutput)logger)Tasks)DebertaV2Configcs0eZdZfddZddZeddZZS) ContextPoolercs2tt|j|j|_t|j|_||_ dSN) super__init__rLinearpooler_hidden_sizedense StableDropoutpooler_dropoutdropoutconfigselfr __class__]/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/deberta_v2/backbone.pyr*s   zContextPooler.__init__cCs8|dddf}||}||}t|jj|}|SNr)rrrrpooler_hidden_act)r hidden_states context_token pooled_outputr#r#r$forward1s   zContextPooler.forwardcC|jjSr)r hidden_sizer r#r#r$ output_dim;szContextPooler.output_dim)__name__ __module__ __qualname__rr*propertyr. __classcell__r#r#r!r$r(s   rc@s4eZdZdZeddZeddZeddZdS) XSoftmaxa Masked Softmax which is optimized for saving memory Args: input (`torch.tensor`): The input tensor that will apply softmax. mask (`torch.IntTensor`): The mask matrix where 0 indicate that element will be ignored in the softmax calculation. dim (int): The dimension that will apply softmax Example: >>> import torch >>> from transformers.models.deberta_v2.modeling_deberta_v2 import XSoftmax >>> # Make a tensor >>> x = torch.randn([4, 20, 100]) >>> # Create a mask >>> mask = (x > 0).int() >>> # Specify the dimension to apply softmax >>> dim = -1 >>> y = XSoftmax.apply(x, mask, dim) cCsX||_|tj}||tt|jj}t ||j}| |d| ||Sr%) dimtotorchbool masked_filltensorfinfodtypeminsoftmax masked_fill_save_for_backward)r inputmaskr5rmaskoutputr#r#r$r*\s  zXSoftmax.forwardcCs$|j\}t||||j|}|ddfSr) saved_tensorsr r5)r grad_outputrD inputGradr#r#r$backwardhs   zXSoftmax.backwardc Csddlmm}ddlm}m}|jd||jdd}|jd|d|jdtj dtj d d ||jd d}|||||jdt t | jd } ||| |} ||| ||jdtj dtjd d S) Nr)r9r>CastLong)to_iSubConstantrr<)value_tByte)torch.onnx.symbolic_helperonnxsymbolic_helpertorch.onnx.symbolic_opset9r9r>opcast_pytorch_to_onnxr7r:int64r;typer<r=uint8) gr rBr5sym_helpr9r>mask_cast_valuer_maskrDr#r#r$symbolicos2  zXSoftmax.symbolicN)r/r0r1__doc__ staticmethodr*rHr^r#r#r#r$r4As  r4c@seZdZddZdS)DropoutContextcCsd|_d|_d|_d|_dS)NrrT)rrBscale reuse_maskr-r#r#r$rs zDropoutContext.__init__N)r/r0r1rr#r#r#r$ras racCst|ts |}d}n|j}||j9}|jr|jnd}|dkr2|dur2dt|d| tj }t|tr?|jdur?||_||fS)Nrr) isinstancerarrbrcrBr7 empty_like bernoulli_r6r8)rA local_contextrrBr#r#r$get_masks    rhc @sZeZdZdZeddZeddZedejj dejj de e e fd ejj fd d Zd S) XDropoutzlOptimized dropout function to save computation and memory by using mask operation instead of multiplication.cCsDt||\}}dd||_|dkr ||||d|jS|S)Ng?rr)rhrbr@r9)ctxrA local_ctxrBrr#r#r$r*s  zXDropout.forwardcCs0|jdkr|j\}||d|jdfS|dfS)Nrr)rbrEr9)rjrFrBr#r#r$rHs zXDropout.backwardrZrArkreturncCs4ddlm}|}t|tr|j}d}|||||S)Nr)symbolic_opset12T) torch.onnxrmrdrar)rZrArkrm dropout_ptrainr#r#r$r^s  zXDropout.symbolicN)r/r0r1r_r`r*rHr7_CGraphValuerfloatrar^r#r#r#r$ris   ricsBeZdZdZfddZddZddZdd d Zd d ZZ S)rz Optimized dropout module for stabilizing the training Args: drop_prob (float): the dropout probabilities cs t||_d|_d|_dSr%)rr drop_probcount context_stack)r rur!r#r$rs  zStableDropout.__init__cCs$|jr|jdkrt||S|S)zr Call the module Args: x (`torch.tensor`): The input tensor to apply dropout r)trainingruriapply get_context)r xr#r#r$r*szStableDropout.forwardcCsd|_d|_dSr%)rvrwr-r#r#r$ clear_contexts zStableDropout.clear_contextTrcCs2|jdurg|_d|_|jD]}||_||_qdSr%)rwrvrcrb)r rcrbcr#r#r$ init_contexts  zStableDropout.init_contextcCsT|jdur'|jt|jkr|jt|j|j}|j|_|jd7_|S|jS)Nr)rwrvlenappendrarur)r rjr#r#r$rzs  zStableDropout.get_context)Tr) r/r0r1r_rr*r|r~rzr3r#r#r!r$rs  rc$eZdZfddZddZZS)DebertaV2SelfOutputcs<tt|j|j|_t|j|j|_t|j |_ dSr) rrrrr,rrlayer_norm_epsrhidden_dropout_probrrr!r#r$rs zDebertaV2SelfOutput.__init__cC&||}||}|||}|Srrrrr r' input_tensorr#r#r$r*   zDebertaV2SelfOutput.forwardr/r0r1rr*r3r#r#r!r$rs rcs.eZdZfddZ    dddZZS)DebertaV2Attentioncs(tt||_t||_||_dSr)rrDisentangledSelfAttentionr rrDrrr!r#r$rs    zDebertaV2Attention.__init__FNc CsJ|j||||||d}|r|\}}|dur|}|||} |r#| |fS| S)N) query_states relative_posrel_embeddings)r rD) r r'attention_maskoutput_attentionsrrr self_output att_matrixattention_outputr#r#r$r*s   zDebertaV2Attention.forwardFNNNrr#r#r!r$r  rcs2eZdZfddZdejdejfddZZS)DebertaV2IntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) rrrrr,intermediate_sizerrd hidden_actstrrintermediate_act_fnrr!r#r$r;s   zDebertaV2Intermediate.__init__r'rlcCs||}||}|Sr)rr)r r'r#r#r$r*Cs  zDebertaV2Intermediate.forward)r/r0r1rr7Tensorr*r3r#r#r!r$r9s rcr)DebertaV2OutputcsBtt|j|j|_t|j|j|_t |j |_ ||_ dSr) rrrrrr,rrrrrrrrr!r#r$rLs   zDebertaV2Output.__init__cCrrrrr#r#r$r*SrzDebertaV2Output.forwardrr#r#r!r$rJs rcs.eZdZfddZ    dddZZS)DebertaV2Layercs,tt||_t||_t||_dSr)rrr attentionr intermediaterrDrr!r#r$r]s   zDebertaV2Layer.__init__NFc CsH|j||||||d}|r|\}}||} || |} |r"| |fS| S)Nrrrr)rrrD) r r'rrrrrrrintermediate_output layer_outputr#r#r$r*cs   zDebertaV2Layer.forward)NNNFrr#r#r!r$r[rrcr) ConvLayercsxtt|dd}t|dd}t|dd|_tj|j|j||dd|d|_t|j|j |_t |j |_ ||_ dS) Nconv_kernel_size conv_groupsrconv_acttanh)paddinggroups)rrgetattrrrConv1dr,convrrrrrr)r r kernel_sizerr!r#r$rs      zConvLayer.__init__c Cs||dddddd}d|}||d|dt|j | |}||}| | |}|durE|}|S| | kr`| dkr[|dd}|d}| |j}||}|S)Nrrr)rpermute contiguousr8r? unsqueezeexpandsizerrrrr6r5squeezer<) r r'residual_states input_maskoutrClayer_norm_inputrD output_statesr#r#r$r*s(     zConvLayer.forwardrr#r#r!r$r~s rcsNeZdZdZfddZddZddZdd d Z   dd dZZ S)DebertaV2Encoderz8Modified BertEncoder with relative position bias supportcsttfddtjD|_tdd|_|jrMtdd|_ |j dkr/j |_ tdd|_ |j d }|j d krE|j d }t |j |_d dtd d dD|_d|jvrmtj jdd|_tdd d krytnd|_d|_dS)Ncsg|]}tqSr#)r.0_rr#r$ sz-DebertaV2Encoder.__init__..relative_attentionFmax_relative_positionsrrposition_bucketsrrcSsg|]}|qSr#)strip)rr{r#r#r$rs norm_rel_ebdnone| layer_normT)elementwise_affiner)rrr ModuleListrangenum_hidden_layerslayerrrrmax_position_embeddingsr Embeddingr,rlowersplitrrrrrgradient_checkpointing)r r pos_ebd_sizer!rr$rsF        zDebertaV2Encoder.__init__cCs2|jr|jjnd}|durd|jvr||}|S)Nr)rrweightrr)r rr#r#r$get_rel_embeddings z"DebertaV2Encoder.get_rel_embeddingcCsV|dkr|dd}||dd}|}|S|dkr)|d}|S)Nrrrr)r5rrbyte)r rextended_attention_maskr#r#r$get_attention_masks  z#DebertaV2Encoder.get_attention_maskNcCsH|jr"|dur"|dur|dn|d}t||d|j|jd}|S)Nr bucket_size max_position)rrbuild_relative_positionrr)r r'rrqr#r#r$ get_rel_posszDebertaV2Encoder.get_rel_posTFc s|dkr |}n |ddk}||}||||}|r"dnd} r(dnd} t|tr4|d} n|} |} | } t|j D]l\}}|rL| | f} |j rg|j rgfdd}t j j||| |||| } n || |||| d} rw| \} }|dkr|jdur||| |} |dur| }t|tr|dt|j kr||dnd} n| } r| |f} qA|r| | f} |std d | | | fDSt| | | d S) Nrrrr#csfdd}|S)Ncsg|RSrr#)inputs)modulerr#r$custom_forwardszODebertaV2Encoder.forward..create_custom_forward..custom_forwardr#)rrr)rr$create_custom_forward sz7DebertaV2Encoder.forward..create_custom_forward)rrrrrcss|] }|dur|VqdSrr#)rvr#r#r$ <sz+DebertaV2Encoder.forward..last_hidden_stater' attentions)r5sumrrrrdrr enumeraterrrxr7utils checkpointrrtupler)r r'routput_hidden_statesrrr return_dictrall_hidden_statesall_attentionsnext_kvrri layer_moduleratt_mr#rr$r*s              zDebertaV2Encoder.forward)NN)TFNNT) r/r0r1r_rrrrr*r3r#r#r!r$rs % rc Cst|}|d}t||k|| k@t|d|t|}tt||tt|d||d|}t||k||||}|S)Nrr)r7signwherer:type_asabsceillog)rrrrmidabs_poslog_pos bucket_posr#r#r$make_log_bucket_positionEs.   rrcCstd|}td|}|dddf|dddf}|dkr*|dkr*t|||}|tj}|d|ddf}|d}|S)af Build relative position according to the query and key We assume the absolute position of query \(P_q\) is range from (0, query_size) and the absolute position of key \(P_k\) is range from (0, key_size), The relative positions from query to key is \(R_{q \rightarrow k} = P_q - P_k\) Args: query_size (int): the length of query key_size (int): the length of key bucket_size (int): the size of position bucket max_position (int): the maximum allowed absolute position Return: `torch.LongTensor`: A tensor with shape [1, query_size, key_size] rN)r7arangerr6longr) query_sizekey_sizerrq_idsk_ids rel_pos_idsr#r#r$rWs     rcCs*||d|d|d|dgS)Nrrrrrr)c2p_pos query_layerrr#r#r$c2p_dynamic_expandx rcCs*||d|d|d|dgS)Nrrrr )r r key_layerr#r#r$p2c_dynamic_expandrrcCs*||dd|d|dfS)Nrrr ) pos_indexp2c_attrr#r#r$pos_dynamic_expandsrcsBeZdZdZfddZddZ    d dd Zd d ZZS) ra Disentangled self-attention module Parameters: config (`DebertaV2Config`): A model config class instance with the configuration to build a new model. The schema is similar to *BertConfig*, for more details, please refer [`DebertaV2Config`] cst|j|jdkrtd|jd|jd|j|_|j|j}t|d||_|j|j|_tj |j|jdd|_ tj |j|jdd|_ tj |j|jdd|_ t|dd |_ |jdurb|jng|_t|d d |_|jrt|d d |_t|d d |_|jdkr|j|_|j|_|jdkr|j|_t|j|_|j sd|jvrtj |j|jdd|_d|jvrt |j|j|_t|j|_dS)NrzThe hidden size (z6) is not a multiple of the number of attention heads ()attention_head_sizeTbias share_att_keyFrrrrrc2pp2c)rrr,num_attention_heads ValueErrorrr all_head_sizerr query_projkey_proj value_projr pos_att_typerrrrrrr pos_dropout pos_key_projpos_query_projattention_probs_dropout_probr)r r_attention_head_sizer!r#r$rs\            z"DisentangledSelfAttention.__init__cCsL|dd|df}||}|ddddd|d|dS)Nrrrrr)rviewrr)r r{attention_heads new_x_shaper#r#r$transpose_for_scoress  z.DisentangledSelfAttention.transpose_for_scoresFNc Cs|dur|}||||j}||||j}||||j} d} d} d|jvr1| d7} d|jvr:| d7} ttj| dtj d| } t || ddtj| |j d} |jrm||}|||||| } | duru| | } | } | d|j| d| d} t| |d}||}t |d| d| d| }|d|j| d| ddd dd }| ddd }||}|r||fS|S) a Call the module Args: hidden_states (`torch.FloatTensor`): Input states to the module usually the output from previous layer, it will be the Q,K and V in *Attention(Q,K,V)* attention_mask (`torch.ByteTensor`): An attention mask matrix of shape [*B*, *N*, *N*] where *B* is the batch size, *N* is the maximum sequence length in which element [i,j] = *1* means the *i* th token in the input can attend to the *j* th token. output_attentions (`bool`, optional): Whether return the attention matrix. query_states (`torch.FloatTensor`, optional): The *Q* state in *Attention(Q,K,V)*. relative_pos (`torch.LongTensor`): The relative position encoding between the tokens in the sequence. It's of shape [*B*, *N*, *N*] with values ranging in [*-max_relative_positions*, *max_relative_positions*]. rel_embeddings (`torch.FloatTensor`): The embedding of relative distances. It's a tensor of shape [\(2 \times \text{max_relative_positions}\), *hidden_size*]. NrrrrrNrrrrr)r,r rr!r"r#r7sqrtr:rrtbmm transposer<rr$disentangled_attention_biasr)r4ryrrr)r r'rrrrrrr value_layerrel_att scale_factorrbattention_scoresattention_probs context_layernew_context_layer_shaper#r#r$r*s&           z!DisentangledSelfAttention.forwardc Cs$|dur|d}t||d|j|jd}|dkr%|dd}n|dkr1|d}n|dkr@td||j}| |j }|d|dddfd}|j r| | ||j|d|jdd}| |||j|d|jdd} n4d |jvr| |||j|d|jdd} d |jvr| |||j|d|jdd}d} d |jvrttj| d tjd |} t|| d d} t||d|dd} tj| d | d|d|d|d gd } | | tj| | jd 7} d |jvrttj|d tjd |} |d|dkrLt|d|d|j|jd |j }|d}n|}t| |d|dd}t||d d}tj|d |d|d|d|dgd d d}| |tj| |jd 7} | S)Nrrrrrrrz2Relative position ids must be of dim 2 or 3 or 4. rrrrN)r5index)rrrrr5rrrrr6devicerr,r rrepeatr!r#r%r&r7r.r:rtr/r0clampgatherrrr<)r rrrrr4ratt_spanpos_query_layer pos_key_layerscorerbc2p_attr r_posp2c_posrr#r#r$r1.s              z5DisentangledSelfAttention.disentangled_attention_biasr) r/r0r1r_rr,r*r1r3r#r#r!r$rs ,  \rcs4eZdZdZfddZ     dddZZS)DebertaV2EmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cstt|dd}t|d|j|_tj|j|j|d|_t|dd|_ |j s,d|_ n t|j |j|_ |j dkrCt|j |j|_ |j|jkrTtj|j|jdd|_t|j|j|_t|j|_||_|d t|j d dS) N pad_token_idrembedding_size) padding_idxposition_biased_inputTFr position_ids)rr)rrrr,rGrr vocab_sizeword_embeddingsrIposition_embeddingsrtype_vocab_sizetoken_type_embeddingsr embed_projrrrrrrregister_bufferr7rr)r rrFr!r#r$rs>       zDebertaV2Embeddings.__init__Nc CsN|dur |}n|dd}|d}|dur$|jddd|f}|dur3tj|tj|jjd}|dur<||}|jdurI||}nt|}|} |j rW| |7} |j j dkrf| |} | | 7} |j |j jkrr|| } || } |dur|| kr|dkr|dd}|d}|| j}| |} || } | S)Nrrr<r:rrr)rrJr7zerosrr:rLrM zeros_likerIrrNrOrGr,rPrr5rrr6r<r) r input_idstoken_type_idsrJrB inputs_embeds input_shape seq_lengthrM embeddingsrOr#r#r$r*sB             zDebertaV2Embeddings.forward)NNNNN)r/r0r1r_rr*r3r#r#r!r$rEs !rEcsZeZdZdZeZdZdgZdgZdZ fddZ dd Z dd d Z e fd dZZS)DebertaV2PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. debertarJrMTc s*tj|jfi|tt||dSr)rr name_or_pathr r rkwargsr!r#r$rsz!DebertaV2PreTrainedModel.__init__cCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rA|jjjd|jjd|j durC|jj|j  dSdSdS)zInitialize the weights.g)meanstdN) rdrrrdatanormal_rinitializer_rangerzero_rrH)r rr#r#r$ _init_weightss    z&DebertaV2PreTrainedModel._init_weightsFcCst|tr ||_dSdSr)rdrr)r rvaluer#r#r$_set_gradient_checkpointings  z4DebertaV2PreTrainedModel._set_gradient_checkpointingc sD|dd}|durtdi|}||}|Stt|j|d}|S)N model_dir)pretrained_model_name_or_pathr#)poprrr from_pretrained)clsr_ri ponet_configmodelr!r#r$ _instantiate s z%DebertaV2PreTrainedModel._instantiate)F)r/r0r1r_r config_classbase_model_prefix_keys_to_ignore_on_load_missing"_keys_to_ignore_on_load_unexpectedsupports_gradient_checkpointingrrfrh classmethodrpr3r#r#r!r$r[s  r[) module_namecseZdZdZfddZddZddZdd Z dd ee j d ee j d ee j dee j dee j dee dee dee de e effddZZS)DebertaV2Modela The bare DeBERTa_v2 Model transformer outputting raw hidden-states without any specific head on top. The DeBERTa model was proposed in [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://arxiv.org/abs/2006.03654) by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen. It's build on top of BERT/RoBERTa with two improvements, i.e. disentangled attention and enhanced mask decoder. With those two improvements, it out perform BERT/RoBERTa on a majority of tasks with 80GB pretraining data. This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (`DebertaV2Config`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. c s8t|t||_t||_d|_||_|dSr%) rrrErZrencoderz_stepsr post_initr^r!r#r$r-s    zDebertaV2Model.__init__cCr+rrZrLr-r#r#r$get_input_embeddings7sz#DebertaV2Model.get_input_embeddingscCs ||j_dSrr|)r new_embeddingsr#r#r$set_input_embeddings:s z#DebertaV2Model.set_input_embeddingscCstd)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel z7The prune function is not implemented in DeBERTa model.)NotImplementedError)r heads_to_pruner#r#r$ _prune_heads=szDebertaV2Model._prune_headsNrUrrVrJrWrrrrlc  s|dur|njj}|dur|njj}|dur|njj}|dur*|dur*td|dur3|} n|dur@|dd} ntd|durK|jn|j} |durYtj| | d}|durftj | tj | d}j |||||d} j | |d||d } | d } j d kr| d }fd d tj D}| d}j }j |}j | }|d dD]}|||d|||d}| |q| d}|s|f| |rd nddSt||r| jnd| jdS)u5 Args: input_ids (`torch.LongTensor` of shape `('batch_size, sequence_length')`): Indices of input sequence tokens in the vocabulary. attention_mask (`torch.FloatTensor` of shape `('batch_size, sequence_length')`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. token_type_ids (`torch.LongTensor` of shape `('batch_size, sequence_length')`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. position_ids (`torch.LongTensor` of shape `('batch_size, sequence_length')`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,config.max_position_embeddings - 1]`. inputs_embeds (`torch.FloatTensor` of shape `('batch_size, sequence_length', hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a dataclass instead of a plain tuple. Returns: Returns `modelscope.outputs.AttentionBackboneModelOutput` Examples: >>> from modelscope.models import Model >>> from modelscope.preprocessors import Preprocessor >>> model = Model.from_pretrained('damo/nlp_debertav2_fill-mask_chinese-lite', task='backbone') >>> preprocessor = Preprocessor.from_pretrained('damo/nlp_debertav2_fill-mask_chinese-lite') >>> print(model(**preprocessor('这是个测试'))) NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embeds)r:rR)rUrVrJrBrWT)rrrrrcsg|]}jjdqSr-)ryrrr-r#r$rsz*DebertaV2Model.forward..Frrr)rrruse_return_dictrrr:r7onesrSrrZryrzrrrrrrr'r)r rUrrVrJrWrrrrXr:embedding_outputencoder_outputsencoded_layersr'layersrrrel_posrsequence_outputr#r-r$r*Es9       zDebertaV2Model.forward)NNNNNNNN)r/r0r1r_rr}rrrr7rr8rrrr*r3r#r#r!r$rxsB     rx)rr)Br_collections.abcrtypingrrrr7torch.utils.checkpointrtorch.nnrtransformers.activationsrtransformers.modeling_utilsr transformers.pytorch_utilsr modelscope.metainfor modelscope.modelsr r modelscope.models.builderrmodelscope.outputsrmodelscope.utilsrloggingmodelscope.utils.constantr configurationr get_loggerModulerautogradFunctionr4objectrarhrirrrrrrrrrrjitscriptrrrrrEr[register_modulebackbone deberta_v2rxr#r#r#r$s`            H )2&#) !   |Y0