o ߥi|(@sddlmZmZmZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZd d lmZmZejeje jd Gd ddeZGdddejZGdddejZGdddejZdS))OptionalTupleUnionN)nn)CrossEntropyLoss)ACT2FN)Models)MODELS)AttentionFillMaskModelOutput)Tasks)DebertaV2ModelDebertaV2PreTrainedModel) module_namecseZdZdZdgZddgZfddZddZd d Z dd e e j d e e j de e j de e j de e j de e j de e de e de e de eeffddZZS)DebertaV2ForMaskedLMaDeBERTa_v2 Model with a `language modeling` head on top. The DeBERTa model was proposed in [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://arxiv.org/abs/2006.03654) by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen. It's build on top of BERT/RoBERTa with two improvements, i.e. disentangled attention and enhanced mask decoder. With those two improvements, it out perform BERT/RoBERTa on a majority of tasks with 80GB pretraining data. This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Preprocessor: This is the fill_mask model of Deberta_v2, the preprocessor of this model is `modelscope.preprocessors.FillMaskTransformersPreprocessor`. Parameters: config (`DebertaV2Config`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. pooler position_idszpredictions.decoder.biasc s,t|t||_t||_|dSN)super__init__r debertaDebertaV2OnlyMLMHeadcls post_init)selfconfigkwargs __class__^/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/deberta_v2/fill_mask.pyr<s    zDebertaV2ForMaskedLM.__init__cCs |jjjSrr predictionsdecoder)rrrr get_output_embeddingsEs z*DebertaV2ForMaskedLM.get_output_embeddingscCs||jj_dSrr!)rnew_embeddingsrrr set_output_embeddingsHsz*DebertaV2ForMaskedLM.set_output_embeddingsN input_idsattention_masktoken_type_ids inputs_embedslabelsoutput_attentionsoutput_hidden_states return_dictreturnc  Cs| dur| n|jj} |j|||||||| d} | d} || } d} |dur7t}|| d|jj|d} | sM| f| dd}| durK| f|S|St| | || j| j dS)uD Args: input_ids (`torch.LongTensor` of shape `('batch_size, sequence_length')`): Indices of input sequence tokens in the vocabulary. attention_mask (`torch.FloatTensor` of shape `('batch_size, sequence_length')`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. token_type_ids (`torch.LongTensor` of shape `('batch_size, sequence_length')`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. position_ids (`torch.LongTensor` of shape `('batch_size, sequence_length')`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. inputs_embeds (`torch.FloatTensor` of shape `('batch_size, sequence_length', hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a dataclass instead of a plain tuple. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Returns: Returns `modelscope.outputs.AttentionFillMaskModelOutput` Examples: >>> from modelscope.models import Model >>> from modelscope.preprocessors import Preprocessor >>> model = Model.from_pretrained('damo/nlp_debertav2_fill-mask_chinese-lite') >>> preprocessor = Preprocessor.from_pretrained('damo/nlp_debertav2_fill-mask_chinese-lite') >>> # Call the model, return some tensors >>> print(model(**preprocessor('你师父差得动你，你师父可[MASK]不动我。'))) >>> # Call the pipeline >>> from modelscope.pipelines import pipeline >>> pipeline_ins = pipeline('fill-mask', model=model, preprocessor=preprocessor) >>> print(pipeline_ins('你师父差得动你，你师父可[MASK]不动我。')) N)r(r)rr*r,r-r.rr )losslogitsr' attentions hidden_states) ruse_return_dictrrrview vocab_sizer r3r4)rr'r(r)rr*r+r,r-r.outputssequence_outputprediction_scoresmasked_lm_lossloss_fctoutputrrr forwardKsFC zDebertaV2ForMaskedLM.forward) NNNNNNNNN)__name__ __module__ __qualname____doc__"_keys_to_ignore_on_load_unexpected_keys_to_ignore_on_load_missingrr$r&rtorchTensorboolrrr r> __classcell__rrrr r sL     rc$eZdZfddZddZZS) DebertaV2PredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dS)N)eps) rrrLinear hidden_sizedense isinstance hidden_actstrrtransform_act_fn LayerNormlayer_norm_epsrrrrr rs   z)DebertaV2PredictionHeadTransform.__init__cCs"||}||}||}|Sr)rNrRrSrr4rrr r>s   z(DebertaV2PredictionHeadTransform.forwardr?r@rArr>rHrrrr rJs  rJcrI)DebertaV2LMPredictionHeadcsLtt||_tj|j|jdd|_t t |j|_ |j |j_ dS)NF)bias) rrrJ transformrrLrMr7r# ParameterrEzerosrYrUrrr rs   z"DebertaV2LMPredictionHead.__init__cCs||}||}|Sr)rZr#rVrrr r>s  z!DebertaV2LMPredictionHead.forwardrWrrrr rXs rXcrI)rcstt||_dSr)rrrXr"rUrrr rs zDebertaV2OnlyMLMHead.__init__cCs||}|Sr)r")rr9r:rrr r>s zDebertaV2OnlyMLMHead.forwardrWrrrr rs r)typingrrrrEtorch.utils.checkpointrtorch.nnrtransformers.activationsrmodelscope.metainformodelscope.models.builderr modelscope.outputsr modelscope.utils.constantr backboner rregister_module fill_mask deberta_v2rModulerJrXrrrrr s"