o ߥih@sdZddlmZmZmZmZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlmmZddlmZddlmZddlmZddlmZdd Zd d Zd d ZeeZeej ddddddddZ!dZ"dZ#dZ$ddZ%ddZ&ddZ'e&ejjj(e'dZ)Gd d!d!e*Z+zdd"l,m-Z.Wne/ye0d#Gd$d%d%ej1Z.YnwGd&d'd'ej1Z2Gd(d)d)ej1Z3Gd*d+d+ej1Z4Gd,d-d-ej1Z5Gd.d/d/ej1Z6Gd0d1d1ej1Z7Gd2d3d3ej1Z8Gd4d5d5ej1Z9Gd6d7d7ej1Z:Gd8d9d9ej1Z;Gd:d;d;ej1Zd?d?ej1Z>Gd@dAdAej1Z?GdBdCdCej1Z@GdDdEdEe@ZAGdFdGdGe@ZBGdHdIdIe@ZCGdJdKdKe@ZDGdLdMdMe@ZEGdNdOdOe@ZFGdPdQdQe@ZGGdRdSdSe@ZHdS)TzPyTorch BERT model.)absolute_importdivisionprint_functionunicode_literalsN) cached_path)mpu)nn)CrossEntropyLosscsfdd}|S)Nctjjj|dSNmeanstdtorchrinitnormal_tensorr b/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/mglm/model/modeling_bert.pyinit_)z!normal_init_method..init_r)r rrrr rnormal_init_method'srcs$td|fdd}|S)z3Init method based on N(0, sigma/sqrt(2*num_layers).@cr r rrr rrr3rz!scaled_init_method..init_)mathsqrt)r r num_layersrrr rscaled_init_method/srcCs*|d}|d}||}|d}|S)N) unsqueeze)attention_maskattention_mask_b1sattention_mask_bs1attention_mask_bssextended_attention_maskrrrbert_extended_attention_mask9s   r'z#/root/data/bert-base-uncased.tar.gzzMhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased.tar.gzzJhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased.tar.gzzKhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased.tar.gzzYhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased.tar.gzzWhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased.tar.gzzLhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese.tar.gz)zbert-base-uncasedzbert-large-uncasedzbert-base-casedzbert-large-casedzbert-base-multilingual-uncasedzbert-base-multilingual-casedzbert-base-chinesezbert_config.jsonzpytorch_model.binz model.ckptc Cs:zddl}ddl}ddl}Wn tytdwtj|}td||j |}g}g}|D]\} } td| | |j || } | | | | q4t ||D]\} } | d} tdd| Drttd d| qW|} | D]Y} |d | r|d | }n| g}|dd ks|dd krt| d} n%|ddks|ddkrt| d} n|ddkrt| d} nt| |d} t|dkrt|d}| |} qx| dddkrt| d} n | d kr|| } z | j| jksJWnty }z |j| j| jf7_d}~wwtd| t| | _qW|S)z, Load tf checkpoints in a pytorch model rNzLoading a TensorFlow models in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z(Converting TensorFlow checkpoint from {}z"Loading TF weight {} with shape {}/css|]}|dvVqdS))adam_vadam_mNr).0nrrr |sz*load_tf_weights_in_bert..z Skipping {}z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightsr ri _embeddingszInitialize PyTorch weight {})renumpy tensorflow ImportErrorprintospathabspathformattrainlist_variables load_variableappendzipsplitanyjoin fullmatchgetattrlenint transposeshapeAssertionErrorargsr from_numpydata)modeltf_checkpoint_pathr6nptftf_path init_varsnamesarraysnamerLarraypointerm_namelnumerrrload_tf_weights_in_bert_sn                 r`cCs |ddt|tdS)zImplementation of the gelu activation function. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) g??r)rerfrrxrrrgelus recCs|t|SN)rsigmoidrcrrrswishsrh)rerelurhc@sheZdZdZ          dd d ZeddZeddZddZddZ ddZ dS) BertConfigzEConfiguration class to store the configuration of a `BertModel`.  re皙?r {Gz?F-q=cCst|tr3t|ddd}t|}Wdn1swY|D] \}}||j|<q'dSt|trj||_ ||_ ||_ ||_ ||_ ||_||_||_| |_| |_| |_| |_| |_||_||_||_dStd)aConstructs BertConfig. Args: vocab_size_or_config_json_file: Vocabulary size of `inputs_ids` in `BertModel`. hidden_size: Size of the encoder layers and the pooler layer. num_hidden_layers: Number of hidden layers in the Transformer encoder. num_attention_heads: Number of attention heads for each attention layer in the Transformer encoder. intermediate_size: The size of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. hidden_act: The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu" and "swish" are supported. hidden_dropout_prob: The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. attention_probs_dropout_prob: The dropout ratio for the attention probabilities. max_position_embeddings: The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). type_vocab_size: The vocabulary size of the `token_type_ids` passed into `BertModel`. initializer_range: The stdev of the truncated_normal_initializer for initializing all weight matrices. rutf-8encodingNzhFirst argument must be either a vocabulary size (int)or the path to a pretrained model config file (str)) isinstancestropenjsonloadsreaditems__dict__rJ vocab_size hidden_sizenum_hidden_layersnum_attention_heads hidden_actintermediate_sizehidden_dropout_probattention_probs_dropout_probmax_position_embeddingstype_vocab_sizeinitializer_range deep_initfp32_layernormfp32_embeddinglayernorm_epsilonfp32_tokentypes ValueError)selfvocab_size_or_config_json_filerrrrrrrrrrrrrrrreader json_configkeyvaluerrr__init__s> )   zBertConfig.__init__cCs*tdd}|D] \}}||j|<q |S)zAConstructs a `BertConfig` from a Python dictionary of parameters.)r)rjr|r})cls json_objectconfigrrrrr from_dicts  zBertConfig.from_dictcCsFt|ddd }|}Wdn1swY|t|S)z9Constructs a `BertConfig` from a json file of parameters.rrrsrtN)rxr{rryrz)r json_filertextrrrfrom_json_files zBertConfig.from_json_filecCs t|Srf)rwto_json_stringrrrr__repr__s zBertConfig.__repr__cCst|j}|S)z0Serializes this instance to a Python dictionary.)copydeepcopyr}routputrrrto_dict s zBertConfig.to_dictcCstj|ddddS)z*Serializes this instance to a JSON string.r T)indent sort_keys )rydumpsrrrrrrszBertConfig.to_json_stringN)rkrlrlrmrernrnror rpFFFFrq) __name__ __module__ __qualname____doc__r classmethodrrrrrrrrrrjs2 F   rj)FusedLayerNormzYBetter speed can be achieved with apex installed from https://www.github.com/nvidia/apex.cs&eZdZdfdd ZddZZS) BertLayerNormrqcs<tt|tt||_tt||_ ||_ dS)z[Construct a layernorm module in the TF style (epsilon inside the square root). N) superrrr Parameterronesr0zerosr3variance_epsilon)rreps __class__rrrs zBertLayerNorm.__init__cCsN|jddd}||djddd}||t||j}|j||jS)NrT)keepdimr )r powrrrr0r3)rrdusrrrforward$szBertLayerNorm.forward)rqrrrrr __classcell__rrrrrsrcs*eZdZdZfddZdddZZS)BertEmbeddingszLConstruct the embeddings from word, position and token_type embeddings. cstt|t|j|j|_t|j|j|_ t|j |j|_ |j |_ |j |_ |j|_t|j|jd|_t|j|_dSNr)rrrr Embeddingr~rword_embeddingsrposition_embeddingsrtoken_type_embeddingsrrrrr LayerNormDropoutrdropoutrrrrrr/s"zBertEmbeddings.__init__Nc Cs@|d}tj|tj|jd}|d|}|dur t|}||}| |}| |}|j sc|||}|j rB|j sB|}|} |j rM|}||}|j rb|j r]|}n<|| }n6|||}|j ry|j sy|}|} |j r|}||}|j r|j r|}n|| }||}|S)Nr)dtypedevicer)sizerarangelongrr! expand_as zeros_likerrrrrrhalftypefloatrr) r input_idstoken_type_ids seq_length position_idswords_embeddingsrr embeddings previous_typerrrrEsL                zBertEmbeddings.forwardrfrrrrrrrrrrrr+s rcs,eZdZfddZddZddZZS)BertSelfAttentioncstt||j|jdkrtd|j|jf|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_dS)NrzLThe hidden size (%d) is not a multiple of the number of attention heads (%d))rrrrrrrJattention_head_size all_head_sizerLinearqueryrrrrrrrrrrrs  zBertSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)Nrrr r)rrrviewpermute)rrd new_x_shaperrrtranspose_for_scoress  z&BertSelfAttention.transpose_for_scorescCs||}||}||}||}||}||}t||dd} | t|j } | |} t j dd| } | | } | } t| |} | dddd} | dd|jf} | j| } | S)Nrdimrr rr)rrrrrmatmulrKrrrrSoftmaxrrr contiguousrrr)r hidden_statesr"mixed_query_layermixed_key_layermixed_value_layer query_layer key_layer value_layerattention_scoresattention_probsr context_layernew_context_layer_shaperrrrs.          zBertSelfAttention.forward)rrrrrrrrrrrrps rc$eZdZfddZddZZS)BertSelfOutputcstt|t|dr|jrtd|j|jd}ntd|jd}t j |j |j dd|_ |j |_ t|j |jd|_t |j|_dSNr)r rrr Tr3r)rrrhasattrrrrrrrrrdenserrrrrrrrr init_methodrrrr$  zBertSelfOutput.__init__cCP||}||}||}|}|jr|}||}|jr&||}|Srfrrrrrrrr input_tensorln_inputrrrrr    zBertSelfOutput.forwardrrrrrr rcr) BertAttentioncs&tt|t||_t||_dSrf)rr rrrrrrrrrrs zBertAttention.__init__cCs|||}|||}|Srfr)rrr" self_outputattention_outputrrrrs  zBertAttention.forwardrrrrrr s  r cr)BertIntermediatecsLtt|tj|j|jdd|_t|j t r t |j |_ dS|j |_ dS)NTr) rr rrrrrrrvrrwACT2FNintermediate_act_fnrrrrrs  zBertIntermediate.__init__cCs||}||}|Srf)rrrrrrrrs  zBertIntermediate.forwardrrrrrr s r cr) BertOutputcstt|t|dr|jrtd|j|jd}ntd|jd}t j |j |j dd|_ |j|_t|j |jd|_t |j|_dSr)rrrrrrrrrrrrrrrrrrrrrrrrrrrzBertOutput.__init__cCrrfrrrrrrrzBertOutput.forwardrrrrrrrrcr) BertLayercs0tt|t||_t||_t||_dSrf) rrrr  attentionr  intermediaterrrrrrr&s  zBertLayer.__init__cCs&|||}||}|||}|Srf)rrr)rrr"r intermediate_output layer_outputrrrr,s   zBertLayer.forwardrrrrrr$ rcs*eZdZfddZ  dddZZS) BertEncodercs2tt|tfddtjD|_dS)Ncsg|]}tqSr)r)r+_rrr :sz(BertEncoder.__init__..)rrrr ModuleListrangerlayerrrrrr5s zBertEncoder.__init__TFc sg}fdd}|r.d}tj}d} ||kr-t|||| ||d}|| 7}||ksntjD]\} } | ||}|rC||q3|rH|rM|||S)Ncsfdd}|S)Ncs2j}|d}|D] }|||d}q |S)Nrr)r)inputslayersx_r)endrstartrrcustom_forwardNs z;BertEncoder.forward..custom..custom_forwardr)r"r!r#r)r!r"rcustomLsz#BertEncoder.forward..customrr)rIrr checkpoint enumeraterB) rrr"output_all_encoded_layerscheckpoint_activationsall_encoder_layersr$r]r chunk_lengthi layer_modulerrrrEs,      zBertEncoder.forward)TFrrrrrr3s  rcr) BertPoolercs.tt|t|j|j|_t|_dSrf) rr-rrrrrTanh activationrrrrroszBertPooler.__init__cCs(|dddf}||}||}|S)Nr)rr/)rrfirst_token_tensor pooled_outputrrrrts  zBertPooler.forwardrrrrrr-m r-cr)BertPredictionHeadTransformcs\tt|t|j|j|_t|jt rt |jn|j|_ t |j|j d|_|j|_dSr)rr3rrrrrrvrrwr transform_act_fnrrrrrrrrrs   z$BertPredictionHeadTransform.__init__cCsH||}||}|}|jr|}||}|jr"||}|Srf)rr4rrrr)rrrrrrrs    z#BertPredictionHeadTransform.forwardrrrrrr3}s  r3cr)BertLMPredictionHeadcsbttt|_tj|d|ddd_|j _ |j _ fdd}|_ d_ dS)NrrFrcsjr|S|Srf)rrrrrrconvert_to_typesz6BertLMPredictionHead.__init__..convert_to_type) rr5rr3 transformrrrdecoderrrtype_converter converted)rrbert_model_embedding_weightsr6rrrrs   zBertLMPredictionHead.__init__cCsR|jsd|_|jr|j|jr|jj|||}|||j }|S)NT) r:rr7rrrrr9r8r3rrrrrs  zBertLMPredictionHead.forwardrrrrrr5s r5cr)BertOnlyMLMHeadcstt|t|||_dSrf)rr<rr5 predictionsrrr;rrrrs zBertOnlyMLMHead.__init__cC||}|Srf)r=)rsequence_outputprediction_scoresrrrr zBertOnlyMLMHead.forwardrrrrrr<r2r<cr)BertOnlyNSPHeadcs"tt|t|jd|_dSNr )rrCrrrrseq_relationshiprrrrrszBertOnlyNSPHead.__init__cCr?rf)rE)rr1seq_relationship_scorerrrrrBzBertOnlyNSPHead.forwardrrrrrrCs rCcr)BertPreTrainingHeadscs.tt|t|||_t|jd|_dSrD) rrGrr5r=rrrrEr>rrrrs zBertPreTrainingHeads.__init__cCs@||}|jD] }|durq ||}q ||}||fSrf)r=rE parameterstype_as)rr@r1rAprFrrrrs   zBertPreTrainingHeads.forwardrrrrrrGrrGcsBeZdZdZfddZddZe      d d d ZZS) PreTrainedBertModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. cs:tt|t|tstd|jj|jj||_ dS)NzParameter config in `{}(config)` should be an instance of class `BertConfig`. To create a model from a Google pretrained model use `model = {}.from_pretrained(PRETRAINED_MODEL_NAME)`) rrKrrvrjrr>rrr)rrrkwargsrrrrs   zPreTrainedBertModel.__init__cCs|t|tjtjfr|jjjd|jjdnt|t r'|j j |jj dt|tjr:|j dur<|j j dSdSdS)z! Initialize the weights. rr raN) rvrrrr0rPrrrrr3zero_fill_)rmodulerrrinit_bert_weightss  z%PreTrainedBertModel.init_bert_weightsNFrqc s|tvr t|} n|} zt| |d} Wnty,td|dt| YdSw| | kr:td| n td| | d} t j | rN| } n*t } td| | t| d }|| Wdn1sqwY| } t j | t}t|}||_||_||_||_td |||g|Ri| }durt j | t}t|g}g}D]$}d}d |vr|d d }d |vr|d d }|r||||qt||D] \}}||<qgggtdd dur _!dfdd |t"|drdnddt#dkr4t$d|j%j&t#dkrEt$d|j%j&| rMt'(| |S)a Instantiate a PreTrainedBertModel from a pre-trained model file or a pytorch state dict. Download and cache the pre-trained model file if needed. Params: pretrained_model_name: either: - a str with the name of a pre-trained model to load selected in the list of: . `bert-base-uncased` . `bert-large-uncased` . `bert-base-cased` . `bert-large-cased` . `bert-base-multilingual-uncased` . `bert-base-multilingual-cased` . `bert-base-chinese` - a path or url to a pretrained model archive containing: . `bert_config.json` a configuration file for the model . `pytorch_model.bin` a PyTorch dump of a BertForPreTraining instance cache_dir: an optional path to a folder in which the pre-trained models will be cached. state_dict: an optional state dictionary (collections.OrderedDict object) to use instead of Google pre-trained models *inputs, **kwargs: additional input for the specific Bert class (ex: num_labels for BertForSequenceClassification) ) cache_dirzModel name '{}' was not found in model name list ({}). We assumed '{}' was a path or url but couldn't find any file associated to this path or url.z, Nzloading archive file {}z(loading archive file {} from cache at {}z)extracting archive file {} to temp dir {}zr:gzzModel config {}r/r0r2r3 _metadatac shdurin |ddi}|||d|jD]\}}|dur1|||dq dS)NrT.)get_load_from_state_dict_modulesr|)rOprefixlocal_metadatarYchild error_msgsloadmetadata missing_keys state_dictunexpected_keysrrr]ls  z1PreTrainedBertModel.from_pretrained..loadbertzbert.)rXrz7Weights of {} not initialized from pretrained model: {}z0Weights from pretrained model not used in {}: {})rS))PRETRAINED_MODEL_ARCHIVE_MAPrFileNotFoundErrorloggererrorr>rFkeysinfor;r<isdirtempfilemkdtemptarfilerx extractall CONFIG_NAMErjrrrrr WEIGHTS_NAMErr]replacerBrCpoprHrrRrrIr:rrshutilrmtree)rpretrained_model_namer`rQrrrrrrL archive_fileresolved_archive_filetempdirserialization_dirarchive config_filerrQ weights_pathold_keysnew_keysrnew_keyold_keyrr[rfrom_pretraineds!                  z#PreTrainedBertModel.from_pretrained)NNFFrqF) rrrrrrPrrrrrrrrKs  rKcs2eZdZdZfddZ    d ddZZS) BertModela BERT model ("Bidirectional Embedding Representations from a Transformer"). Params: config: a BertConfig class instance with the configuration to build a new model Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `output_all_encoded_layers`: boolean which controls the content of the `encoded_layers` output as described below. Default: `True`. Outputs: Tuple of (encoded_layers, pooled_output) `encoded_layers`: controled by `output_all_encoded_layers` argument: - `output_all_encoded_layers=True`: outputs a list of the full sequences of encoded-hidden-states at the end of each attention block (i.e. 12 full sequences for BERT-base, 24 for BERT-large), each encoded-hidden-state is a torch.FloatTensor of size [batch_size, sequence_length, hidden_size], - `output_all_encoded_layers=False`: outputs only the full sequence of hidden-states corresponding to the last attention block of shape [batch_size, sequence_length, hidden_size], `pooled_output`: a torch.FloatTensor of size [batch_size, hidden_size] which is the output of a classifier pretrained on top of the hidden state associated to the first character of the input (`CLF`) to train on the Next-Sentence task (see BERT's paper). Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = modeling.BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> model = modeling.BertModel(config=config) >>> all_encoder_layers, pooled_output = model(input_ids, token_type_ids, input_mask) cs>tt||t||_t||_t||_| |j dSrf) rrrrrrencoderr-poolerapplyrPrrrrrs    zBertModel.__init__NTFc Cs|dur t|}|durt|}|dd}|jt|jjd}d|d}| ||}|j||||d}|d} |j D] } | durKqD| | } | | } |rZ|r^|d}|| fS)Nrr )rragr'r(r) r ones_likerr!tonextrrHrrrrI) rrrr"r'r(r&embedding_outputencoded_layersr@rJr1rrrrs4      zBertModel.forward)NNTFrrrrrr * rc4eZdZdZfddZ     dddZZS) BertForPreTrainingaz BERT model with pre-training heads. This module comprises the BERT model followed by the two pre-training heads: - the masked language modeling head, and - the next sentence classification head. Params: config: a BertConfig class instance with the configuration to build a new model. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `masked_lm_labels`: masked language modeling labels: torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss is only computed for the labels set in [0, ..., vocab_size] `next_sentence_label`: next sentence classification loss: torch.LongTensor of shape [batch_size] with indices selected in [0, 1]. 0 => next sentence is the continuation, 1 => next sentence is a random sentence. Outputs: if `masked_lm_labels` and `next_sentence_label` are not `None`: Outputs the total_loss which is the sum of the masked language modeling loss and the next sentence classification loss. if `masked_lm_labels` or `next_sentence_label` is `None`: Outputs a tuple comprising - the masked language modeling logits of shape [batch_size, sequence_length, vocab_size], and - the next sentence classification logits of shape [batch_size, 2]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> model = BertForPreTraining(config) >>> masked_lm_logits_scores, seq_relationship_logits = model(input_ids, token_type_ids, input_mask) c>tt||t||_t||jjjj|_ | |j dSrf) rrrrrbrGrrr0rrrPrrrrrs   zBertForPreTraining.__init__NFcCs|j|||d|d\}}|||\} } |durE|durEtdd} | | d|jj|d} | | dd|d} | | }|S| | fSNFrr ignore_indexr )rbrr rrr~r)rrrr"masked_lm_labelsnext_sentence_labelr(r@r1rArFloss_fctmasked_lm_lossnext_sentence_loss total_lossrrrrs.  zBertForPreTraining.forwardNNNNFrrrrrrs 0 rc2eZdZdZfddZ    dddZZS) BertForMaskedLMa BERT model with the masked language modeling head. This module comprises the BERT model followed by the masked language modeling head. Params: config: a BertConfig class instance with the configuration to build a new model. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `masked_lm_labels`: masked language modeling labels: torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss is only computed for the labels set in [0, ..., vocab_size] Outputs: if `masked_lm_labels` is not `None`: Outputs the masked language modeling loss. if `masked_lm_labels` is `None`: Outputs the masked language modeling logits of shape [batch_size, sequence_length, vocab_size]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> model = BertForMaskedLM(config) >>> masked_lm_logits_scores = model(input_ids, token_type_ids, input_mask) crrf) rrrrrbr<rrr0rrrPrrrrras   zBertForMaskedLM.__init__NFc CsX|j|||d|d\}}||}|dur*tdd} | |d|jj|d} | S|S)NFrrr)rbrr rrr~) rrrr"rr(r@rrArrrrrrhs   zBertForMaskedLM.forwardNNNFrrrrrr8s ( rcr) BertForNextSentencePredictiona7 BERT model with next sentence prediction head. This module comprises the BERT model followed by the next sentence classification head. Params: config: a BertConfig class instance with the configuration to build a new model. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `next_sentence_label`: next sentence classification loss: torch.LongTensor of shape [batch_size] with indices selected in [0, 1]. 0 => next sentence is the continuation, 1 => next sentence is a random sentence. Outputs: if `next_sentence_label` is not `None`: Outputs the total_loss which is the sum of the masked language modeling loss and the next sentence classification loss. if `next_sentence_label` is `None`: Outputs the next sentence classification logits of shape [batch_size, 2]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> model = BertForNextSentencePrediction(config) >>> seq_relationship_logits = model(input_ids, token_type_ids, input_mask) cs4tt||t||_t||_||jdSrf) rrrrrbrCrrrPrrrrrs  z&BertForNextSentencePrediction.__init__NFc CsT|j|||d|d\}}||}|dur(tdd} | |dd|d} | S|Sr)rbrr r) rrrr"rr(rr1rFrrrrrrs    z%BertForNextSentencePrediction.forwardrrrrrrrs )rc4eZdZdZd fdd Z    d ddZZS) BertForSequenceClassificationaBERT model for classification. This module is composed of the BERT model with a linear layer on top of the pooled output. Params: `config`: a BertConfig class instance with the configuration to build a new model. `num_labels`: the number of classes for the classifier. Default = 2. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `labels`: labels for the classification output: torch.LongTensor of shape [batch_size] with indices selected in [0, ..., num_labels]. Outputs: if `labels` is not `None`: Outputs the CrossEntropy classification loss of the output with the labels. if `labels` is `None`: Outputs the classification logits of shape [batch_size, num_labels]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> num_labels = 2 >>> model = BertForSequenceClassification(config, num_labels) >>> logits = model(input_ids, token_type_ids, input_mask) r cNtt||||_t||_t|j|_ t |j ||_ | |jdSrf)rrr num_labelsrrbrrrrrr classifierrrPrrrrrrrs  z&BertForSequenceClassification.__init__NFc Cs\|j|||d|d\}}||}||}|dur,t} | |d|j|d} | S|SNFrr)rbrrr rr) rrrr"labelsr(rr1logitsrlossrrrrs  z%BertForSequenceClassification.forwardr rrrrrrrs+ rcr) BertForMultipleChoicea] BERT model for multiple choice tasks. This module is composed of the BERT model with a linear layer on top of the pooled output. Params: `config`: a BertConfig class instance with the configuration to build a new model. `num_choices`: the number of classes for the classifier. Default = 2. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, num_choices, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, num_choices, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, num_choices, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `labels`: labels for the classification output: torch.LongTensor of shape [batch_size] with indices selected in [0, ..., num_choices]. Outputs: if `labels` is not `None`: Outputs the CrossEntropy classification loss of the output with the labels. if `labels` is `None`: Outputs the classification logits of shape [batch_size, num_labels]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[[31, 51, 99], [15, 5, 0]], [[12, 16, 42], [14, 28, 57]]]) >>> input_mask = torch.LongTensor([[[1, 1, 1], [1, 1, 0]],[[1,1,0], [1, 0, 0]]]) >>> token_type_ids = torch.LongTensor([[[0, 0, 1], [0, 1, 0]],[[0, 1, 1], [0, 0, 1]]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> num_choices = 2 >>> model = BertForMultipleChoice(config, num_choices) >>> logits = model(input_ids, token_type_ids, input_mask) csHtt||t||_t|j|_t |j d|_ | |j dS)Nr)rrrrrbrrrrrrrrrPrrrrr>s  zBertForMultipleChoice.__init__NFcCs|jdd\}}|d|d}|d|d} |d|d} |j|| | d|d\} } || } || } | d|}|durNt}|||}|S|S)Nr rFr)rLreshaperrbrrr )rrrr"rr( batch_size num_choicesflat_input_idsflat_token_type_idsflat_attention_maskrr1rreshaped_logitsrrrrrrEs.    zBertForMultipleChoice.forwardrrrrrrrrrcr) BertForTokenClassificationa BERT model for token-level classification. This module is composed of the BERT model with a linear layer on top of the full hidden state of the last layer. Params: `config`: a BertConfig class instance with the configuration to build a new model. `num_labels`: the number of classes for the classifier. Default = 2. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `labels`: labels for the classification output: torch.LongTensor of shape [batch_size] with indices selected in [0, ..., num_labels]. Outputs: if `labels` is not `None`: Outputs the CrossEntropy classification loss of the output with the labels. if `labels` is `None`: Outputs the classification logits of shape [batch_size, sequence_length, num_labels]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> num_labels = 2 >>> model = BertForTokenClassification(config, num_labels) >>> logits = model(input_ids, token_type_ids, input_mask) r crrf)rrrrrrbrrrrrrrrrPrrrrrs   z#BertForTokenClassification.__init__NFc Cs|j|||d|d\}}t ||}Wdn1s"wY||}|durBt} | |d|j|d} | S|Sr) rbrget_cuda_rng_trackerforkrrr rr) rrrr"rr(r@rrrrrrrrs    z"BertForTokenClassification.forwardrrrrrrrrcs+rcr) BertForQuestionAnsweringa BERT model for Question Answering (span extraction). This module is composed of the BERT model with a linear layer on top of the sequence output that computes start_logits and end_logits Params: `config`: a BertConfig class instance with the configuration to build a new model. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `start_positions`: position of the first token for the labeled span: torch.LongTensor of shape [batch_size]. Positions are clamped to the length of the sequence and position outside of the sequence are not taken into account for computing the loss. `end_positions`: position of the last token for the labeled span: torch.LongTensor of shape [batch_size]. Positions are clamped to the length of the sequence and position outside of the sequence are not taken into account for computing the loss. Outputs: if `start_positions` and `end_positions` are not `None`: Outputs the total_loss which is the sum of the CrossEntropy loss for the start and end token positions. if `start_positions` or `end_positions` is `None`: Outputs a tuple of start_logits, end_logits which are the logits respectively for the start and end position tokens of shape [batch_size, sequence_length]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> model = BertForQuestionAnswering(config) >>> start_logits, end_logits = model(input_ids, token_type_ids, input_mask) cs:tt||t||_t|jd|_| |j dSrD) rrrrrbrrr qa_outputsrrPrrrrrs  z!BertForQuestionAnswering.__init__NFcCs|j|||d|d\}}||} | jddd\} } | d} | d} |durn|durnt|dkr9|d}t|dkrF|d}| d} |d| |d| t| d} | | |}| | |}||d}|S| | fS) NFrrrrrrr )rbrrDsqueezerIrclamp_r )rrrr"start_positions end_positionsr(r@rr start_logits end_logits ignored_indexr start_lossend_lossrrrrrs2            z BertForQuestionAnswering.forwardrrrrrrrs -r)Ir __future__rrrrrloggingrr;rrrlrjryrtorch.nn.functionalr functionalFdata_utils.file_utilsr megatron_utilrtorch.nnr rrr' getLoggerrresetLevelINFOrcrnroTF_WEIGHTS_NAMEr`rerhrir objectrj#apex.normalization.fused_layer_normrrr9r:Modulerrrr r rrrr-r3r5r<rCrGrKrrrrrrrrrrrrs      ?f E;'':+  _VHHKPT