o }oit @sddlmZddlmZmZmZmZmZmZddl m Z ddl Z ddl Z ddlmZmZmZddlmZddlmZddlmZddlmZdd lmZdd lmZmZdd l m!Z!dd l"m#Z#dd l$m%Z%m&Z&ddl'm(Z)ddl*m+Z+m,Z,ddl m-Z-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4m5Z5ddl6m7Z7m8Z8ddl9m:Z:m;Z;dZm?Z?ddl@mAZBWneCeDfydZJs*z"bert_data_step..) next isinstancetuplelensetaddr is_pipeline_first_stageis_pipeline_last_stageupdateitemsr)dataloder_iterbatch_batchoutputr6r4r7bert_data_step9s    rGmodelrDcCsX|d|d|d|dd}|jjdkr|d|d<d |vr%t||d <|d i|S) a This subsets the batch keys to the ones actually used by forward pass of the model, and then calls the model's forward pass. if "cu_seqsens" are defined in the batch, then the packed sequence parameters are also passed to the model for forward pass efficiency. r*r)r+r,) input_idsattention_mask lm_labelsr,rr- tokentype_ids cu_seqlenspacked_seq_paramsNr6)confignum_tokentypesget_packed_seq_params)rHrD forward_argsr6r6r7bert_forward_stepQs   rSrO BertConfigcCsR|j}|dks|dksJd|dtr|dkrtjStS|dkr&tjStS)z Return MCore layer spec based on the bert type. For bert_type == 'megatron', use mcore's default layer spec; For bert_type == 'huggingface', use Post-LayerNorm layer spec. megatron huggingfacezUnknown bert type z9, supported type for bert model is: megatron, huggingface) bert_typeHAVE_TEr$'bert_layer_with_transformer_engine_specrbert_layer_local_specr)rOrWr6r6r7default_layer_specgs r[c@s<eZdZUdZdZeed<dZeed<dZeed<dZ e ded <d Z e ed <d Z eed <dZeeed<dZe ed<dZeed<dZeed<dZdZe ed<eZeeedgeffed<eZeed<eZeed<dZe ded<dZ eed<dZ!eed<dZ"eed<dZ#eed<dZ$eed <d&d!ee d"d#fd$d%Z%dS)'rTzZ "Model config for BERT model. Adpated from megatron.core.models.bert.bert_model.BertModelFfp16_lm_cross_entropyTparallel_output#share_embeddings_and_output_weightslearned_absolute)r_ropeposition_embedding_typei' rotary_baseg?rotary_percentNseq_len_interpolation_factori seq_lengthattention_softmax_in_fp32masked_softmax_fusionmake_vocab_size_divisible_bytransformer_layer_specforward_step_fn data_step_fnrU)rUrVrW add_poolerbert_binary_head add_lm_headrPmask_vocab_padding_tokensvp_stager'&MCoreBertModelWrapperWithPostLNSupportcCs|j}|r|j}|j||dksJdddlm}|j}t|ts'||}|jdur4|j r1dnd|_|p7d}t did|j d|j d|d |jd |d t ||j|jd |jr\|n:dd |jd|jd|dd|jd|dd|jd|jd|jd|jd|jd|jd|j ddd|Sd |jd|jd|dd|jd|dd|jd|jd|jd|jd|jd|jd|j ddd|S)zConfigure the BERT Model. For bert_type == 'megatron', num_tokentypes in embedding is controlled by whether model has binary head. For bert_type == 'huggingface', tokentypes embedding is always added with num_tokentypes = 2. rzLMake sure the number of model chunks is the same across all pipeline stages.)r NrWrmrOrPrj vocab_size tokenizermax_sequence_length pre_processF)ignore_virtualrq post_processr\r]r^rarcrdadd_binary_headreturn_embeddingsrqr6)$virtual_pipeline_model_parallel_sizepipeline_model_parallel_size num_layers megatron.corer rjr:rrPrnrrrWrmr rtrirprer?r@r\r]r^rarcrd)selfrurqvp_sizep_sizer rjr6r6r7configure_models             zBertConfig.configure_modelN)&__name__ __module__ __qualname____doc__r\bool__annotations__r]r^rarrbintrcfloatrdrrerfrgdeallocate_pipeline_outputsrir[rjrrrrSrkrGrlrWrmrnrorPrprr6r6r6r7rT}s0                 c s\eZdZdZ ddedffdd Z     dd ed ed ed edef fdd ZZS)rra  This class is used for working with HF Bert Checkpoints. These checkpoints by default have post layer norm, while the vanilla mcore bert model does not support it. when bert_type is set to 'huggingface', it will initialize post layer norm BERT model. rUTNrur&c shtt|j|i|||_||_||_|jdks'|jdks'Jd|jdt|j|j|j |j |jdkr7dnd|j| ddd|_ |j rN|j |jj|jret|jj|jj|j|jj|jj|_|j rd|_|jjrxt|jj|j|_tj|jj|j|j|jjdd|j |j o|jd |_d|_|jrt |jjd |jj|jj!|_|j s|j r|"dSdS) NrUrVz