o wiJx@sddlZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlZddlZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lm Z dd lm!Z!dd l"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+ddl,m-Z-ddl.m/Z/e/d\Z0Z1dZ2zddl3Z3Wn e4ydZ2Ynwerddl5m6Z6ddl7m8Z8d@de9e:ej;ffddZdAddd!e e?defd"d#Z@dddefd$d%ZAdAddd!e e?defd&d'ZBdAddd!e e?de efd(d)ZCdedejDfd*d+ZEde9e:efdejDfd,d-ZFeGd.ddee&jGZHeGd/d0d0eHZIeGd1d2d2eHZJeGd3d4d4eHZKeGd5d6d6eHZLeGd7d8d8eHZMeGd9d:d:eHZNGd;d<dZRgd?ZSdS)BN) dataclass)partial) TYPE_CHECKINGAnyCallableLiteralOptionalUnion)GPTInferenceWrapper)InferenceWrapperConfig)GPTModel)OptimizerConfig) ModuleSpec)TransformerConfig)get_batch_on_this_cp_rank)nn)fn)get_vocab_sizeio)MaskedTokenLossReduction)MegatronOptimizerModuleOptimizerModule)logging) safe_importtransformer_engineTF)GenerationConfig) TokenizerSpecreturnc Csddlm}t|}t|trt|dkr|d}n|}t}t}|dd|vr:|d|d|d|s@|rE| d| rN| d i}| D] \}} ||vre| j d d ||<qT||vrp| ||<qTd ||<qTt|} | S) a-Process a single batch of data from the dataloader iterator. This function handles the data loading step for GPT models, managing pipeline parallelism by distributing data appropriately across pipeline stages. Args: dataloader_iter: Iterator over the dataloader use_mtp: Whether the Multi-Token Prediction Module is used. Input needs to be passed into the last ppieline stage if mtp is used. Returns: dict[str, torch.Tensor]: Processed batch with required tensors moved to appropriate devices rparallel_stateattention_mask cu_seqlenscu_seqlens_argmin max_seqlen)tokens position_ids)labels loss_maskT) non_blockingN) megatron.corernext isinstancetuplelensetaddis_pipeline_first_stageupdateis_pipeline_last_stageitemscudacpur) dataloader_iteruse_mtprbatch_batchrequired_device_keysrequired_host_keys_batch_required_keyskeyvaloutputrA`/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/base.py gpt_data_step9s2          rCcCs\|d|d|dd}d|vrtsJdn|d|d<d|vr't||d<|d i|S) aExecute a forward step for the GPT model. This function prepares the arguments needed for the model's forward pass and handles both normal and packed sequence processing. Args: model: The GPT model batch: The input batch containing tokens, positions, and other required inputs Returns: torch.Tensor: Output tensor from the model forward pass r%r&r') input_idsr&r'r!zThe dataloader did not provide an attention mask, however Transformer Engine was not detected. This requires Transformer Engine's implementation of fused or flash attention.r"packed_seq_paramsNrA)HAVE_TEget_packed_seq_params)modelr9 forward_argsrArArBgpt_forward_stepqs  rJconfig GPTConfigcCsZddlm}|j|j|jt|jo|jdud}t|dddur%|j|d<|j di|S)zCreate a Transformer Engine layer specification based on the provided config. Args: config: GPT configuration object Returns: ModuleSpec: Module specification for Transformer Engine based layers rgpt_layer_specsN) num_expertsmoe_grouped_gemm qk_layernormfp8use_transformer_engine_op_fuseruse_te_op_fuserrA) megatron.core.models.gptrNnum_moe_expertsrPrQboolrRgetattrrS*get_gpt_layer_with_transformer_engine_spec)rKrNkwargsrArArBtransformer_engine_layer_specs  r[vp_stagecCsddlm}|||dS)zCreate a full Transformer Engine layer specification with autocast support. Args: config: GPT configuration object Returns: ModuleSpec: Module specification for full TE layers r)#get_gpt_full_te_layer_autocast_spec)transformer_configr\)Vnemo.collections.nlp.models.language_modeling.megatron.gpt_full_te_layer_autocast_specr])rKr\r]rArArB"transformer_engine_full_layer_specs r`cCs&ddlm}|j|j|j|j|jdS)zCreate a local layer specification without Transformer Engine. Args: config: GPT configuration object Returns: ModuleSpec: Module specification for local implementation layers rrM)rOrPrQ normalization)rUrNget_gpt_layer_local_specrVrPrQra)rKrNrArArBlocal_layer_specs rccCs&tr|jr t||dSt|St|S)a%Determine the most appropriate layer specification based on availability. Uses Transformer Engine specs if available, otherwise falls back to local implementation. Args: config: GPT configuration object Returns: ModuleSpec: The selected module specification r\)rF&use_transformer_engine_full_layer_specr`r[rc)rKr\rArArBdefault_layer_specs   rfcCslt|ddr4ddlm}t|jtr)dt|jjvr#|j||d}n ||}n|j}|||t |dSdS)zPass in the MTP block spec if model has MTP layers. Args: config: GPT configuration object Returns: ModuleSpec: The MTP module specification mtp_num_layersNr)get_gpt_mtp_block_specr\rd)use_transformer_enginer\) rX(megatron.core.models.gpt.gpt_layer_specsrhr,transformer_layer_specrinspect signature parametersrF)rKr\rhspecrArArBmtp_block_specs   rpcCs|jrtjS|jr tjStjS)zExtract the appropriate torch dtype from a Megatron Core configuration. Args: config: Megatron Core Transformer configuration Returns: torch.dtype: The appropriate torch dtype (float16, bfloat16, or float32) )fp16torchfloat16bf16bfloat16floatrKrArArBtorch_dtype_from_mcore_configs  rxcCs"|drtjS|drtjStjS)zExtract the appropriate torch dtype from a dictionary configuration. Args: config: Dictionary containing configuration parameters Returns: torch.dtype: The appropriate torch dtype (float16, bfloat16, or float32) rqrt)rrrsrurvrwrArArBtorch_dtype_from_dict_configs  ryc@sheZdZUdZdZeed<dZeed<dZeed<dZ e ed<d Z e d ed <d Z e ed <dZeed<dZeeed<dZe ed<dZeed<dZeed<dZeed<eZeed<dZeed<dZeed<dZeed<dZeed<eZeee dgeffed<e!Z"e ed<e#Z$e ed<dZ%eded <dZ&ee ed!<dZ'eee(e)e(e*ffed"<d'd(d%d&Z+dS))rLzConfiguration class for GPT models. Extends TransformerConfig with additional parameters specific to GPT models and provides utility methods for model configuration. Ffp16_lm_cross_entropyTparallel_output#share_embeddings_and_output_weightsmake_vocab_size_divisible_bylearned_absolute)rropeposition_embedding_typei' rotary_baseg?rotary_percentNseq_len_interpolation_factori seq_lengthattention_softmax_in_fp32masked_softmax_fusioncross_entropy_loss_fusiongradient_accumulation_fusiondeallocate_pipeline_outputs#scatter_embedding_sequence_paralleltp_only_amax_redrerkforward_step_fn data_step_fnrgeneration_config vocab_sizetp_comm_overlap_cfgr MCoreGPTModelcCs|jr|jdkrtsJdt|ddsJd|j}t|ddp&t|dd}|t|dd p3t|d d d uO}|p@t|d d d u}|rU|sU|j}|j||d ksUJd d d l} d dlm } |pbd }|j } t | t s~d| | jvrz| ||d} n| |} |jd ur|j} |d urtd| d|jd| |jdnt||j|j} tj} |jrttjdd} d| tjjvrdt||di}ni}| 6t|f| | |j|j|j|j |j!|j"|j#|j$|p| j%d|d|p| j&d|d|j'|d|}Wd n 1swYtrq|j(rq| )dkr9t*|+D]\}}|d kr(qt,|dr7| -}|.|q| /dkrqtj01}|2D]'}t*|+D]\}}|d kr\qQt,|drm|3| 4| 5|qQqI|S)aConfigure and instantiate a Megatron Core GPT model based on this configuration. Args: tokenizer: Tokenizer used with the model pre_process: Whether to include pre-processing in the model, defaults to first pipeline stage post_process: Whether to include post-processing in the model, defaults to last pipeline stage vp_stage: Virtual pipeline stage Returns: MCoreGPTModel: Configured Megatron Core GPT model instance full_iterationz.Transformer Engine is required for cudagraphs.use_te_rng_trackerFzmTransformer engine's RNG tracker is required for cudagraphs, it can be enabled with use_te_rng_tracker=True'.'account_for_embedding_in_pipeline_split"account_for_loss_in_pipeline_split"num_layers_in_first_pipeline_stageN!num_layers_in_last_pipeline_stagepipeline_model_parallel_layoutrzLMake sure the number of model chunks is the same across all pipeline stages.rr\rdzUse preset vocab_size: z, original vocab_size: z, dummy tokens: .meta)devicerp)ignore_virtualr\)rkrmax_sequence_lengthrzr{r|rrrr pre_process post_processrr\set_tensor_parallel_groupset_context_parallel_group)6enable_cuda_graphcuda_graph_scoperFrX$virtual_pipeline_model_parallel_sizepipeline_model_parallel_size num_layersrlr*rrkr,rrmrnrrinforr~ contextlib nullcontextinit_model_with_meta_devicerrrrr__init__rprrzr{r|rrrrr1r3rre$get_tensor_model_parallel_world_size enumeratemoduleshasattrget_tensor_model_parallel_grouprget_context_parallel_world_sizer5Streamget_model_module_listrget_context_parallel_group!get_context_parallel_global_ranks)self tokenizerrrr\vp_sizeis_pipeline_asymmetricis_flexible_pp_layoutp_sizerlrrkrmodel_init_device_contextrZrHindexchildtp_group cp_streammodulerArArBconfigure_model=s                zGPTConfig.configure_model)NNN)rr),__name__ __module__ __qualname____doc__rzrW__annotations__r{r|r~intrrrrrvrrrrrr_grad_accum_fusion_availablerrrrrerfrkr rrrJrrCrrrrstrdictrrrArArArBrLs4                   c@reZdZUdZdZeed<dZeed<dZeed<dZ eed <dZ eed <d Z e ed <d Z e ed <d Ze ed<dS) GPTConfig126MzConfiguration for a 126M parameter GPT model. Predefined configuration for a small GPT model with 12 layers, 768 hidden size, and 12 attention heads. r ri hidden_sizei ffn_hidden_sizenum_attention_headsTbias_activation_fusionbias_dropout_add_fusionreNrrrrrrrrrrrrrWrrerArArArBr        rc@reZdZUdZdZeed<dZeed<dZeed<dZ eed <d Z eed <d Z e ed <d Z e ed<d Ze ed<dS) GPTConfig5BzConfiguration for a 5B parameter GPT model. Predefined configuration for a medium-sized GPT model with 24 layers, 4096 hidden size, and 32 attention heads. rrrri@r rTrrreNrrArArArBrrrc@r) GPTConfig7BzConfiguration for a 7B parameter GPT model. Predefined configuration for a medium-sized GPT model with 32 layers, 4096 hidden size, and 32 attention heads. rrrrrri*rrTrrreNrrArArArBrrrc@r) GPTConfig20BzConfiguration for a 20B parameter GPT model. Predefined configuration for a large GPT model with 44 layers, 6144 hidden size, and 48 attention heads. rr,riri`r0rTrrreNrrArArArBrrrc@r) GPTConfig40BzConfiguration for a 40B parameter GPT model. Predefined configuration for a large GPT model with 48 layers, 8192 hidden size, and 64 attention heads. rrrri rir@rTrrreNrrArArArBrrrc@seZdZUdZdZeed<dZeed<dZeed<dZ eed <dZ eed <d Z e ed <d Z e ed <dZeed<dZeed<dZeed<dZeed<dS) GPTConfig175BzConfiguration for a 175B parameter GPT model. Predefined configuration for a massive GPT model with 96 layers, 12288 hidden size, and 96 attention heads. rr`ri0rirrghidden_dropoutattention_dropoutTrrrelayernorm_zero_centered_gammaN)rrrrrrrrrrrrrvrrrWrrerrArArArBrs           rc sLeZdZdZdddgfdedeededdeeej gej fdee f fd d Z d(d ee d dfd dZ     d)dejdejdeejdeejdeejd ejf ddZd eeejffddZd ejfddZd(d ejfddZd(d ejfddZ d*dejd e d!e d efd"d#Zed efd$d%Zed efd&d'ZZS)+r zGPT model implementation using Megatron Core and PyTorch Lightning. This class provides a high-level interface for training and using GPT models with proper integration with NeMo's infrastructure. NrKoptimrrmodel_transformmodel_context_managerscsVt||_||_|pttdddd|_|j|||_||_ d|_ d|_ dS)aInitialize the GPT model. Args: config: Configuration for the GPT model optim: Optional optimizer module tokenizer: Optional tokenizer specification model_transform: Optional function to transform the model after initialization model_context_managers: Optional list of context managers to apply when configuring and instantiating the model. g-C6?T)lruse_distributed_optimizerrwN) superrrKrrr rconnectrr_training_loss_reduction_validation_loss_reduction)rrKrrrr __class__rArBr/s   zGPTModel.__init__r\rcCsddlm}t|ds>t}|jD]}||q|jj|j |d|_ Wdn1s/wY||j |j ddSdS)zConfigure the underlying model if not already configured. This method ensures the model is instantiated from the configuration. r)restore_modelopt_staterrdN)trainer) )nemo.collections.llm.modelopt.model_utilsrrr ExitStackr enter_contextrKrrr_trainer)rr\rstackcmrArArBrLs      zGPTModel.configure_modelrDr&r!r' decoder_inputc Cs6|durd|ini}|j|||f|||d|} | S)aForward pass through the GPT model. Args: input_ids: Input token IDs position_ids: Position IDs for the input attention_mask: Optional attention mask labels: Optional labels for computing loss decoder_input: Optional decoder input inference_context: Optional parameters for inference packed_seq_params: Optional parameters for packed sequence processing Returns: torch.Tensor: Output tensor from the model NrE)rr'inference_context)r) rrDr&r!r'rrrE extra_kwargs output_tensorrArArBforwardas zGPTModel.forwardcCs |j|S)zProcess a batch of data from the dataloader. Args: dataloader_iter: Iterator over the dataloader Returns: dict[str, torch.Tensor]: Processed batch )rKr)rr7rArArB data_steps zGPTModel.data_stepcCs|j||S)zExecute a forward step using the provided batch. Args: batch: Input batch Returns: torch.Tensor: Output from the forward pass )rKr)rr9rArArB forward_steps zGPTModel.forward_stepcC ||S)zExecute a training step. Args: batch: Input batch batch_idx: Optional batch index Returns: torch.Tensor: Loss value rrr9 batch_idxrArArB training_steps zGPTModel.training_stepcCr)zExecute a validation step. Args: batch: Input batch batch_idx: Optional batch index Returns: torch.Tensor: Loss value rrrArArBvalidation_steps zGPTModel.validation_step params_dtype&inference_batch_times_seqlen_thresholdinference_max_seq_lengthcCs|j}|rt|tur nt|dd}|s|dust|tur"tdd}t|jdr/|jj}n|jdur9|jj}ntdt |jj ||||d}t ||}|S)aGet an inference wrapper for the model. Creates and configures a GPTInferenceWrapper around the model for efficient inference. Args: params_dtype: Data type for parameters inference_batch_times_seqlen_threshold: Threshold for optimizing inference inference_max_seq_length: Maximum sequence length for inference (prefill and decode) Returns: GPTInferenceWrapper: Wrapped model for inference rNz>Exact McoreGPTModel instance not found in the model structure.rzlUnable to find vocab size. Either pass in a tokenizer with vocab size, or set vocab size in the model config)rr r padded_vocab_sizer ) rtyperrX ValueErrorrrKrrr rr )rr r r  mcore_modelrinference_wrapper_configmodel_inference_wrapperrArArBget_inference_wrappers2       zGPTModel.get_inference_wrappercCs|jst|_|jS)zGet the loss reduction module for training. Returns: MaskedTokenLossReduction: Loss reduction module for training )rrrrArArBtraining_loss_reductionsz GPTModel.training_loss_reductioncCs|js tdd|_|jS)zGet the loss reduction module for validation. Returns: MaskedTokenLossReduction: Loss reduction module for validation T)r)rrrrArArBvalidation_loss_reductions z"GPTModel.validation_loss_reductionN)NNNNN)r )rrrrrLrrrrModulelistrrrrrTensorrrrrrrrdtyper rpropertyrrr __classcell__rArArrBr (sl  %    1 r cCszddlm}|d}|dd}dur|d|}n |dt|}d|vr2|dnd}|||||ddS) aGExtract packed sequence parameters from the batch. Creates and returns a PackedSeqParams object with appropriate parameters for packed sequence processing. Args: batch: Input batch containing packed sequence information Returns: PackedSeqParams: Parameters for packed sequence processing r)PackedSeqParamsr"r#Nr$thd) cu_seqlens_q cu_seqlens_kv max_seqlen_q max_seqlen_kv qkv_format)megatron.core.packed_seq_paramsrsqueezegetitemrrargmin)r9rr"r#r$rArArBrGs rG)r rLrCrJr[rc)Fr)Trrl dataclassesr functoolsrtypingrrrrrr lightning.pytorchpytorchLrrtorch.distributedJmegatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapperr Imegatron.core.inference.model_inference_wrappers.inference_wrapper_configr "megatron.core.models.gpt.gpt_modelr rmegatron.core.optimizerr $megatron.core.transformer.spec_utilsr,megatron.core.transformer.transformer_configrmegatron.core.utilsrrnemo.collections.llmrnemo.lightningrr nemo.lightning.megatron_parallelrnemo.lightning.pytorch.optimrr nemo.utilsrnemo.utils.import_utilsr_rFrfused_weight_gradient_mlp_cuda ImportError transformersr1nemo.collections.common.tokenizers.tokenizer_specrrrrrCrJr[rr`rcrfrprrxryIOMixinrLrrrrrrLightningModuleConnectorMixinFNMixinrG__all__rArArArBsx                     8!  Z #