o }oiZ#@sddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZddlmZddlmZmZdd lmZmZmZdd lmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&dZ'ej(j)e'ddej*ej+fddZ,ddej-dddddddf de.de.deej/dee.d e.d!e0d"e.d#e.d$e.d%ee1ej*e dej*ej2fd&d'Z3ej(j)ee'd(dd)ddd*efd+ee4d,e4d"e.d#e.d-e0d.edej5fd/d0Z6d1ej5dej5fd2d3Z7dS)4)CallableOptionalN)Callback)DistributedDataParallelConfig) lightning)pretrain)MockDataModule) GPTConfig175BGPTModel) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing) bf16_mixed)0userbuffers_bf16_h100_h12288_tp4_mbs1_seqlen2048)GarbageCollectionCallback)MegatronCommOverlapCallback)TimingCallback gpt3_175bnamereturncCstjtttdS)a^ Factory function to create a GPT3 175B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the GPT3 175B model. Examples: CLI usage: $ nemo llm pretrain model=gpt3_175b ... Python API usage: >>> model_config = model() >>> print(model_config) )config)runConfigr r rrZ/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/gpt3_175b.pymodel)srT@i{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csbtjtj||||||dddtjtddddddd } tjtjdd| |ddd||t| d d d } | S) a= Configure the NeMo Lightning Trainer for GPT3 175B model. This function sets up the distributed training strategy optimized for the large 175B model. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=gpt3_175b ... Python API usage: >>> trainer_config = trainer(num_nodes=64, num_gpus_per_node=8) >>> print(trainer_config) Note: This configuration uses extensive parallelism to handle the large model size efficiently. T)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collective) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpur!2 Fi) acceleratoraccumulate_grad_batchesr,deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr+r)pluginsstrategyuse_distributed_samplerval_check_interval)rrnlMegatronStrategyrTrainerr) r#r$r%r&r'r(r)r*r+r,rGtrainerrrrrM<sJ+rM)targetrdefaultFdirrperformance_modefnc Csbtj|tt||ttgdtjtddddt||t|ddt ddt d }|r/t |}|S) a_ Create a pre-training recipe for GPT3 175B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. performance_mode (bool): If true, enables optimizations for maximum performance. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory gpt3_175b $ nemo llm pretrain --factory "gpt3_175b(num_nodes=64, name='my_175b_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="gpt3_175b_pretrain", num_nodes=64) >>> print(recipe) Note: This recipe is optimized for the large 175B model and requires significant computational resources. )r)r*r,i) seq_lengthglobal_batch_sizemicro_batch_sizer)rPrr g9̗?)max_lr)rrMdatalogoptimresume) rPartialrrMrrrr r rr "pretrain_performance_optimizations)rPrr)r*rQrRreciperrrpretrain_recipes & r_r^cCs`|jjsg|j_tjtddd}tjtdtdddd}|jj||gd|jj_ d|j j _ |S)a Create a performance-optimized pre-training recipe for GPT3 175B model. This method enables performance optimizations that may not be suitable for all use cases. It builds upon the standard pre-training recipe and adds additional performance enhancements. Args: recipe (run.Partial): Base pre-train recipe to which performance optimizations will be added Returns: run.Partial: Partial configuration for performance-optimized pre-training. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. d)gc_interval_traingc_interval_valTr=F)tp_comm_overlaptp_comm_overlap_cfgdefer_embedding_wgrad_computewgrad_deferral_limit(overlap_param_gather_with_optimizer_step) rMr,rrrrrextendrFr.rZruse_precision_aware_optimizer)r^garbage_collection_callbackmcomm_overlap_callbackrrrr]s.   r])8typingrrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrJnemo.collections.llm.apir"nemo.collections.llm.gpt.data.mockrnemo.collections.llm.gpt.modelr r (nemo.collections.llm.recipes.log.defaultr r r 'nemo.collections.llm.recipes.optim.adamr6nemo.collections.llm.recipes.precision.mixed_precisionr;nemo.collections.llm.recipes.tp_overlap_configs.userbuffersr3nemo.lightning.pytorch.callbacks.garbage_collectionr6nemo.lightning.pytorch.callbacks.megatron_comm_overlaprnemo.utils.exp_managerrNAMEclifactoryrLightningModulerbfloat16intdtypeboollistrLrMstrr\r_r]rrrrs                T 9