o }oi#@sddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZmZmZdd lmZdd lmZdd lmZdd l m!Z!dZ"ej#j$e"ddej%ej&fddZ'ddej(ddddddddf de)de)deej*dee)de)de+d e)d!e)d"e)d#e)d$ee,ej%e dej%ej-fd%d&Z.ej#j$ee"d'dd(ddd)efd*ee/d+e/d!e)d"e)d,e+d-edej0fd.d/Z1d0ej0dej0fd1d2Z2dS)3)CallableOptionalN)Callback)DistributedDataParallelConfig) lightning)pretrain)MockDataModule) mixtral_8x22b) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing)MegatronCommOverlapCallback)MegatronTokenDropCallback)TimingCallbackmixtral_8x22b_64knamereturncCst}d|j_|S)an Factory function to create a Mixtral 8x22B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Mixtral 8x22B model. Examples: CLI usage: $ nemo llm pretrain model=mixtral_8x22b_64k ... Python API usage: >>> model_config = model() >>> print(model_config) )r modelconfig seq_length) model_configrb/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/mixtral_8x22b_64k.pyr%srTi{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelismexpert_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csntjtj|||||||dddtjtddddddd } tjtjdd| |ddd| |tjtjd d | d d d } | S)a Configure the NeMo Lightning Trainer for Mixtral 8x22B model. This function sets up the distributed training strategy optimized for the large Mixtral 8x22B model. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. expert_parallelism (int): Degree of expert parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=mixtral_8x22b_64k ... Python API usage: >>> trainer_config = trainer(num_nodes=16, num_gpus_per_node=8) >>> print(trainer_config) Note: This configuration uses extensive parallelism to handle the large model size efficiently. T)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collective) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelexpert_model_parallel_sizegradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpu2 z bf16-mixed) precisionFi) acceleratoraccumulate_grad_batchesr+deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr*r(pluginsstrategyuse_distributed_samplerval_check_interval)runConfignlMegatronStrategyrTrainerMegatronMixedPrecision) r!r"r#r$r%r&r'r(r)r*r+rItrainerrrrrR:sL-rR)targetrdefaultFdirrperformance_modefnc Csbtj|tt||ttgdtjtddddt||t|ddt dd t d }|r/t |}|S) a Create a pre-training recipe for Mixtral 8x22B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. performance_mode (bool): If true, enables optimizations for maximum performance. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory mixtral_8x22b_64k $ nemo llm pretrain --factory "mixtral_8x22b_64k(num_nodes=16, name='my_mixtral_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="mixtral_pretrain", num_nodes=16) >>> print(recipe) )r(r)r+rir=)rglobal_batch_sizemicro_batch_sizer)rUrr ga2U0*3?)max_lr)rrRdatalogoptimresume) rLPartialrrRrMrrr r r r "pretrain_performance_optimizations)rUrr(r)rVrWreciperrrpretrain_recipes# rbracCsP|jjtttjtdddgd|jj_d|jj_ d|jj_ d|j j _ |S)a Create a performance-optimized pre-training recipe for Mixtral 8x22B model. This method enables performance optimizations that may not be suitable for all use cases. It builds upon the standard pre-training recipe and adds additional performance enhancements. Args: recipe (run.Partial): Base pre-train recipe to which performance optimizations will be added Returns: run.Partial: Partial configuration for performance-optimized pre-training. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. FT)(overlap_param_gather_with_optimizer_stepalign_param_gatherr=r)rRr+extendrLrMrrrIr7r1r6r]ruse_precision_aware_optimizer)rarrrr`s    r`)3typingrrlightning.pytorchpytorchplnemo_runrLtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrNnemo.collections.llm.apir"nemo.collections.llm.gpt.data.mockrnemo.collections.llm.recipesr (nemo.collections.llm.recipes.log.defaultr r r 'nemo.collections.llm.recipes.optim.adamr 6nemo.lightning.pytorch.callbacks.megatron_comm_overlapr/nemo.lightning.pytorch.callbacks.moe_token_droprnemo.utils.exp_managerrNAMEclifactoryrMLightningModulerbfloat16intdtypeboollistrPrRstrr_rbr`rrrrs                W 4