o wi$@s*ddlmZddlmZddlZddlZddlm Z ddl m Z ddl m ZddlmZmZmZmZmZmZddlmZmZded ejejfd d Z           d'dededeejdeedededededededed ed!ed"ed#ed$ee eje d ejej!f"d%d&Z"dS)()OptionalN)Callback)DistributedDataParallelConfig) lightning)Nemotron3Config4BNemotron3Config8BNemotron3Config22BNemotron4Config15BNemotron4Config340B NemotronModel) bf16_mixed fp16_mixedversionreturncCsd}|dkr tt}nW|dkrtt}nM|dkr tt}nC|dkr,tjtdd}n7|dkr8tjtd d}n+|d krBtt}n!|d krNtjtdd}n|d krZtjtd d}n |d krctt}|dusnJd|tjt|dS)a A function to create a Nemotron models. Args: version (str): The version of the Nemotron model to create. one of ["nemotron3_4b", "nemotron3_8b", "nemotron3_22b", "nemotron3_22b_16k", "nemotron3_22b_64k", "nemotron4_15b", "nemotron4_15b_16k", "nemotron4_15b_64k", "nemotron4_340b"]. Returns: run.Config[pl.LightningModule]: Configuration for the Nemotron model. N nemotron3_4b nemotron3_8b nemotron3_22bnemotron3_22b_16ki@) seq_lengthnemotron3_22b_64ki nemotron4_15bnemotron4_15b_16knemotron4_15b_64knemotron4_340bzInvalid version: )config)runConfigrrrr r r )rrrb/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/recipes/nemotron.pynemotron_model#s*      rF{ bf16-mixed tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps precisionaccumulate_grad_batcheslimit_test_batcheslimit_val_batcheslog_every_n_stepsval_check_interval callbackscCstjtj||||||ddddtjtddddddd }d}| dkr%t}n| dkr,t}tjtjd||| | | | ||||d|d }|S) a Configure the NeMo Lightning Trainer for Nemotron models. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. precision (str): Precision configuration, one of fp32, 16-mixed or bf16-mixed. accumulate_grad_batches (int): Number of steps per gradient accumulation. limit_test_batches (int): Limit the number of test batches. limit_val_batches (int): Limit the number of validation batches. log_every_n_steps (int): Log every n steps. val_check_interval (int): Run validation every N steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. T)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collective) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_include_optimizerckpt_async_saveckpt_parallel_loadddpNz16-mixedr$gpuF) acceleratorr7devicesr2r3r4r5r0r.pluginsstrategyuse_distributed_samplerr6)rrnlMegatronStrategyrr r Trainer)r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7rLprecision_plugintrainerrrrnemotron_trainerHsV,rS)r r!NNr!Fr!r"r#r$r!r%r%r&r'N)#typingrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrN'nemo.collections.llm.gpt.model.nemotronrrrr r r 6nemo.collections.llm.recipes.precision.mixed_precisionr r strrLightningModulerintdtypeboollistrPrSrrrrsz      &