o }oi@s>ddlmZddlZddlZddlmZddlmZddl m Z ddl m Z ddlmZmZmZddlmZdd lmZd Z    d$dededeejdeededededededeeejedeje jfddZejje j ed   d%de!de!dee!d ee!d!e!dededej"fd"d#Z#dS)&)OptionalN)Callback)DistributedDataParallelConfig) lightning)llm) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing) bf16_mixeddefaultT{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksreturnc Csjtjtj||||||o|dkdddtjtddddddd } tjtjdd| |ddd||t| d d d } | S) aConfigure the NeMo Lightning Trainer (for no model in particular). This function sets up the distributed training strategy. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=llama31_8b ... Python API usage: >>> trainer_config = trainer(num_nodes=2, num_gpus_per_node=8) >>> print(trainer_config) Note: This configuration uses extensive parallelism to handle the large model size efficiently. rT)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collective) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpu2 Fi) acceleratoraccumulate_grad_batchesrdeviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsrrpluginsstrategyuse_distributed_samplerval_check_interval)runConfignlMegatronStrategyrTrainerr ) rrrrrrrrrrr5trainerr>m/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/modelopt/recipes/distillation_recipe.pyr= sJ* r=)targetnamestudent_model_pathteacher_model_pathdistillation_config_pathdirrAc CsRtjtj|||t||dtjtjddddt||t|ddt dd t d }|S) aCreate a generic distillation recipe. This function sets up a complete configuration for distillation, including trainer, data, logging, optimization, and resumption settings. Args: student_model_path (Path): Path to student model NeMo checkpoint to be trained. teacher_model_path (Path): Path to teacher model NeMo checkpoint to distill from. distillation_config_path (Optional[str]): Path to distillation config file. dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the distillation run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. performance_mode (bool): If true, enables optimizations for maximum performance. Returns: run.Partial: Partial configuration for distillation. Python recipe usage: >>> import nemo_run as run >>> from nemo.collections import llm >>> from nemo.collections.llm.modelopt.recipes import distillation_recipe >>> recipe = distillation_recipe( ... student_model_path="/path/to/student/nemo/ckpt/", ... teacher_model_path="/path/to/teacher/nemo/ckpt/", ... distillation_config_path="/path/to/distillation/config.yaml", ... ) >>> # Override the configuration with desired components: >>> recipe.data = run.Config(llm.PreTrainingDataModule, ...) >>> recipe.trainer.strategy.tensor_model_parallel_size = 8 >>> ... >>> run.run(recipe) )rri ir) seq_lengthglobal_batch_sizemicro_batch_size)rA)rErAr giUMu>)max_lr)rBrCrDr=datalogoptimresume) r8Partialrdistillr=r9MockDataModulerr r r)rBrCrDrErArrreciper>r>r?distillation_recipess,rR) r rNNrTrr rN)NNr rr )$typingrnemo_runr8torch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorr:nemo.collectionsr(nemo.collections.llm.recipes.log.defaultrrr 'nemo.collections.llm.recipes.optim.adamr 6nemo.collections.llm.recipes.precision.mixed_precisionr NAMEintdtypeboollistr9r<r=clifactoryrOstrrNrRr>r>r>r?s          S