o wiC8.@s:ddlmZddlmZddlZddlZddlZddl m Z ddl m Z ddl mZddlmZddlmZmZddlmZdd lmZmZmZdd lmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&dej'j(_)dZ*ej+j,e*ddGde-dej.ej/fddZ0ej+j,e*ddGde-dej.ej/fddZ1ej+j,ee*ddddej2dddddddddddd dfd!e-d"e3d#e3d$ej4d%ee3d&e3d'e5d(e3d)e3d*e3d+e3d,e3d-e3d.e3d/e3d0e5d1ee6ej.e dej.ej7f$d2d3Z8ej+j,ee*ddd4ddddddddddddd d5d6dd efd!ee-d7e-de-d(e3d)e3d"e3d'e5d#e3d*e3d+e3d,e3d-e3d.e3d/e3d0e5d8e3d9e3d:e3d;e5dej9f(d   4      5       6  ?dHd@e-de-d!ee-d7e-d(e3d)e3d"e3d'e5d#e3d8e3d*e3d+e3d,e3d-e3d.e3d/e3d0e5d9e3d:e3d;e5dAee-dej9f,dBdCZ;dDej9dej9fdEdFZ>> model_config = model() >>> print(model_config) r )configr#)rrr MambaModelNemotronHConfig56Br#r r!r!r"model@s   r')targetr 2Fdirtensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_stepsval_check_intervallimit_test_batcheslimit_val_batcheslog_every_n_steps save_top_kckpt_async_save callbackscCstjtj|||||dd|ddtjtdddddd }tttjt| ||ddddg}tjtj||| d|g|| | d dtg| dd }|S) am Configure the NeMo Lightning Trainer for NemotronH Hybrid 56B model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=nemotronh_56b ... Python API usage: >>> trainer_config = trainer(num_nodes=1, num_gpus_per_node=1) >>> print(trainer_config) Note: For more information on distributed training strategies, refer to the NeMo documentation on multi-GPU and multi-node training. T torch_distlog_allF)check_for_nan_in_gradoverlap_grad_reduceoverlap_param_gathergrad_reduce_in_fp32) tensor_model_parallel_sizepipeline_model_parallel_sizecontext_parallel_sizepipeline_dtypesequence_parallelckpt_load_optimizerckpt_save_optimizerr>save_ckpt_formatckpt_load_strictnessddp)every_n_train_stepsdirpathr=always_save_contextsave_optim_on_train_endsave_context_on_train_endgpur)devicesr6r8 acceleratorstrategyloggerr?r<r;num_sanity_val_stepsuse_distributed_samplerpluginsr9enable_checkpointing) rrnlMegatronStrategyrrrTrainerr)r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?rXtrainerr!r!r"raTsb. radefault r seq_lengthgbsmbsperformance_modecCsztj|t|dt|||||||| | | | | |d tjt|||t|ddt||t|ddt ddt d}|r;t |}|S) a Create a pre-training recipe for NemotronH Hybrid 56B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory nemotronh_56b $ nemo llm pretrain --factory "nemotronh_56b(num_nodes=32, name='my_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="nemotronh_56b_pretrain", num_nodes=32) >>> print(recipe) r ) r/r8r6r0r1r5r7r9r:r;r<r=r>reglobal_batch_sizemicro_batch_sizer#rr/rr ga2U0*3?)max_lrr'radatalogoptimresume) rPartialr'rarr r#r r r r performance_optimizations)r/rrr6r7r0r5r1r8r9r:r;r<r=r>rerfrgrhfnreciper!r!r"pretrain_recipes@,rwnemotronh-56b-pretrainnone resume_path peft_schemecCstjtjtjtj|dd}tjtjt|dt | |||||| | | |||d tjt | ||t |ddtj ||t |ddtdd d d |d }|d usO|dkrZd|j j_d|jj_ntd||rgt|}|S)a Create a fine-tuning recipe for NemotronH Hybrid 56B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. resume_path (str): Path to the NeMo checkpoint (refer to notes below on how to convert a pytorch checkpoint to NeMo) vocab_file (str): Path to vocab file (defaults to None) num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory nemotronh_56b Python API usage: >>> recipe = finetune_recipe(name="nemotronh_56b_finetune", num_nodes=32) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. For converting an SSM pytorch checkpoint, use the following line of python code: llm.MambaModel(llm.NemotronHConfig56B(), tokenizer=tokenizer(vocab_file=vocab_file)).import_ckpt( path="pytorch://ABSOLUTE_PATH_TO_CKPT/your_pytorch_state_dict_file", model_config=llm.NemotronHConfig56B()) This line will cache the nemo checkpoint to following directory: /root/.cache/nemo/models/your_pytorch_state_dict_file )path)restore_configr ) r8r6r0r1r5r7r9r:r;r<r=r>rirrlg-C6?rr-)rmmin_lr warmup_stepsrnNryr)gh㈵>zUnrecognized peft scheme: )rrr^ AutoResume RestoreConfigrsrrr'rar r#r r r lowerrXrFrqr$lr ValueErrorrt)rzrr/rr6r7r0r5r1rer8r9r:r;r<r=r>rfrgrhr{ nemo_resumervr!r!r"finetune_recipesN5   rrvcCs|jjtjtdd|S)a Create a performance-optimized pre-training recipe for NemotronH Hybrid 56B model. This method enables performance optimizations that may not be suitable for all use cases. It builds upon the standard pre-training recipe and adds additional performance enhancements. Args: recipe (run.Partial): Base pre-train recipe to which performance optimizations will be added Returns: run.Partial: Partial configuration for performance-optimized pre-training. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. T)tp_comm_overlap)rar?appendrrr)rvr!r!r"rtds rt)N)rxNNrbr+r)r)Tr*rcr,r,r-r+r*r.Frdr*Fry)=typingrlightning.pytorchpytorchplnemo_runrtorch torch._dynamo$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorr^nemo.collectionsrnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr (nemo.collections.llm.recipes.log.defaultr r r 'nemo.collections.llm.recipes.optim.adamr 6nemo.collections.llm.recipes.precision.mixed_precisionr3nemo.collections.nlp.modules.common.tokenizer_utilsr nemo.lightning.pytorch.callbacksr6nemo.lightning.pytorch.callbacks.megatron_comm_overlaprnemo.utils.exp_managerr_dynamor$suppress_errorsNAMEclifactorystrrLightningModuler#r'bfloat16intdtypeboollistr`rarsrwrrtr!r!r!r"s                       b      M      ^