o }oi4,@sddlmZddlmZddlZddlZddlZddl m Z ddl m Z ddl mZddlmZddlmZmZddlmZdd lmZmZmZdd lmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&dej'j(_)dZ*ej+j,e*ddDde-dej.ej/fddZ0ej+j,e*ddDde-dej.ej/fddZ1ej+j,ee*ddddej2ddddddddddd d!dfd"e-d#e3d$e3d%ej4d&ee3d'e3d(e5d)e3d*e3d+e3d,e3d-e3d.e3d/e3d0e3d1e5d2ee6ej.e dej.ej7f$d3d4Z8ej+j,ee*ddd5dddddddddddd d!d6d7defd"ee-d8e-de-d)e3d*e3d#e3d(e5d$e3d+e3d,e3d-e3d.e3d/e3d0e3d1e5d9e3d:e3d;e3dej9f&d   5      6      ! 7  ?dEd@e-de-d"ee-d8e-d)e3d*e3d#e3d(e5d$e3d9e3d+e3d,e3d-e3d.e3d/e3d0e3d1e5d:e3d;e3dAee-dej9f*dBdCZ;dS)F)OptionalN)Callback)DistributedDataParallelConfig) lightning)llm)finetunepretrain)MockDataModule) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing) bf16_mixed)get_nmt_tokenizer)ModelCheckpoint)MegatronCommOverlapCallback)TimingCallbackT nemotronh_8bname vocab_filereturncCs*|r tjtdd|ddStjtddddS)zZ Factory function to create a tokenizer configuration for NemotronH Hybrid model. tiktokenTiktokenTokenizerT)library model_nameruse_fast huggingfaceznvidia/Nemotron-H-8B-Base-8K)rrr)runConfigrrr!]/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/nemotronh_8b.py tokenizer*sr#cCs tjtjttjt|ddS)ar Factory function to create a NemotronH Hybrid 8B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the NemotronH Hybrid 8B model. Examples: CLI usage: $ nemo llm pretrain model=nemotronh_8b ... Python API usage: >>> model_config = model() >>> print(model_config) r )configr#)rrr MambaModelNemotronHConfig8Br#r r!r!r"model@s   r')targetr 2 Fdirtensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_stepsval_check_intervallimit_test_batcheslimit_val_batcheslog_every_n_steps save_top_kckpt_async_save callbackscCstjtj|||||dd|ddtjtdddddd }tttjtdddtjt| ||dddd g}tjtj||| d |g|| | d dt g| dd }|S) ak Configure the NeMo Lightning Trainer for NemotronH Hybrid 8B model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=nemotronh_8b ... Python API usage: >>> trainer_config = trainer(num_nodes=1, num_gpus_per_node=1) >>> print(trainer_config) Note: For more information on distributed training strategies, refer to the NeMo documentation on multi-GPU and multi-node training. T torch_distlog_allF)check_for_nan_in_gradoverlap_grad_reduceoverlap_param_gathergrad_reduce_in_fp32) tensor_model_parallel_sizepipeline_model_parallel_sizecontext_parallel_sizepipeline_dtypesequence_parallelckpt_load_optimizerckpt_save_optimizerr?save_ckpt_formatckpt_load_strictnessddpnccl)tp_comm_bootstrap_backendtp_comm_overlap)every_n_train_stepsdirpathr>always_save_contextsave_optim_on_train_endsave_context_on_train_endgpur)devicesr7r9 acceleratorstrategyloggerr@r=r<num_sanity_val_stepsuse_distributed_samplerpluginsr:enable_checkpointing) rrnlMegatronStrategyrrrrTrainerr)r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@r\trainerr!r!r"reTsl.redefault r seq_lengthgbsmbscCsntj|t|dt|||||||| | | | | |d tjt|||t|ddt||t|ddt ddt d}|S) a Create a pre-training recipe for NemotronH Hybrid 8B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory nemotronh_8b $ nemo llm pretrain --factory "nemotronh_8b(num_nodes=32, name='my_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="nemotronh_8b_pretrain", num_nodes=32) >>> print(recipe) r ) r0r9r7r1r2r6r8r:r;r<r=r>r?riglobal_batch_sizemicro_batch_sizer#rr0rr ga2U0*3?)max_lrr'redatalogoptimresume) rPartialr'rerr r#r r r r )r0rrr7r8r1r6r2r9r:r;r<r=r>r?rirjrkfnreciper!r!r"pretrain_recipes<+rynemotronh-8b-pretrainnone resume_path peft_schemecCstjtjtjtj|dd}tjtjt|dt | |||||| | | |||d tjt | ||t |ddtj ||t |ddtdd d d |d }|d usO|dkr[d|j j_d|jj_|Std|)a Create a fine-tuning recipe for NemotronH Hybrid 8B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. resume_path (str): Path to the NeMo checkpoint (refer to notes below on how to convert a pytorch checkpoint to NeMo) vocab_file (str): Path to vocab file (defaults to None) num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory nemotronh_8b Python API usage: >>> recipe = finetune_recipe(name="nemotronh_8b_finetune", num_nodes=32) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. For converting an SSM pytorch checkpoint, use the following line of python code: llm.MambaModel(llm.NemotronHConfig8B(), tokenizer=tokenizer(vocab_file=vocab_file)).import_ckpt( path="pytorch://ABSOLUTE_PATH_TO_CKPT/your_pytorch_state_dict_file", model_config=llm.NemotronHConfig8B()) This line will cache the nemo checkpoint to following directory: /root/.cache/nemo/models/your_pytorch_state_dict_file )path)restore_configr ) r9r7r1r2r6r8r:r;r<r=r>r?rlrrog-C6?rr-)rpmin_lr warmup_stepsrqNr{r+gh㈵>zUnrecognized peft scheme: )rrrb AutoResume RestoreConfigrvrrr'rer r#r r r lowerr\rGrtr$lr ValueError)r|rr0rr7r8r1r6r2rir9r:r;r<r=r>r?rjrkr} nemo_resumerxr!r!r"finetune_recipesJ4   r)N)rzNNrfr+r+r)Tr*rgr,r,r-r.r,r/Frhr*r{)<typingrlightning.pytorchpytorchplnemo_runrtorch torch._dynamo$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrbnemo.collectionsrnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr (nemo.collections.llm.recipes.log.defaultr r r 'nemo.collections.llm.recipes.optim.adamr 6nemo.collections.llm.recipes.precision.mixed_precisionr3nemo.collections.nlp.modules.common.tokenizer_utilsr nemo.lightning.pytorch.callbacksr6nemo.lightning.pytorch.callbacks.megatron_comm_overlaprnemo.utils.exp_managerr_dynamor$suppress_errorsNAMEclifactorystrrLightningModuler#r'bfloat16intdtypeboollistrdrervryrr!r!r!r"s                       g      J