o wi0&@sddlmZddlmZddlZddlZddlm Z ddl m Z ddl m ZddlmZddlmZmZddlmZdd lmZmZmZdd lmZdd lmZdd lmZdd l m!Z!dZ"ej#j$e"dd>de%dej&ej'fddZ(ej#j$e"dd>de%dej&ej'fddZ)ej#j$ee"d              d?de*de*d eej+d!ee*d"e*d#e,d$e*d%e*d&e*d'e*d(e*d)e*d*e*d+ee-ej&e dej&ej.fd,d-Z/ej#j$ee"ddd.ddddddddddd/ddefd0ee%d1e%de%d$e*d%e*de*de*d&e*d'e*d(e*d)e*d*e*d2e*d3e*d4e*dej0f d5d6Z1ej#j$ee"d  .       /        7d@d0ee%d1e%d8e%de%d$e*d%e*d9e*d:e*d2e*d&e*d'e*d(e*d)e*d*e*d3e*d4e*d;ee%dej0f$d>> model_config = model() >>> print(model_config) r)configr)rrr MambaModelBaseMambaConfig780Mrrrrrmodel3s  r#)targetrFd2 tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_stepsval_check_intervallimit_test_batcheslimit_val_batcheslog_every_n_steps callbackscCs`tjtj||||||dddtjtdddddd }tjtjdd| |||t|d| | | | d}|S)ah Configure the NeMo Lightning Trainer for Mamba2 780M model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=mamba2_780m ... Python API usage: >>> trainer_config = trainer(num_nodes=1, num_gpus_per_node=1) >>> print(trainer_config) Note: For more information on distributed training strategies, refer to the NeMo documentation on multi-GPU and multi-node training. TF)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gather) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpur%) acceleratoraccumulate_grad_batchesr8devicesr3r1pluginsstrategyuse_distributed_samplerr4r5r6r7)rrnlMegatronStrategyrTrainerr)r+r,r-r.r/r0r1r2r3r4r5r6r7r8rLtrainerrrrrQJsH1rQdefaultdirr seq_lengthgbsmbscCsdtj|tt||||||| | | ttgd tjt| | |tdt||t |ddt ddt dS)a Create a pre-training recipe for Mamba2 780M model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory mamba2_780M $ nemo llm pretrain --factory "mamba2_780M(num_nodes=1, name='my_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="mamba2_780M_pretrain", num_nodes=1) >>> print(recipe) ) r3r1r+r,r2r4r5r6r7r8rUglobal_batch_sizemicro_batch_sizerrrTrr ga2U0*3?)max_lrr#rQdatalogoptimresume) rPartialr#rQrrr rr r r r )rTrrr1r2r+r,r3r4r5r6r7rUrVrWfnrrrpretrain_recipes4,  rdnone resume_pathr=r> peft_schemecCstjtjtjtj|dd}tjtj||ddddd}tjtjd|d}tjtjdd || | | | | |tjtjd t j d |g|dd }tj t j t|d |tjt j|||t|d dt j||t|ddtdddd|d}|dusu|dkrd |jj_d|jj_|Std|)a Create a fine-tuning recipe for Mamba2 780M model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. resume_path (str): Path to the NeMo checkpoint (refer to notes below on how to convert a pytorch checkpoint to NeMo) tokenizer_model (str): Path to tokenizer model (defaults to None) num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory mamba2_780m Python API usage: >>> recipe = finetune_recipe(name="mamba2_780m_finetune", num_nodes=1) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. For converting an SSM pytorch checkpoint, use the following line of python code: llm.MambaModel(llm.BaseMambaConfig780M(), tokenizer=tokenizer()).import_ckpt( path="pytorch://ABSOLUTE_PATH_TO_CKPT/your_pytorch_state_dict_file", model_config=llm.BaseMambaConfig780M()) This line will cache the nemo checkpoint to following directory: /root/.cache/nemo/models/your_pytorch_state_dict_file )path)restore_configTF)r=r>rCckpt_load_optimizerckpt_save_optimizerrDr*)every_n_train_stepsdirpathrGr%z bf16-mixed) precision params_dtype) rHrIrJr3r4r5r6r7r1rKr8rLrMrrXrr[g-C6?rr()r\min_lr warmup_stepsr]Nregh㈵>zUnrecognized peft scheme: )rrrN AutoResume RestoreConfigrOModelCheckpointrPMegatronMixedPrecisiontorchbfloat16rbrrr#SquadDataModulerr r r lowerrQrLr=r`r lr ValueError)rTrrfrr1r2r=r>rUr3r4r5r6r7rVrWrg nemo_resumerLcheckpoint_callbackrQreciperrrfinetune_recipest8    r)N)r%r%NNr%Fr%r&r'r'r(r)r*N)NrRNNr%r&r%r%rSr'r'r(r)r*r&r%re)3typingrlightning.pytorchpytorchplnemo_runrrv$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrNnemo.collectionsrnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr (nemo.collections.llm.recipes.log.defaultr r r 'nemo.collections.llm.recipes.optim.adamr 6nemo.collections.llm.recipes.precision.mixed_precisionr3nemo.collections.nlp.modules.common.tokenizer_utilsrnemo.utils.exp_managerrNAMEclifactorystrrLightningModulerr#intdtypeboollistrPrQrbrdrrrrrsV                    X      G