o }oi0"@sddlmZddlmZddlZddlZddlm Z ddl m Z ddl m ZddlmZddlmZmZddlmZdd lmZmZmZdd lmZdd lmZdd lmZdd l m!Z!dZ"ej#j$e"dd=de%dej&ej'fddZ(ej#j$e"dd=de%dej&ej'fddZ)ej#j$e"d              d>de*de*deej+d ee*d!e*d"e,d#e*d$e*d%e*d&e*d'e*d(e*d)e*d*ee-ej&e dej&ej.fd+d,Z/ej#j$ee"d-dd.ddddddddddd/ddefd0ee%d1e%de%d#e*d$e*de*de*d%e*d&e*d'e*d(e*d)e*d2e*d3e*d4e*dej0f d5d6Z1ej#j$ee"d-  .     /        7d?d0ee%d1e%d#e*d$e*d8e*d9e*d2e*d%e*d&e*d'e*d(e*d)e*d3e*d4e*d:ee%dej0f d;d<Z2dS)@)OptionalN)Callback)DistributedDataParallelConfig) lightning)llm)finetunepretrain)MockDataModule) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing) bf16_mixed)get_nmt_tokenizer)TimingCallback mamba2_8bnametokenizer_modelreturncCstjtdd|ddS)z? Factory function to create a tokenizer configuration. megatronGPTSentencePieceTokenizerT)library model_nameruse_fast)runConfigrrrZ/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/mamba2_8b.py tokenizer%sr cCs tjtjttjt|ddS)ae Factory function to create a Mamba2 8B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Mamba2 Hybrid 8B model. Examples: CLI usage: $ nemo llm pretrain model=mamba2_8b ... Python API usage: >>> model_config = model() >>> print(model_config) r)configr )rrr MambaModelNVIDIAMambaConfig8Br rrrrmodel3s  r$Fd2 tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_stepsval_check_intervallimit_test_batcheslimit_val_batcheslog_every_n_steps callbackscCs`tjtj||||||dddtjtdddddd }tjtjdd| |||t|d| | | | d}|S)ak Configure the NeMo Lightning Trainer for Mamba2 Hybrid 8B model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=mamba2_8b ... Python API usage: >>> trainer_config = trainer(num_nodes=1, num_gpus_per_node=1) >>> print(trainer_config) Note: For more information on distributed training strategies, refer to the NeMo documentation on multi-GPU and multi-node training. TF)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gather) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpur&) acceleratoraccumulate_grad_batchesr8devicesr3r1pluginsstrategyuse_distributed_samplerr4r5r6r7)rrnlMegatronStrategyrTrainerr)r+r,r-r.r/r0r1r2r3r4r5r6r7r8rLtrainerrrrrQJsH1rQ)targetrdefaultdirr seq_lengthgbsmbscCsltj|t|dt||||||| | | ttgd tjt| | |t|ddt||t |ddt ddt dS) a Create a pre-training recipe for Mamba2 Hybrid 8B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory mamba2_8b $ nemo llm pretrain --factory "mamba2_8b(num_nodes=1, name='my_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="mamba2_8b_pretrain", num_nodes=1) >>> print(recipe) r) r3r1r+r,r2r4r5r6r7r8rVglobal_batch_sizemicro_batch_sizer rrUrr ga2U0*3?)max_lrr$rQdatalogoptimresume) rPartialr$rQrrr r r r r r )rUrrr1r2r+r,r3r4r5r6r7rVrWrXfnrrrpretrain_recipes4,  renoner=r> peft_schemecCstjtjtjtj|dd}tjtj||ddddd}tjtjd|d}tjtjdd || | | | | |tjtjd t j d |g|dd }tj t j t|d |tjt j|||t|d dt j||t|ddtdddd|d}|dusu|dkrd|jj_d|jj_|Std|)a Create a fine-tuning recipe for Mamba2 8B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. resume_path (str): Path to the NeMo checkpoint (refer to notes below on how to convert a pytorch checkpoint to NeMo) tokenizer_model (str): Path to tokenizer model (defaults to None) num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory mamba2_8b Python API usage: >>> recipe = finetune_recipe(name="mamba2_8b_finetune", num_nodes=1) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. For converting an SSM pytorch checkpoint, use the following line of python code: llm.MambaModel(llm.NVIDIAMambaConfig8B(), tokenizer=tokenizer(tokenizer_model=tokenizer_model)).import_ckpt( path="pytorch://ABSOLUTE_PATH_TO_CKPT/your_pytorch_state_dict_file", model_config=llm.NVIDIAMambaConfig8B()) This line will cache the nemo checkpoint to following directory: /root/.cache/nemo/models/your_pytorch_state_dict_file )path)restore_configTF)r=r>rCckpt_load_optimizerckpt_save_optimizerrDr*)every_n_train_stepsdirpathrGr&z bf16-mixed) precision params_dtype) rHrIrJr3r4r5r6r7r1rKr8rLrMrrYrr\g-C6?rr()r]min_lr warmup_stepsr^Nrfr%gh㈵>zUnrecognized peft scheme: )rrrN AutoResume RestoreConfigrOModelCheckpointrPMegatronMixedPrecisiontorchbfloat16rcrrr$SquadDataModuler r r r lowerrQrLr=rar!lr ValueError) resume_pathrrUrr1r2r=r>rVr3r4r5r6r7rWrXrg nemo_resumerLcheckpoint_callbackrQreciperrrfinetune_recipest8    r)N)r%r&NNr&Fr&r%r'r'r(r)r*N)NrSr&r%r%r&rTr'r'r(r)r*r%r&rf)3typingrlightning.pytorchpytorchplnemo_runrrv$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrNnemo.collectionsrnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr (nemo.collections.llm.recipes.log.defaultr r r 'nemo.collections.llm.recipes.optim.adamr 6nemo.collections.llm.recipes.precision.mixed_precisionr3nemo.collections.nlp.modules.common.tokenizer_utilsrnemo.utils.exp_managerrNAMEclifactorystrrLightningModuler r$intdtypeboollistrPrQrcrerrrrrsJ                     X      G