o }oi@.*@s&ddlmZddlmZddlZddlZddlm Z ddl m Z m Z ddlmZddlmZddlmZddlmZmZmZdd lmZmZdd lmZdd lmZdd lm Z dd l!m"Z"dZ#ej$j%e#ddej&ej'fddZ(ej$j%e e#ddddddddddddddddddddddd dd!d"de fd#ee)d$e)d%e*d&e*d'eej+d(ee*d)e*d*e,d+e*d,e*d-e*d.e)d/e*d0e-d1e*d2e*d3e*d4e*d5e,dej.f(d6d7Z/d8ej.dej.fd9d:Z0ej$j%e#d;ddej&e j1fd dCd#ee)d$e)d+e*d,e*d?ee)d@e,dej.fdAdBZ3dS)D)OptionalN) lightning)finetunepretrain)MockDataModule) PEFT_STR2CLS)default_finetune_recipe) default_logdefault_resumetensorboard_logger)nemotron_modelnemotron_trainer),distributed_fused_adam_with_cosine_annealing)GarbageCollectionCallback)MegatronCommOverlapCallback)TimingCallback nemotron3_8bnamereturncCs ttdS)ag Factory function to create a Nemotron3 8B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Nemotron3 8B model. Examples: CLI usage: $ nemo llm pretrain model=nemotron3_8b ... Python API usage: >>> model_config = model() >>> print(model_config) )version)r NAMErr]/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/nemotron3_8b.pymodel$s r)targetrdefaultFiz bf16-mixedg? iiigiUMu>ga2U0*3?dirrtensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps precisionaccumulate_grad_batchesgradient_clip_vallimit_test_batcheslimit_val_batcheslog_every_n_stepsval_check_intervalperformance_modecCstj|ttdid|d|d|d|d|d|d|d| d | d | d | d |d |d|d|dttgtjt|||dt||t|ddt | ||||| dt d}|rbt |}|S)a Create a pre-training recipe for Nemotron3 8B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. precision (str): Precision configuration, one of fp32, 16-mixed or bf16-mixed. accumulate_grad_batches (int): Number of steps per gradient accumulation. gradient_clip_val (float): Value for gradient clipping. limit_test_batches (int): Limit the number of test batches. limit_val_batches (int): Limit the number of validation batches. log_every_n_steps (int): Log every n steps. val_check_interval (int): Run validation every N steps. global_batch_size (int): Global batch size. micro_batch_size (int): Micro batch size. seq_length (int): Sequence length. warmup_steps (int): Number of warmup steps. constant_steps (int): Number of constant steps. min_lr (float): Minimum learning rate. max_lr (float): Maximum learning rate. performance_mode (bool): If true, enables optimizations for maximum performance. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory nemotron3_8b $ nemo llm pretrain --factory "nemotron3_8b(num_nodes=1, name='my_nemotron_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="nemotron_pretrain", num_nodes=1) >>> print(recipe) Note: This recipe uses a mock dataset, look for the finetune examples to see how to change the dataset. r#r$r%r&r'r(r)r*r+r,r-r/r0r1r2 callbacks) seq_lengthglobal_batch_sizemicro_batch_sizer)r"rr )r, warmup_stepsconstant_stepsmin_lrmax_lr clip_grad)rtrainerdatalogoptimresumeNr) runPartialrr Configrrr r rr "pretrain_performance_optimizations)r"rr#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r6r7r5r8r9r:r;r3fnreciperrrpretrain_recipe8spU      'rHrGcCsN|jjsg|j_tjtddd}tjtdd}|jj||gd|jj_|S)a Create a performance-optimized pre-training recipe for Nemotron3 8B model. This method enables performance optimizations that may not be suitable for all use cases. It builds upon the standard pre-training recipe and adds additional performance enhancements. Args: recipe (run.Partial): Base pre-train recipe to which performance optimizations will be added Returns: run.Partial: Partial configuration for performance-optimized pre-training. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. d)gc_interval_traingc_interval_valT)tp_comm_overlapF) r=r4rBrDrrextendpluginsgrad_reduce_in_fp32)rGgarbage_collection_callbackmcomm_overlap_callbackrrrrEs$ rE_nemocCstjtjtjtjdddS)a Configure automatic resumption from a NeMo checkpoint converted from Huggingface for Nemotron3 8B model. More info about the Huggingface model can be found at: https://huggingface.co/nvidia/nemotron-3-8b-base-4k. This NeMo checkpoint should be converted from Huggingface beforehand, using nemo.collections.llm.import_ckpt. When converting the checkpoint, the NeMo checkpoint will be saved in NEMO_HOME (set to ~/.cache/nemo by default). This function sets up the configuration to resume training from path nemo://nvidia/nemotron-3-8b-base-4k. This translates to the full path {NEMO_HOME}/models/nvidia/nemotron-3-8b-base-4k. Returns: run.Config[nl.AutoResume]: Configuration for resuming from NeMo checkpoint. Note: This is particularly useful for fine-tuning scenarios where you want to start from the pre-trained Nemotron3 8B model. z#nemo://nvidia/nemotron-3-8b-base-4k)path)restore_config)rBrDnl AutoResume RestoreConfigrrrr nemo_resumesrXlora peft_schemepacked_sequencecCsttd|||||}|dus|dkr d|jj_d|jj_n|dvr6t t ||_ d|jj_nt d|d |jj_|S) a Create a fine-tuning recipe for Nemotron3 8B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. packed_sequence (Optional[bool]): Packing multiple training sequences into one long sequence for training efficiency. Default sequence length is 2048. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory nemotron3_8b Python API usage: >>> recipe = finetune_recipe(name="nemotron3_8b_finetune", num_nodes=2) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. zthhaus/nemotron3-8bNnonergh㈵>)rYdorag-C6?zUnrecognized peft scheme: F)rrlowerr=strategytensor_model_parallel_sizer@configlrrBrDrpeft ValueErrorcross_entropy_loss_fusion)r"rr)r*rZr[rGrrrfinetune_recipes)     rf)NrrrrYF)4typingrlightning.pytorchpytorchplnemo_runrBtorchnemorrUnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockrnemo.collections.llm.peftr-nemo.collections.llm.recipes.finetune_defaultr(nemo.collections.llm.recipes.log.defaultr r r %nemo.collections.llm.recipes.nemotronr r 'nemo.collections.llm.recipes.optim.adamr3nemo.lightning.pytorch.callbacks.garbage_collectionr6nemo.lightning.pytorch.callbacks.megatron_comm_overlaprnemo.utils.exp_managerrrclifactoryrDLightningModulerstrintdtypeboolfloatrCrHrErVrXrfrrrrs                ! *