o }oi*@sddlmZddlmZddlZddlZddlm Z ddl m Z ddl m ZddlmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddl m!Z!dZ"ej#j$e"ddej%ej&fddZ'          d8de(de(deej)dee(de(de*de(d e(d!e(d"ee+ej%e dej%ej,fd#d$Z-ej#j$ee"d%  & '   ( ) *  d9d+ee.d,e.d-e.de(d e(d.e(d/e(d0ee.d1ee(d2ee*dej/fd3d4Z0d5ej/d0ee.dej/fd6d7Z1dS):)OptionalN)Callback)DistributedDataParallelConfig) lightning)llm)Llama32EmbeddingConfig3BLlamaEmbeddingModel)finetune) PEFT_STR2CLS)default_finetune_recipe) bf16_mixed)GarbageCollectionCallback)MegatronCommOverlapCallback)TimingCallbacknvembed_llama_3b)namereturncCstjtttdS)ay Factory function to create a NVEmbed Llama3.2 3b model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the NVEmbed Llama3.2 3B model. Examples: CLI usage: $ nemo llm pretrain model=nvembed_llama_3b ... Python API usage: >>> model_config = model() >>> print(model_config) )config)runConfigrrrrc/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/llama_embedding_3b.pymodel&srF{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csbtjtj||||||dddtjtddddddd } tjtjdd| |ddd||t| d d d } | S) au Configure the NeMo Lightning Trainer for NVEmbed Llama3.2 3B model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=nvembed_llama_3b ... Python API usage: >>> trainer_config = trainer(num_nodes=2, num_gpus_per_node=8) >>> print(trainer_config) Note: For more information on distributed training strategies, refer to the NeMo documentation on multi-GPU and multi-node training. Tcheck_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collective) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpur2 Fi) acceleratoraccumulate_grad_batchesr&deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr%r#pluginsstrategyuse_distributed_samplerval_check_interval)rrnlMegatronStrategyrTrainerr ) rrrr r!r"r#r$r%r&rBtrainerrrrrH9sJ,rH)targetrdefaultmeta-llama/Llama-3.2-3B@loradirr resume_pathmicro_batch_sizeglobal_batch_size peft_scheme seq_lengthpacked_sequencec Cs|durd}| dusJdtt|||||| } |dus#|dkr.d| jj_d| jj_n/|dvrVt t || _ d| j _ d | j _d | jj_d | jj_d | jj_ntd ||| jj_t j tj|||| jjj| jjj| jjj| jjjd d| _| S)aj Create a fine-tuning recipe for NVEmbed Llama3.2 3B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. resume_path (str): Path to the Huggingface model or pretrained distributed checkpoint for resume num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. micro_batch_size (int): Size of micro batch. global_batch_size (int): Size of global batch. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. seq_length (int): Maximum number of tokens per microbatch. packed_sequence (Optional[bool]): If true, fine-tuning sequences will be packed into batches up to the given maximum seq_length for better efficiency. pack sequence is not supported for embedding model training. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory nvembed_llama_3b Python API usage: >>> recipe = finetune_recipe(name="nvembed_llama_3b_finetune", num_nodes=2) >>> print(recipe) Note: This recipe uses the SPECTER dataset for fine-tuning. For more information on fine-tuning LLMs with NeMo, see the fine-tuning guide in the `examples/llm/finetune/` directory. Niz>pack_sequence is not supported for Embedding model finetuning.nonergh㈵>)rNdorarFg-C6?zUnrecognized peft scheme: )num_hard_negativesnegative_sample_strategyadd_bosadd_eos)rTrQrRdataset_kwargs)r rlowerrHrBr-optimrlrrrr peftdimalphause_distributed_optimizercross_entropy_loss_fusion ValueErrorrTrSpecterDataModulerYrZr[r\data) rOrrPr#r$rQrRrSrTrUreciperrrfinetune_recipes81        rjricCsd|jj_t|jdsg|j_|dus|dkr9d|jj_tj t dddddd|jj_ |jj tj t ddnd g|j_|jj t t|jj t td d |S) a Modify the given recipe to optimize settings for performance. This method enables performance optimizations that may not be suitable for all use cases. Intended to build upon the standard fine-tuning recipe. Args: recipe (run.Partial): Base fine-tuning recipe to which performance optimizations will be added peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. Returns: run.Partial: Partial configuration for performance-optimized fine-tuning. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. rr&NrVFTr')tp_comm_overlap linear_qkvd)rHrBr-hasattrr&r^rAr)rrrr6appendrratarget_modulesrr )rirSrrr"finetune_performance_optimizationss:     rq) rrNNrFrrrN) NrJrKrrrLrMrNNN)2typingrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrEnemo.collectionsrnemo.collections.llmrrnemo.collections.llm.apir nemo.collections.llm.peftr -nemo.collections.llm.recipes.finetune_defaultr 6nemo.collections.llm.recipes.precision.mixed_precisionr 3nemo.lightning.pytorch.callbacks.garbage_collectionr 6nemo.lightning.pytorch.callbacks.megatron_comm_overlaprnemo.utils.exp_managerrNAMEclifactoryrLightningModulerintdtypeboollistrGrHstrPartialrjrqrrrrs                 U   X