o }oi!@sddlmZddlmZddlZddlZddlm Z ddl m Z ddl m ZddlmZddlmZddlmZmZdd lmZdd lmZdd lmZd Zejjed dejej fddZ!          d/de"de"deej#dee"de"de$de"de"de"dee%eje dejej&fdd Z'ejjeed!  " #    $   d0d%ee(d&e(d'e(de"de"d(e"d)e"d*ee(d+ee"d,ee$dej)fd-d.Z*dS)1)OptionalN)Callback)DistributedDataParallelConfig) lightning)llm)finetune)Llama32Reranker1BConfig ReRankerModel) PEFT_STR2CLS)default_finetune_recipe) bf16_mixedllama_reranker_1b)namereturncCstjtttdS)av Factory function to create a Llama Reranker 1B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Llama Reranker 1B model. Examples: CLI usage: $ nemo llm pretrain model=llama_reranker_1b ... Python API usage: >>> model_config = model() >>> print(model_config) )config)runConfigr rrrb/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/llama_reranker_1b.pymodel#srF{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csbtjtj||||||dddtjtddddddd } tjtjdd| |ddd||t| d d d } | S) av Configure the NeMo Lightning Trainer for NVEmbed Llama3.2 1B model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=llama_reranker_1b ... Python API usage: >>> trainer_config = trainer(num_nodes=2, num_gpus_per_node=8) >>> print(trainer_config) Note: For more information on distributed training strategies, refer to the NeMo documentation on multi-GPU and multi-node training. T)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collective) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpur2 Fi) acceleratoraccumulate_grad_batchesr"deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr!rpluginsstrategyuse_distributed_samplerval_check_interval)rrnlMegatronStrategyrTrainerr ) rrrrrrrr r!r"r=trainerrrrrC6sJ,rC)targetrdefaultmeta-llama/Llama-3.2-1B@dirr resume_pathmicro_batch_sizeglobal_batch_size peft_scheme seq_lengthpacked_sequencec Cs|durd}| dusJdtt|||||| } |dus#|dkr.d| jj_d| jj_n/|dvrVt t || _ d| j _ d | j _d | jj_d | jj_d | jj_ntd ||| jj_d| jj_t j tj|||| jjj| jjj| jjj| jjjd dd| _| S)a Create a fine-tuning recipe for Llama Reranker 1B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. resume_path (str): Path to the Huggingface model or pretrained distributed checkpoint for resume num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. micro_batch_size (int): Size of micro batch. global_batch_size (int): Size of global batch. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. seq_length (int): Maximum number of tokens per microbatch. packed_sequence (Optional[bool]): If true, fine-tuning sequences will be packed into batches up to the given maximum seq_length for better efficiency. pack sequence is not supported for embedding model training. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory nvembed_llama_1b Python API usage: >>> recipe = finetune_recipe(name="nvembed_llama_1b_finetune", num_nodes=2) >>> print(recipe) Note: This recipe uses the SPECTER dataset for fine-tuning. Niz>pack_sequence is not supported for Embedding model finetuning.nonergh㈵>)loradorarFg-C6?zUnrecognized peft scheme: )num_hard_negativesnegative_sample_strategyadd_bosadd_eosr)rMrJrKdataset_kwargs num_workers)r rlowerrCr=r(optimrlrrrr peftdimalphause_distributed_optimizercross_entropy_loss_fusion ValueErrorrMrSrSpecterReRankerDataModulerTrUrVdata) rHrrIrr rJrKrLrMrNreciperrrfinetune_recipes</        re) rrNNrFrrrN) NrErFrrrrGNNN)+typingrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorr@nemo.collectionsrnemo.collections.llm.apir'nemo.collections.llm.gpt.model.rerankerrr nemo.collections.llm.peftr -nemo.collections.llm.recipes.finetune_defaultr 6nemo.collections.llm.recipes.precision.mixed_precisionr NAMEclifactoryrLightningModulerintdtypeboollistrBrCstrPartialrerrrrs              U