o }oiQ'@sddlmZddlmZddlZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZmZddlmZdd lmZmZdd lmZmZmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+dZ,ej-j.e,ddej/ej0fddZ1          d3de2de2deej3dee2de2d e4d!e2d"e2d#e2d$ee5ej/e dej/ej6fd%d&Z7ej-j.ee,d'dd(ddefd)ee8d*e8d!e2d"e2dej9f d+d,Z:ej-j.ee,d' -  (   .d4d/e8d)ee8d*e8d!e2d"e2d0ee8dej9fd1d2Z;dS)5)OptionalN)Callback)DistributedDataParallelConfig)OptimizerConfig) lightning)finetunepretrain) PEFT_STR2CLS)default_finetune_trainer nemo_resume) default_logdefault_resumetensorboard_logger) bf16_mixed)MockDataModule)SquadDataModule) T5Config220MT5Model)WarmupAnnealingScheduler)MegatronOptimizerModule)TimingCallbackt5_220mnamereturncCstjtttdS)aX Factory function to create a T5 220M model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the T5 220M model. Examples: CLI usage: $ nemo llm pretrain model=t5_220m ... Python API usage: >>> model_config = model() >>> print(model_config) )config)runConfigrrrrX/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/t5_220m.pymodel)sr F@Btensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Cs`tjtj||||||dddtjtdddddd } tjtjdd| |ddd||t| d d d } | S) a[ Configure the NeMo Lightning Trainer for T5 model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=t5_220m ... Python API usage: >>> trainer_config = trainer(num_nodes=2, num_gpus_per_node=8) >>> print(trainer_config) Note: For more information on distributed training strategies, refer to the NeMo documentation on multi-GPU and multi-node training. T)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gather) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpur!2 F) acceleratoraccumulate_grad_batchesr-deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr,r*pluginsstrategyuse_distributed_samplerval_check_interval)rrnlMegatronStrategyrTrainerr) r$r%r&r'r(r)r*r+r,r-rHtrainerrrrrN<sH,rN)targetrdefaultdirrc Cstjtdddddd}tjtddddd }tj|tt||ttgd tjtd d d d dt ||t |ddtjt ||dt dS)a Create a pre-training recipe for T5 220m model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory t5_220m $ nemo llm pretrain --factory "t5_220m(num_nodes=2, name='my_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="t5_220m_pretrain", num_nodes=2) >>> print(recipe) adam-C6?T{Gz? optimizerlruse_distributed_optimizerbf16 weight_decayNr#h㈵>) warmup_steps warmup_ratior,min_lr)r*r+r-r! seq_lengthseq_length_decglobal_batch_sizemicro_batch_sizerrQrrr lr_schedulerr rNdatalogoptimresume) rrrrPartialr rNrrr rrr )rQrr*r+fn opt_configrhrrrpretrain_recipes8  rqloracheckpoint_path peft_schemec Cstjtdddddd}tjtdddd }tjttt||d tjtd d d d dt ||t |ddtjt ||dt |d}|dusG| dkrSd |jj_d|jj_|S| dvrjtt| |_d|jj_|Std|)ak Create a fine-tuning recipe for T5 220M model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: checkpoint_path (str): Path to pretrained checkpoint dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory t5_220m Python API usage: >>> recipe = finetune_recipe(name="t5_220m_finetune", num_nodes=1) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. rRrSTrTrUr=r@r[)r\r,r^)r*r+r_r`r!rarrfrgriNnonegh㈵>)rsdorazUnrecognized peft scheme: )rrrrrnrr r rr rrr lowerrNrHr2rlrrWr peft ValueError) rtrQrr*r+rurprhreciperrrfinetune_recipesJ'      r|) r!r!NNr!Fr!r"r#N)rrNrPr!r"rs)<typingrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrmegatron.core.optimizerrnemorrKnemo.collections.llm.apirrnemo.collections.llm.peftr -nemo.collections.llm.recipes.finetune_defaultr r (nemo.collections.llm.recipes.log.defaultr r r6nemo.collections.llm.recipes.precision.mixed_precisionr!nemo.collections.llm.t5.data.mockr"nemo.collections.llm.t5.data.squadr nemo.collections.llm.t5.model.t5rr)nemo.lightning.pytorch.optim.lr_schedulerr%nemo.lightning.pytorch.optim.megatronrnemo.utils.exp_managerrNAMEclifactoryrLightningModuler intdtypeboollistrMrNstrrnrqr|rrrrs                 T  =