o }oi\'@sddlmZddlmZddlZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZmZddlmZdd lmZmZdd lmZmZmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+dZ,ej-j.e,ddej/ej0fddZ1          d5de2de2deej3d ee2d!e2d"e4d#e2d$e2d%e2d&ee5ej/e dej/ej6fd'd(Z7ej-j.ee,d)dd*ddefd+ee8d,e8d#e2d$e2dej9f d-d.Z:ej-j.ee,d) /  *   0d6d1e8d+ee8d,e8d#e2d$e2d2ee8dej9fd3d4Z;dS)7)OptionalN)Callback)DistributedDataParallelConfig)OptimizerConfig) lightning)finetunepretrain) PEFT_STR2CLS)default_finetune_trainer nemo_resume) default_logdefault_resumetensorboard_logger) bf16_mixed)MockDataModule)SquadDataModule) T5Config11BT5Model)WarmupAnnealingScheduler)MegatronOptimizerModule)TimingCallbackt5_11bnamereturncCstjtttdS)aU Factory function to create a T5 11B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the T5 11B model. Examples: CLI usage: $ nemo llm pretrain model=t5_11b ... Python API usage: >>> model_config = model() >>> print(model_config) )config)runConfigrrrrW/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/t5_11b.pymodel)sr F@Btensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Cs`tjtj||||||dddtjtdddddd } tjtjdd| |ddd||t| d d d } | S) aZ Configure the NeMo Lightning Trainer for T5 model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=t5_11b ... Python API usage: >>> trainer_config = trainer(num_nodes=2, num_gpus_per_node=8) >>> print(trainer_config) Note: For more information on distributed training strategies, refer to the NeMo documentation on multi-GPU and multi-node training. T)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gather) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpur"2 F) acceleratoraccumulate_grad_batchesr/deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr.r,pluginsstrategyuse_distributed_samplerval_check_interval)rrnlMegatronStrategyrTrainerr) r&r'r(r)r*r+r,r-r.r/rJtrainerrrrrP<sH,rP)targetrdefaultdirrc Cstjtdddddd}tjtddddd }tj|tt||ttgd tjtd d d ddt ||t |ddtjt ||dt dS)a| Create a pre-training recipe for T5 11b model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory t5_11b $ nemo llm pretrain --factory "t5_11b(num_nodes=2, name='my_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="t5_11b_pretrain", num_nodes=2) >>> print(recipe) adam-C6?T{Gz? optimizerlruse_distributed_optimizerbf16 weight_decayNr%h㈵>) warmup_steps warmup_ratior.min_lr)r,r-r/i seq_lengthseq_length_decglobal_batch_sizemicro_batch_sizerrSrrr lr_schedulerr rPdatalogoptimresume) rrrrPartialr rPrrr rrr )rSrr,r-fn opt_configrkrrrpretrain_recipes<   rtloracheckpoint_path peft_schemec Cstjtdddddd}tjtdddd }tjttt||d tjtd d d d dt ||t |ddtjt ||dt |d}|dusG| dkrSd|jj_d|jj_|S| dvrjtt| |_d|jj_|Std|)ah Create a fine-tuning recipe for T5 11B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: checkpoint_path (str): Path to pretrained checkpoint dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory t5_11b Python API usage: >>> recipe = finetune_recipe(name="t5_11b_finetune", num_nodes=1) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. rTrUTrVrWr?rBr])r^r.r`)r,r-rarbr"rdrrirjrlNnoner!gh㈵>)rvdorazUnrecognized peft scheme: )rrrrrqrr r rr rrr lowerrPrJr4rorrYr peft ValueError) rwrSrr,r-rxrsrkreciperrrfinetune_recipesJ'      r) r!r"NNr"Fr#r$r%N)ruNrRr"r$rv)<typingrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrmegatron.core.optimizerrnemorrMnemo.collections.llm.apirrnemo.collections.llm.peftr -nemo.collections.llm.recipes.finetune_defaultr r (nemo.collections.llm.recipes.log.defaultr r r6nemo.collections.llm.recipes.precision.mixed_precisionr!nemo.collections.llm.t5.data.mockr"nemo.collections.llm.t5.data.squadr nemo.collections.llm.t5.model.t5rr)nemo.lightning.pytorch.optim.lr_schedulerr%nemo.lightning.pytorch.optim.megatronrnemo.utils.exp_managerrNAMEclifactoryrLightningModuler intdtypeboollistrOrPstrrqrtrrrrrs                 T  ?