o wic)@sFddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZmZddlmZdd lmZdd lmZdd lmZmZmZdd l m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'dZ(ej)j*e(ddej+ej,fddZ-          d5de.de.deej/dee.de.de0d e.d!e.d"e.d#ee1ej+e dej+ej2fd$d%Z3ej)j*ee(d&dd'ddefd(ee4d)e4d e.d!e.dej5f d*d+Z6ej)j*ee(d,d&dd'ddefd(ee4d)e4d e.d!e.d-edej5f d.d/Z7ej)j*ee(d&  '   0 d6d(ee4d)e4d e.d!e.d1ee4d2e0dej5fd3d4Z8dS)7)CallableOptionalN)Callback)DistributedDataParallelConfig) lightning) GemmaConfig2B GemmaModel)finetunepretrain)MockDataModule) PEFT_STR2CLS)default_finetune_recipe) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing) bf16_mixed)MegatronCommOverlapCallback)TimingCallbackgemma_2bnamereturncCstjtttdS)a[ Factory function to create a Gemma 2B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Gemma 2B model. Examples: CLI usage: $ nemo llm pretrain model=gemma_2b ... Python API usage: >>> model_config = model() >>> print(model_config) )config)runConfigrrrrb/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/recipes/gemma_2b.pymodel%srF{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Cs`tjtj||||||dddtjtdddddd } tjtjdd| |ddd||t| d d d } | S) ab Configure the NeMo Lightning Trainer for Gemma 2B model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=gemma_2b ... Python API usage: >>> trainer_config = trainer(num_nodes=2, num_gpus_per_node=8) >>> print(trainer_config) Note: For more information on distributed training strategies, refer to the NeMo documentation on multi-GPU and multi-node training. T)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gather) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpur2 Fi) acceleratoraccumulate_grad_batchesr,deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr+r)pluginsstrategyuse_distributed_samplerval_check_interval)rrnlMegatronStrategyrTrainerr) r#r$r%r&r'r(r)r*r+r,rFtrainerrrrrL8sH,rL)targetrdefaultdirrc CsRtj|tt||ttgdtjtddddt||t|ddt dd t d S) a Create a pre-training recipe for Gemma 2B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory gemma_2b $ nemo llm pretrain --factory "gemma_2b(num_nodes=2, name='my_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="gemma_2b_pretrain", num_nodes=2) >>> print(recipe) )r)r*r,i ir) seq_lengthglobal_batch_sizemicro_batch_sizer)rOrrga2U0*3?)max_lr)rrLdatalogoptimresume) rPartialrrLrrr rrrr)rOrr)r*fnrrrpretrain_recipes rZ _optimizedrYcCs.t|||||d}|jjtjtdd|S)a) Create a performance-optimized pre-training recipe for Gemma 2B model. This recipe enables performance optimizations that may not be suitable for all use cases. It builds upon the standard pre-training recipe and adds additional performance enhancements. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for performance-optimized pre-training. Examples: $ nemo llm pretrain --factory gemma_2b_optimized Python API usage: >>> recipe = pretrain_recipe_performance(name="gemma_2b_perf", num_nodes=4) >>> print(recipe) Note: Use this recipe with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. )rrOr)r*rYF)tp_comm_overlap)rZrLr,appendrrr)rOrr)r*rYreciperrrpretrain_recipe_performances#r_lora peft_schemepacked_sequencecCsttd|||||}ddi|j_|dus|dkr'd|jj_d|jj _ |S|dvr>t t ||_d |jj _ |Std |) a Create a fine-tuning recipe for Gemma 2B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. packed_sequence (Optional[bool]): Packing multiple training sequences into one long sequence for training efficiency. Default sequence length is 2048. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory gemma_2b Python API usage: >>> recipe = finetune_recipe(name="gemma_2b_finetune", num_nodes=2) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. zgoogle/gemma-2badd_bosTNnoner gh㈵>)r`dorag-C6?zUnrecognized peft scheme: )r rrTdataset_kwargslowerrLrFr5rVrlrrrr peft ValueError)rOrr)r*rarbr^rrrfinetune_recipes(     rk) rrNNr Frr!r"N)NrNrr!r`F)9typingrrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrInemo.collections.llmrrnemo.collections.llm.apir r "nemo.collections.llm.gpt.data.mockr nemo.collections.llm.peftr -nemo.collections.llm.recipes.finetune_defaultr (nemo.collections.llm.recipes.log.defaultrrr'nemo.collections.llm.recipes.optim.adamr6nemo.collections.llm.recipes.precision.mixed_precisionr6nemo.lightning.pytorch.callbacks.megatron_comm_overlaprnemo.utils.exp_managerrNAMEclifactoryrLightningModulerintdtypeboollistrKrLstrrXrZr_rkrrrrs               T  + -