o wi&@s$ddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZm Z m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'dZ(ej)j*e(d d5de+dej,ej-fddZ.          d6de+de+deej/dee+de+d e0d!e+d"e+d#e+d$ee1ej,e dej,ej2fd%d&Z3ej)j*ee(d'dd(dddefd)ee4d*e4d!e+d"e+de+d+edej5fd,d-Z6ej)j*ee(d'  . (   /  d7d)ee4d0e4d*e4d!e+d"e+d1ee4dee+d2ee0dej5fd3d4Z7dS)8)CallableOptionalN)Callback)DistributedDataParallelConfig) lightning)finetunepretrain)MockDataModule)PackedSequenceSpecs)Gemma3Config1B Gemma3Model) PEFT_STR2CLS)default_finetune_recipe) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing) bf16_mixed)TimingCallback gemma3_1bname seq_lengthreturncCstjt|d}tjt|dS)a^ Factory function to create a Gemma3 1B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Gemma3 1B model. Examples: CLI usage: $ nemo llm pretrain model=gemma3_1b ... Python API usage: >>> model_config = model() >>> print(model_config) r)config)runConfigr r )rconfr c/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/recipes/gemma3_1b.pymodel'sr"F tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csbtjtj||||||dddtjtddddddd } tjtjdd| |ddd||t| d d d } | S) a Configure the NeMo Lightning Trainer for Gemma3 1B model. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Note: This configuration uses extensive parallelism to handle the large model size efficiently. T)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collective) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpur#2 r%Fi) acceleratoraccumulate_grad_batchesr/deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr.r,pluginsstrategyuse_distributed_samplerval_check_interval)rrnlMegatronStrategyrTrainerr) r&r'r(r)r*r+r,r-r.r/rItrainerr r r!gemma3_trainer=sJ!rP)targetrdefaultdirrfnc CsZtj|t|dt||ttgdtjt|dddt||t|ddt dd t d }|S) a Create a pre-training recipe for Gemma3 1B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory gemma3_1b $ nemo llm pretrain --factory "gemma3_1b(num_nodes=1, name='my_1b_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="gemma3_1b_pretrain", num_nodes=1) >>> print(recipe) Note: This recipe is optimized for the large 8B model and requires significant computational resources. r)r,r-r/ir#)rglobal_batch_sizemicro_batch_sizer)rSrrga2U0*3?)max_lr)r"rOdatalogoptimresume) rPartialr"rPrrr rrrr)rSrr,r-rrTreciper r r!pretrain_recipes% r^google/gemma-3-1b-ptlora resume_path peft_schemepacked_sequencec Cs|dur |rdnd}tt||||||}|dus|dkr*d|jj_d|jj_n/|dvrRt t ||_ d|j _ d |j _d |jj_d |jj_d |jj_ntd |||jj_||j_|rsd di|j_t j t|d|j_|S)aY Create a fine-tuning recipe for Gemma3 1B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. resume_path (str): Path to the NeMo checkpoint name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. seq_length (int): Maximum number of tokens per microbatch. packed_sequence (Optional[bool]): If true, fine-tuning sequences will be packed into batches up to the given maximum seq_length for better efficiency. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory gemma3_1b Python API usage: >>> recipe = finetune_recipe(name="gemma3_1b_finetune", num_nodes=1) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. Niinoner#gh㈵>)r`dorar$Fg-C6?zUnrecognized peft scheme: pad_to_max_lengthT)packed_sequence_size)rr"lowerrOrIr5rZrlrrrr peftdimalphause_distributed_optimizercross_entropy_loss_fusion ValueErrorrrXdataset_kwargsr packed_sequence_specs) rSrarr,r-rbrrcr]r r r!finetune_recipes(.         rs)r) r#r#NNr#Fr#r$r%N)Nr_rRr#r$r`NN)8typingrrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrLnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr -nemo.collections.llm.gpt.data.packed_sequencer %nemo.collections.llm.gpt.model.gemma3r r nemo.collections.llm.peftr -nemo.collections.llm.recipes.finetune_defaultr(nemo.collections.llm.recipes.log.defaultrrr'nemo.collections.llm.recipes.optim.adamr6nemo.collections.llm.recipes.precision.mixed_precisionrnemo.utils.exp_managerrNAMEclifactoryintrLightningModuler"dtypeboollistrNrPstrr\r^rsr r r r!s                 J 5