o wi33@s8ddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZddlmZdd lmZdd lmZmZmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*dZ+ej,j-e+ddej.ej/fddZ0          d:de1de1deej2dee1d e1d!e3d"e1d#e1d$e1d%ee4ej.e dej.ej5fd&d'Z6ej,j-ee+d(dd)ddd*efd+ee7d,e7d"e1d#e1d-e3d.edej8fd/d0Z9d1ej8dej8fd2d3Z:ej,j-ej;e+d(  ) 4   5d;d+ee7d,e7d6ee7d"e1d#e1d7ee7dej8fd8d9ZtjtjtjtjttjttjtjtjdddddS)af Factory function to create a Gemma3 VL 4B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Gemma3 VL 4B model. Examples: CLI usage: $ nemo llm pretrain model=gemma3vl_4b ... Python API usage: >>> model_config = model() >>> print(model_config) ii ) input_size hidden_size)language_transformer_configvision_transformer_configvision_projection_config)config) runConfigr Gemma3VLModelGemma3VLConfig4BrGemma3Config4BGemma3VLVisionConfig!Gemma3VLMultimodalProjectorConfigr'r'e/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/vlm/recipes/gemma3vl_4b.pymodel)s  r)T{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csftjtj||||||dddtjtddddddddd } tjtjdd| |dd d ||t| d d d } | S)a Configure the NeMo Lightning Trainer for Gemma3 VL 4B model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=gemma3vl_4b ... Python API usage: >>> trainer_config = trainer(num_nodes=1, num_gpus_per_node=8) >>> print(trainer_config) Toptim_grads_params)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collectivedata_parallel_sharding_strategyN) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpfsdpgpur+2 Fi) acceleratoraccumulate_grad_batchesr7deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr6r4pluginsstrategyuse_distributed_samplerval_check_interval)r r!nlMegatronStrategyrTrainerr) r.r/r0r1r2r3r4r5r6r7rUtrainerr'r'r(r[HsN( r[)targetrdefaultFdirrperformance_modefnc Csbtj|tt||ttgdtjtddddt||t|ddt ddt d }|r/t |}|S) a Create a pre-training recipe for Gemma3 VL 4B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. performance_mode (bool): If true, enables optimizations for maximum performance. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory gemma3vl_4b ... $ nemo llm pretrain --factory "gemma3vl_4b(num_nodes=2, name='my_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="gemma3vl_4b_pretrain", num_nodes=2) >>> print(recipe) )r4r5r7r+) seq_lengthglobal_batch_sizemicro_batch_sizerr^rrga2U0*3?)max_lrr)r[datalogoptimresume) r Partialr)r[r!rrr rrr "pretrain_performance_optimizations)r^rr4r5r_r`reciper'r'r(pretrain_recipes # rorncCsZ|jjsg|j_tjtddd}tjtdd}tt}|jj|||gd|jj_ |S)a Create a performance-optimized pre-training recipe for Gemma3 VL 4B model. This method enables performance optimizations that may not be suitable for all use cases. It builds upon the standard pre-training recipe and adds additional performance enhancements. Args: recipe (run.Partial): Base pre-train recipe to which performance optimizations will be added Returns: run.Partial: Partial configuration for performance-optimized pre-training. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. d)gc_interval_traingc_interval_valT)tp_comm_overlapF) r[r7r r!rrrextendrTr:)rngarbage_collection_callbackmcomm_overlap_callbacktoken_drop_callbackr'r'r(rms, rmgoogle/gemma-3-4b-itnone resume_path peft_schemec Cs6tjtjfddtjdddddddddddtjtddddddd}tjtjd d|d dd |t|d tt gd }tj t j t |tjtdddtjtddddt j||t|ddtddddt|d}|duso|dkr{d|jj_d|jj_|S|dkrtjtjdgdd|_d |jj_|Std!|)"a Create a fine-tuning recipe for Gemma3 VL 4B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. resume_path (str): Path to the NeMo checkpoint name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Parameter efficient fine-tuning scheme to use. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory gemma3vl_4b Python API usage: >>> recipe = finetune_recipe(name="gemma3vl_4b_finetune", num_nodes=1) >>> print(recipe) r*r+NrTlog_all)r9r:r;r<r=)r?r@rArB"encoder_tensor_model_parallel_size$encoder_pipeline_model_parallel_sizerCrDrFckpt_parallel_saverGckpt_parallel_save_optimckpt_load_strictnessrErHrJrMiFi) rNrOrPrRrSr6r4rTrUrWr7rarLrx)pretrained_model_name)rbrcrd tokenizer num_workersrregh㈵>gH׊>)rfmin_lr warmup_stepsrgryloraF)z*.language_model.*.linear_qkvz*.language_model.*.linear_qz*.language_model.*.linear_kvz*.language_model.*.linear_projz*.language_model.*.linear_fc1z*.language_model.*.linear_fc2)freeze_vision_modeltarget_modulesg-C6?zUnrecognized peft scheme: )r r!rXrYtorchbfloat16rrZrrrlrfinetuner)rr r rrr lowerr[rUr?rjrlrrLoRApeft ValueError) r^rrzr4r5r{rUr[rnr'r'r(finetune_recipes$      r) r*r+NNr+Tr,r,r-N)Nr]rxr,r,ry)=typingrrlightning.pytorchpytorchplnemo_runr r$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrXnemo.collectionsrr"nemo.collections.common.tokenizersr nemo.collections.llm.apir -nemo.collections.llm.recipes.finetune_defaultr (nemo.collections.llm.recipes.log.defaultr r r'nemo.collections.llm.recipes.optim.adamr6nemo.collections.llm.recipes.precision.mixed_precisionr'nemo.collections.vlm.gemma3vl.data.mockr3nemo.lightning.pytorch.callbacks.garbage_collectionr6nemo.lightning.pytorch.callbacks.megatron_comm_overlapr/nemo.lightning.pytorch.callbacks.moe_token_droprnemo.utils.exp_managerrNAMEclifactoryr!LightningModuler)intdtypeboollistrZr[strrlrormrrr'r'r'r(s                  S 6-