o }oi3@s8ddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZddlmZdd lmZdd lmZmZmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*dZ+ej,j-e+ddej.ej/fddZ0          d9de1de1deej2dee1de1d e3d!e1d"e1d#e1d$ee4ej.e dej.ej5fd%d&Z6ej,j-ee+d'dd(ddd)efd*ee7d+e7d!e1d"e1d,e3d-edej8fd.d/Z9d0ej8dej8fd1d2Z:ej,j-ej;e+d'  ( 3   4d:d*ee7d+e7d5ee7d!e1d"e1d6ee7dej8fd7d8ZtjtjtjtjttjttjtjtjdddddS)ai Factory function to create a Gemma3 VL 27B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Gemma3 VL 27B model. Examples: CLI usage: $ nemo llm pretrain model=gemma3vl_27b ... Python API usage: >>> model_config = model() >>> print(model_config) ii) input_size hidden_size)language_transformer_configvision_transformer_configvision_projection_config)config) runConfigr Gemma3VLModelGemma3VLConfig27BrGemma3Config27BGemma3VLVisionConfig!Gemma3VLMultimodalProjectorConfigr'r']/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/vlm/recipes/gemma3vl_27b.pymodel)s  r)T{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csftjtj||||||dddtjtddddddddd } tjtjdd| |dd d ||t| d d d } | S)a Configure the NeMo Lightning Trainer for Gemma3 VL 27B model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=gemma3vl_27b ... Python API usage: >>> trainer_config = trainer(num_nodes=1, num_gpus_per_node=8) >>> print(trainer_config) Toptim_grads_params)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collectivedata_parallel_sharding_strategyN) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpfsdpgpur+2 Fi) acceleratoraccumulate_grad_batchesr6deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr5r3pluginsstrategyuse_distributed_samplerval_check_interval)r r!nlMegatronStrategyrTrainerr) r-r.r/r0r1r2r3r4r5r6rTtrainerr'r'r(rZHsN( rZ)targetrdefaultFdirrperformance_modefnc Csbtj|tt||ttgdtjtddddt||t|ddt ddt d }|r/t |}|S) a Create a pre-training recipe for Gemma3 VL 27B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. performance_mode (bool): If true, enables optimizations for maximum performance. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory gemma3vl_27b ... $ nemo llm pretrain --factory "gemma3vl_27b(num_nodes=2, name='my_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="gemma3vl_27b_pretrain", num_nodes=2) >>> print(recipe) )r3r4r6r+) seq_lengthglobal_batch_sizemicro_batch_sizerr]rrga2U0*3?)max_lrr)rZdatalogoptimresume) r Partialr)rZr!rrr rrr "pretrain_performance_optimizations)r]rr3r4r^r_reciper'r'r(pretrain_recipes # rnrmcCsZ|jjsg|j_tjtddd}tjtdd}tt}|jj|||gd|jj_ |S)a Create a performance-optimized pre-training recipe for Gemma3 VL 27B model. This method enables performance optimizations that may not be suitable for all use cases. It builds upon the standard pre-training recipe and adds additional performance enhancements. Args: recipe (run.Partial): Base pre-train recipe to which performance optimizations will be added Returns: run.Partial: Partial configuration for performance-optimized pre-training. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. d)gc_interval_traingc_interval_valT)tp_comm_overlapF) rZr6r r!rrrextendrSr9)rmgarbage_collection_callbackmcomm_overlap_callbacktoken_drop_callbackr'r'r(rls, rlgoogle/gemma-3-27b-itnone resume_path peft_schemec Cs@tjtjfddtjdddddddddddtjtddddddd}tjtjdd|d dd |t|d tt gd }tj t j t |tjtd ddtjtddddt j||t|ddtddddt|d}|duso|dkr{d|jj_d|jj_|S|dkrd|jj_tjtjdgdd|_d |jj_|Std!|)"a Create a fine-tuning recipe for Gemma3 VL 27B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. resume_path (str): Path to the NeMo checkpoint name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Parameter efficient fine-tuning scheme to use. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory gemma3vl_27b Python API usage: >>> recipe = finetune_recipe(name="gemma3vl_27b_finetune", num_nodes=1) >>> print(recipe) r+NrTlog_all)r8r9r:r;r<)r>r?r@rA"encoder_tensor_model_parallel_size$encoder_pipeline_model_parallel_sizerBrCrEckpt_parallel_saverFckpt_parallel_save_optimckpt_load_strictnessrDrGrIrLiFi) rMrNrOrQrRr5r3rSrTrVr6r`rKrw)pretrained_model_name)rarbrc tokenizer num_workersrrdgh㈵>gH׊>)remin_lr warmup_stepsrfrxr*loraF)z*.language_model.*.linear_qkvz*.language_model.*.linear_qz*.language_model.*.linear_kvz*.language_model.*.linear_projz*.language_model.*.linear_fc1z*.language_model.*.linear_fc2)freeze_vision_modeltarget_modulesg-C6?zUnrecognized peft scheme: )r r!rWrXtorchbfloat16rrYrrrkrfinetuner)rr r rrr lowerrZrTr>rirlrrLoRApeft ValueError) r]rryr3r4rzrTrZrmr'r'r(finetune_recipes$      r) r*r+NNr+Tr+r*r,N)Nr\rwr+r*rx)=typingrrlightning.pytorchpytorchplnemo_runr r$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrWnemo.collectionsrr"nemo.collections.common.tokenizersr nemo.collections.llm.apir -nemo.collections.llm.recipes.finetune_defaultr (nemo.collections.llm.recipes.log.defaultr r r'nemo.collections.llm.recipes.optim.adamr6nemo.collections.llm.recipes.precision.mixed_precisionr'nemo.collections.vlm.gemma3vl.data.mockr3nemo.lightning.pytorch.callbacks.garbage_collectionr6nemo.lightning.pytorch.callbacks.megatron_comm_overlapr/nemo.lightning.pytorch.callbacks.moe_token_droprnemo.utils.exp_managerrNAMEclifactoryr!LightningModuler)intdtypeboollistrYrZstrrkrnrlrrr'r'r'r(s                  S 6-