o }oi18@s:ddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZddlmZdd lmZdd lmZmZmZdd lmZdd lm Z dd l!m"Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+dZ,ej-j.e,ddej/ej0fddZ1            d)CallableOptionalN)Callback)DistributedDataParallelConfig) lightning)llmvlm) AutoTokenizer)pretrain) nemo_resume) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing) bf16_mixed)MockDataModule)GarbageCollectionCallback)MegatronCommOverlapCallback)MegatronTokenDropCallback)TimingCallbackllama4_omni_e16namereturncCsFtjtjtjtjttjttjtjtjdddddddddS)a Factory function to create a Llama4 16-Experts (Scout) VLM model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Llama4 16-Experts (Scout) VLM model model. Examples: CLI usage: $ nemo llm pretrain model=llama4_omni_e16 ... Python API usage: >>> model_config = model() >>> print(model_config) mcore_affineiiF)projector_type input_size hidden_sizeffn_hidden_sizebiasbias_activation_fusion)language_transformer_configvision_transformer_configvision_projection_config)config) runConfigrLlama4OmniModelLlama4ScoutExperts16ConfigrLlama4Experts16ConfigLlama4VisionConfigMultimodalProjectorConfigr,r,`/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/vlm/recipes/llama4_omni_e16.pymodel)s"  r.T {tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelismexpert_tensor_parallelismexpert_model_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csjtjtj||||||||dddtjtddddddddd} tjtjdd| | dd d | |t| d d d } | S)az Configure the NeMo Lightning Trainer for Llama4 16-Experts (Scout) VLM model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=llama4_e128 ... Python API usage: >>> trainer_config = trainer(num_nodes=2, num_gpus_per_node=8) >>> print(trainer_config) Note: For more information on distributed training strategies, refer to the NeMo documentation on multi-GPU and multi-node training. Toptim_grads_params)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collectivedata_parallel_sharding_strategyN) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelexpert_tensor_parallel_sizeexpert_model_parallel_sizegradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpfsdpgpur02r2 Fi) acceleratoraccumulate_grad_batchesr@deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr?r=pluginsstrategyuse_distributed_samplerval_check_interval)r%r&nlMegatronStrategyrTrainerr)r5r6r7r8r9r:r;r<r=r>r?r@r_trainerr,r,r-reMsR. re)targetrdefaultFdirrperformance_modefnc Csbtj|tt||ttgdtjtddddt||t|ddt dd t d }|r/t |}|S) a Create a pre-training recipe for Llama4 128-Experts (Maverick) model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. performance_mode (bool): If true, enables optimizations for maximum performance. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory llama4_e128 ... $ nemo llm pretrain --factory "llama4_e128(num_nodes=2, name='my_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="llama4_e128_pretrain", num_nodes=2) >>> print(recipe) Note: For more details on pre-training LLMs with NeMo, see the pre-training guide in the `examples/llm/pretrain/` directory. )r=r>r@ ir0) seq_lengthglobal_batch_sizemicro_batch_sizerrhrrga2U0*3?)max_lrr.redatalogoptimresume) r%Partialr.rer&rLlama4MockDataModuler rrr "pretrain_performance_optimizations)rhrr=r>rirjreciper,r,r-pretrain_recipes ' rzrycCsZ|jjsg|j_tjtddd}tjtdd}tt}|jj|||gd|jj_ |S)a Create a performance-optimized pre-training recipe for Llama4 128-Experts (Maverick) model. This method enables performance optimizations that may not be suitable for all use cases. It builds upon the standard pre-training recipe and adds additional performance enhancements. Args: recipe (run.Partial): Base pre-train recipe to which performance optimizations will be added Returns: run.Partial: Partial configuration for performance-optimized pre-training. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. d)gc_interval_traingc_interval_valTtp_comm_overlapF) rer@r%r&rrrextendr^rC)rygarbage_collection_callbackmcomm_overlap_callbacktoken_drop_callbackr,r,r-rxs, rxnone peft_schemecCsntjtjddddddtjtjtddddddd }tjtjdd|d dd |t|d tt tjt dd gd }tj t j t|tjtdddtjtdddddt j||t|ddtddddtdd}|dusm|dkrd|jj_d|jj_d|jj_d|jj_d|jj_|S|dkrd|jj_d|jj_d|jj_tjtj dgdd |_!d!|jj_|St"d"|)#a? Create a fine-tuning recipe for Llava1.5 7B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory llama4_omni_e16 Python API usage: >>> recipe = finetune_recipe(name="llama4_omni_e16_finetune", num_nodes=1) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. For more information on fine-tuning LLMs with NeMo, see the fine-tuning guide in the `examples/llm/finetune/` directory. r/r1r0rT)rBrCrDrErF)rHrNrOrI$encoder_pipeline_model_parallel_sizerMrJrSrUrWiFir~) rXrYrZr\r]r?r=r^r_rar@rkz)meta-llama/Llama-4-Scout-17B-16E-Instruct)pretrained_model_nameN)rlrmrn tokenizerimage_processor num_workersrrogh㈵>gH׊>)rpmin_lr warmup_stepsrqrr3loraF)z*.language_model.*.linear_qkvz*.language_model.*.linear_qz*.language_model.*.linear_kvz*.language_model.*.linear_projz*.language_model.*.linear_fc1z*.language_model.*.linear_fc2)freeze_vision_modeltarget_modulesg-C6?zUnrecognized peft scheme: )#r%r&rbrctorchbfloat16rrdrrrrvrfinetuner.rwr r rrr lowerrer_rMrH!expert_tensor_model_parallel_sizerIrtr$lrrLoRApeft ValueError)rhrr=r>rr_reryr,r,r-finetune_recipes&            r) r/r0NNr0Tr/r1r2r3r4N)Nrgr2r3r)>typingrrlightning.pytorchpytorchplnemo_runr%r$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrbnemo.collectionsrr"nemo.collections.common.tokenizersr nemo.collections.llm.apir -nemo.collections.llm.recipes.finetune_defaultr (nemo.collections.llm.recipes.log.defaultr r r'nemo.collections.llm.recipes.optim.adamr6nemo.collections.llm.recipes.precision.mixed_precisionr%nemo.collections.vlm.llama4.data.mockrrw3nemo.lightning.pytorch.callbacks.garbage_collectionr6nemo.lightning.pytorch.callbacks.megatron_comm_overlapr/nemo.lightning.pytorch.callbacks.moe_token_droprnemo.utils.exp_managerrNAMEclifactoryr&LightningModuler.intdtypeboollistrdrestrrvrzrxrrr,r,r,r-s               $     [ :-