o }oiO8@s:ddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZddlmZdd lmZdd lmZmZmZdd lmZdd lm Z dd l!m"Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+dZ,ej-j.e,ddej/ej0fddZ1            d)CallableOptionalN)Callback)DistributedDataParallelConfig) lightning)llmvlm) AutoTokenizer)pretrain) nemo_resume) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing) bf16_mixed)MockDataModule)GarbageCollectionCallback)MegatronCommOverlapCallback)MegatronTokenDropCallback)TimingCallbackllama4_omni_e128namereturncCsFtjtjtjtjttjttjtjtjdddddddddS)a Factory function to create a Llama4 128-Experts (Maverick) VLM model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Llama4 128-Experts (Maverick) VLM model model. Examples: CLI usage: $ nemo llm pretrain model=llama4_omni_e128 ... Python API usage: >>> model_config = model() >>> print(model_config) mcore_affineiiF)projector_type input_size hidden_sizeffn_hidden_sizebiasbias_activation_fusion)language_transformer_configvision_transformer_configvision_projection_config)config) runConfigrLlama4OmniModelLlama4MaverickExperts128ConfigrLlama4Experts128ConfigLlama4VisionConfigMultimodalProjectorConfigr,r,a/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/vlm/recipes/llama4_omni_e128.pymodel)s"  r.T@{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismexpert_tensor_parallelismexpert_model_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csjtjtj||||||||dddtjtddddddddd} tjtjdd| | dd d | |t| d d d } | S)a~ Configure the NeMo Lightning Trainer for Llama4 128-Experts (Maverick) VLM model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=llama4_e128 ... Python API usage: >>> trainer_config = trainer(num_nodes=2, num_gpus_per_node=8) >>> print(trainer_config) Note: For more information on distributed training strategies, refer to the NeMo documentation on multi-GPU and multi-node training. Toptim_grads_params)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collectivedata_parallel_sharding_strategyN) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelexpert_tensor_parallel_sizeexpert_model_parallel_sizegradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpfsdpgpur02 Fi) acceleratoraccumulate_grad_batchesr@deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr?r=pluginsstrategyuse_distributed_samplerval_check_interval)r%r&nlMegatronStrategyrTrainerr)r5r6r7r8r9r:r;r<r=r>r?r@r`trainerr,r,r-rfMsR. rf)targetrdefaultFdirrperformance_modefnc Csbtj|tt||ttgdtjtddddt||t|ddt dd t d }|r/t |}|S) a Create a pre-training recipe for Llama4 128-Experts (Maverick) model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. performance_mode (bool): If true, enables optimizations for maximum performance. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory llama4_e128 ... $ nemo llm pretrain --factory "llama4_e128(num_nodes=2, name='my_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="llama4_e128_pretrain", num_nodes=2) >>> print(recipe) Note: For more details on pre-training LLMs with NeMo, see the pre-training guide in the `examples/llm/pretrain/` directory. )r=r>r@ ir0) seq_lengthglobal_batch_sizemicro_batch_sizerrirrga2U0*3?)max_lrr.rfdatalogoptimresume) r%Partialr.rfr&rLlama4MockDataModuler rrr "pretrain_performance_optimizations)rirr=r>rjrkreciper,r,r-pretrain_recipes ' r{rzcCsZ|jjsg|j_tjtddd}tjtdd}tt}|jj|||gd|jj_ |S)a Create a performance-optimized pre-training recipe for Llama4 128-Experts (Maverick) model. This method enables performance optimizations that may not be suitable for all use cases. It builds upon the standard pre-training recipe and adds additional performance enhancements. Args: recipe (run.Partial): Base pre-train recipe to which performance optimizations will be added Returns: run.Partial: Partial configuration for performance-optimized pre-training. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. d)gc_interval_traingc_interval_valTtp_comm_overlapF) rfr@r%r&rrrextendr_rC)rzgarbage_collection_callbackmcomm_overlap_callbacktoken_drop_callbackr,r,r-rys, rynone peft_schemecCsntjtjddddddtjtjtddddddd }tjtjdd|d dd |t|d tt tjt dd gd }tj t j t|tjtdddtjtdddddt j||t|ddtddddtdd}|dusm|dkrd|jj_d|jj_d|jj_d|jj_|S|dkrd|jj_d|jj_d|jj_d|jj_tjt j!dgdd |_"d!|jj_|St#d"|)#aA Create a fine-tuning recipe for Llava1.5 7B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory llama4_omni_e128 Python API usage: >>> recipe = finetune_recipe(name="llama4_omni_e128_finetune", num_nodes=1) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. For more information on fine-tuning LLMs with NeMo, see the fine-tuning guide in the `examples/llm/finetune/` directory. r/r1r0rT)rBrCrDrErF)rHrNrOrI$encoder_pipeline_model_parallel_sizerMrJrSrUrXiFir) rYrZr[r]r^r?r=r_r`rbr@rlz)meta-llama/Llama-4-Scout-17B-16E-Instruct)pretrained_model_nameN)rmrnro tokenizerimage_processor num_workersrrpgh㈵>gH׊>)rqmin_lr warmup_stepsrrrrWlorar3F)z*.language_model.*.linear_qkvz*.language_model.*.linear_qz*.language_model.*.linear_kvz*.language_model.*.linear_projz*.language_model.*.linear_fc1z*.language_model.*.linear_fc2)freeze_vision_modeltarget_modulesg-C6?zUnrecognized peft scheme: )$r%r&rcrdtorchbfloat16rrerrrrwrfinetuner.rxr r rrr lowerrfr`rH!expert_tensor_model_parallel_sizerOrur$lrrMrIrLoRApeft ValueError)rirr=r>rr`rfrzr,r,r-finetune_recipes&            r) r/r0NNr0r/r1Tr2r3r4N)Nrhr2r3r)>typingrrlightning.pytorchpytorchplnemo_runr%r$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrcnemo.collectionsrr"nemo.collections.common.tokenizersr nemo.collections.llm.apir -nemo.collections.llm.recipes.finetune_defaultr (nemo.collections.llm.recipes.log.defaultr r r'nemo.collections.llm.recipes.optim.adamr6nemo.collections.llm.recipes.precision.mixed_precisionr%nemo.collections.vlm.llama4.data.mockrrx3nemo.lightning.pytorch.callbacks.garbage_collectionr6nemo.lightning.pytorch.callbacks.megatron_comm_overlapr/nemo.lightning.pytorch.callbacks.moe_token_droprnemo.utils.exp_managerrNAMEclifactoryr&LightningModuler.intdtypeboollistrerfstrrwr{ryrrr,r,r,r-s               $     [ :-