o }oi5&@sddlmZmZddlmZmZmZddlmZ ddl Z ddl Z ddl mZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZmZmZdd l m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+dZ,eGdddej-Z.e j/j0e,d d8dee1de j2e j3fddZ4          d9d e1d!e1d"ee j5d#ee1d$e1d%e6d&e1d'e1d(e1d)ee7e j2ede j2ej8fd*d+Z9e j/j0ee,d,dd-dddddefd.ee:d/e:d&e1d'e1de1d0e6d1e6d2ede j;fd3d4Z.s zNevaConfig8B.)default_factorylanguage_transformer_configcCs tjddS)Nz!openai/clip-vit-large-patch14-336)pretrained_model_name_or_path)r HFCLIPVisionConfigrrrrr 0svision_transformer_configcCstjddddS)Nii) input_size hidden_sizeffn_hidden_size)r MultimodalProjectorConfigrrrrr 5svision_projection_configFfreeze_language_modelTfreeze_vision_modelfreeze_vision_projectionN)__name__ __module__ __qualname____doc__ transformersrrr"r __annotations__r%rr*r+boolr,r-rrrrr(s    rnamerrreturncCstjtjttd}|S)z Factory function to create a NeVA (CLIP-ViT-L + LLaMa3 8B) model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the NeVA (CLIP-ViT-L + LLaMa3 8B) model. )config)runConfigr NevaModelr)r model_configrrrmodel=s r=F{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csbtjtj||||||dddtjtddddddd } tjtjdd| |ddd ||t| dd d } | S) a} Configure the NeMo Lightning Trainer for NeVA (CLIP-ViT-L + LLaMa3 8B) model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm finetune trainer=neva_llama3_8b ... Python API usage: >>> trainer_config = trainer(num_nodes=2, num_gpus_per_node=8) >>> print(trainer_config) Note: For more information on distributed training strategies, refer to the NeMo documentation on multi-GPU and multi-node training. TF)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collective) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpur>2 i) acceleratoraccumulate_grad_batchesrKdeviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsrJrHpluginsstrategyuse_distributed_samplerval_check_interval)r9r:nlMegatronStrategyrTrainerr) rBrCrDrErFrGrHrIrJrKrftrainerrrrrlKsJ,rl)targetr6defaultdirr6performance_modepacked_sequencefnc Csjtj|t|dt||ttgdtjt|ddd|dt||t|ddt d d t d }|r3t |}|S) a Create a finetuning recipe for NeVA (CLIP-ViT-L + LLaMa3 8B) model. Using pre-train fn targets to use appropriate llm api call. This function sets up a complete configuration for finetuning, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the finetuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. performance_mode (bool): If true, enables optimizations for maximum performance. fn (Callable): The finetuning function to use. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory neva_llama3_8b Python API usage: >>> recipe = finetune_recipe(name="neva_llama3_8b_finetune", num_nodes=2) >>> print(recipe) r)rHrIrKir>)rglobal_batch_sizemicro_batch_size num_workersrqr5)ror6rga2U0*3?)max_lr)r=rldatalogoptimresume) r9Partialr=rlr:rrrrrr"finetune_performance_optimizations) ror6rHrIrrprqrrreciperrrfinetune_recipes.& rr~cCst|jdr |jjdurg|j_|jjjdkr$|jjtjtddn |jjtjtdd|jjtt |jjtt ddd|jj _ |S)a Create a performance-optimized finetuning recipe for Llama3 8B model. This method enables performance optimizations that may not be suitable for all use cases. It builds upon the standard finetuning recipe and adds additional performance enhancements. Args: recipe (run.Partial): Base finetune recipe to which performance optimizations will be added Returns: run.Partial: Partial configuration for performance-optimized finetuning. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. rKNr>T)tp_comm_overlapFd) hasattrrlrKrfrQappendr9r:rrrrerM)r~rrrr}s2 r})r) r>r>NNr?Fr>r@rAN)> dataclassesrrtypingrrrlightning.pytorchpytorchplnemo_runr9torch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedr,megatron.core.transformer.transformer_configr nemor rinemo.collectionsr nemo.collections.llm.apir $nemo.collections.llm.gpt.model.llamar(nemo.collections.llm.recipes.log.defaultrrr'nemo.collections.llm.recipes.optim.adamr6nemo.collections.llm.recipes.precision.mixed_precisionr#nemo.collections.vlm.neva.data.mockr3nemo.lightning.pytorch.callbacks.garbage_collectionr6nemo.lightning.pytorch.callbacks.megatron_comm_overlaprnemo.utils.exp_managerrNAME NevaConfigrclifactoryintr:LightningModuler=dtyper4listrkrlstrr|rr}rrrrs                    U @