o }oi:>@srddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZm Z m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-dZ.ej/j0e.ddej1ej2fddZ3ddej4dddddddf de5de5d eej6d!ee5d"e5d#e7d$e5d%e5d&e5d'ee8ej1e dej1ej9fd(d)Z:ej/j0ee.d*dd+d,dd-efd.ee;d/e;d$e5d%e5d0e7d1edejej/j0ee.d*  +   7   -d?d.ee;d/e;d$e5d%e5d8ee;d9ee5d:ee7d0e7dejZ@dS)@)CallableOptionalN)Callback)DistributedDataParallelConfig) lightning)finetunepretrain)MockDataModule)PackedSequenceSpecs)Llama3Config70B LlamaModel) PEFT_STR2CLS)default_finetune_recipe) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing) bf16_mixed)/userbuffers_bf16_h100_h8192_tp4_mbs1_seqlen8192)GarbageCollectionCallback)MegatronCommOverlapCallback)TimingCallback llama3_70bnamereturncCstjtttdS)aa Factory function to create a Llama3 70B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Llama3 70B model. Examples: CLI usage: $ nemo llm pretrain model=llama3_70b ... Python API usage: >>> model_config = model() >>> print(model_config) )config)runConfigr r rr[/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/llama3_70b.pymodel*sr!Ti{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csbtjtj||||||dddtjtddddddd } tjtjdd| |ddd||t| d d d } | S) a= Configure the NeMo Lightning Trainer for Llama3 70B model. This function sets up the distributed training strategy optimized for the large 70B model. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=llama3_70b ... Python API usage: >>> trainer_config = trainer(num_nodes=4, num_gpus_per_node=8) >>> print(trainer_config) Note: This configuration uses extensive parallelism to handle the large model size efficiently. Tcheck_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collective) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpu2 Fi) acceleratoraccumulate_grad_batchesr/deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr.r,pluginsstrategyuse_distributed_samplerval_check_interval)rrnlMegatronStrategyrTrainerr) r&r'r(r)r*r+r,r-r.r/rLtrainerrrr rR=sJ+rR)targetrdefaultrAFdirrperformance_modefnc Csbtj|tt||ttgdtjtddddt||t|ddt dd t d }|r/t |}|S) a_ Create a pre-training recipe for Llama3 70B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. performance_mode (bool): If true, enables optimizations for maximum performance. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory llama3_70b $ nemo llm pretrain --factory "llama3_70b(num_nodes=4, name='my_70b_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="llama3_70b_pretrain", num_nodes=4) >>> print(recipe) Note: This recipe is optimized for the large 70B model and requires significant computational resources. )r,r-r/i irA) seq_lengthglobal_batch_sizemicro_batch_sizer)rUrrga2U0*3?)max_lr)r!rRdatalogoptimresume) rPartialr!rRrrr rrrr"pretrain_performance_optimizations)rUrr,r-rVrWreciperrr pretrain_recipes ' rcrbcCs`|jjsg|j_tjtddd}tjtdtdddd}|jj||gd|jj_ d|j j _ |S)a Create a performance-optimized pre-training recipe for Llama3 70B model. This method enables performance optimizations that may not be suitable for all use cases. It builds upon the standard pre-training recipe and adds additional performance enhancements. Args: recipe (run.Partial): Base pre-train recipe to which performance optimizations will be added Returns: run.Partial: Partial configuration for performance-optimized pre-training. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. d)gc_interval_traingc_interval_valTF)tp_comm_overlaptp_comm_overlap_cfgdefer_embedding_wgrad_computewgrad_deferral_limit(overlap_param_gather_with_optimizer_step) rRr/rrrrrextendrKr2r^ruse_precision_aware_optimizer)rbgarbage_collection_callbackmcomm_overlap_callbackrrr ras.   ralora peft_schemerXpacked_sequencec CsJ|dur|}|dur|rdnd}|dur)|dus|dkr!d}n|dvr)d}ttd|||||}|dus>|dkrNd |jj_d|jj_d |jj_ n4|dvr{t t ||_ d |j _d |j _d |jj_d |jj_d |jj_d|jj_ ntd|||jj_||j_|rddi|j_t j t|d|j_|rt||}|S)af Create a fine-tuning recipe for Llama3 70B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. seq_length (int): Maximum number of tokens per microbatch. packed_sequence (Optional[bool]): If true, fine-tuning sequences will be packed into batches up to the given maximum seq_length for better efficiency. By default, this value equals performance_mode. performance_mode (bool): If true, enables optimizations for maximum performance. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory llama3_70b $ nemo llm finetune --factory "llama3_70b(num_nodes=4, name='my_70b_finetune')" Python API usage: >>> recipe = finetune_recipe(name="llama3_70b_finetune", num_nodes=4) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. Be aware that fine-tuning a 70B model requires substantial computational resources. Niinoner")rqdorarAzmeta-llama/Meta-Llama-3-70Br%gh㈵>rCFg-C6?zUnrecognized peft scheme: pad_to_max_lengthT)packed_sequence_size)lowerrr!rRrLr6r7r^rlrrrr peftdimalphause_distributed_optimizercross_entropy_loss_fusion ValueErrorrXr\dataset_kwargsr packed_sequence_specs"finetune_performance_optimizations) rUrr,r-rrrXrsrVrbrrr finetune_recipesB0             rcCst|jdr |jjdurg|j_|dus|dkrFd|jj_d|jj_d|jj_tj t dddddd|jj_ |jj tj t ddd d n d |jj_d|jj_d|jj_d g|j_|jj tj t dd d|jj_d|jj_|jj t t|jj t tddd|jj_|S)a Modify the given recipe to optimize settings for performance. This method enables performance optimizations that may not be suitable for all use cases. Intended to build upon the standard fine-tuning recipe. Args: recipe (run.Partial): Base fine-tuning recipe to which performance optimizations will be added peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. Returns: run.Partial: Partial configuration for performance-optimized fine-tuning. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. r/Nrtr"r#TFr0rg)rhrjrkr$ linear_qkv)rhrd)hasattrrRr/ryrLr6r7r9rrrr?appendrr{target_modulesrKr2r;rrr^rrn)rbrrrrr r[sX          r)NrTNr%rqNNF)Atypingrrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrOnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr -nemo.collections.llm.gpt.data.packed_sequencer $nemo.collections.llm.gpt.model.llamar r nemo.collections.llm.peftr -nemo.collections.llm.recipes.finetune_defaultr(nemo.collections.llm.recipes.log.defaultrrr'nemo.collections.llm.recipes.optim.adamr6nemo.collections.llm.recipes.precision.mixed_precisionr;nemo.collections.llm.recipes.tp_overlap_configs.userbuffersr3nemo.lightning.pytorch.callbacks.garbage_collectionr6nemo.lightning.pytorch.callbacks.megatron_comm_overlaprnemo.utils.exp_managerrNAMEclifactoryrLightningModuler!bfloat16intdtypeboollistrQrRstrr`rcrarrrrrr s                  T :0 ^