o }oi-@s2ddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZddlmZmZdd lmZdd lmZdd lmZmZmZdd l m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)dZ*ej+j,e*ddej-ej.fddZ/ddej0ddddddddf de1de1deej2d ee1d!e1d"e3d#e1d$e1d%e1d&e1d'ee4ej-e dej-ej5fd(d)Z6ej+j,ee*d*dd+ddd,efd-ee7d.e7d$e1d%e1d/e3d0edej8fd1d2Z9d3ej8dej8fd4d5Z:ej+j,ee*d*  +   6 ,d;d-ee7d.e7d$e1d%e1d7ee7d8e3dej8fd9d:Z;dS)<)CallableOptionalN)Callback)DistributedDataParallelConfig) lightning)finetunepretrain)MockDataModule)MixtralConfig8x22B MixtralModel) PEFT_STR2CLS)default_finetune_recipe) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing)GarbageCollectionCallback)MegatronCommOverlapCallback)MegatronTokenDropCallback)TimingCallback mixtral_8x22bnamereturncCstjtttdS)aj Factory function to create a Mixtral 8x22B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Mixtral 8x22B model. Examples: CLI usage: $ nemo llm pretrain model=mixtral_8x22b ... Python API usage: >>> model_config = model() >>> print(model_config) )config)runConfigr r rr^/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/mixtral_8x22b.pymodel(srTi{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelismexpert_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csntjtj|||||||dddtjtddddddd } tjtjdd| |ddd| |tjtjd d | d d d } | S)a Configure the NeMo Lightning Trainer for Mixtral 8x22B model. This function sets up the distributed training strategy optimized for the large Mixtral 8x22B model. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. expert_parallelism (int): Degree of expert parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=mixtral_8x22b ... Python API usage: >>> trainer_config = trainer(num_nodes=16, num_gpus_per_node=8) >>> print(trainer_config) Note: This configuration uses extensive parallelism to handle the large model size efficiently. T)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collective) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelexpert_model_parallel_sizegradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpu2 z bf16-mixed) precisionFi) acceleratoraccumulate_grad_batchesr/deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr.r,pluginsstrategyuse_distributed_samplerval_check_interval)rrnlMegatronStrategyrTrainerMegatronMixedPrecision) r%r&r'r(r)r*r+r,r-r.r/rMtrainerrrrrT;sL-rT)targetrdefaultFdirrperformance_modefnc Csbtj|tt||ttgdtjtddddt||t|ddt dd t d }|r/t |}|S) a Create a pre-training recipe for Mixtral 8x22B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. performance_mode (bool): If true, enables optimizations for maximum performance. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory mixtral_8x22b $ nemo llm pretrain --factory "mixtral_8x22b(num_nodes=16, name='my_mixtral_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="mixtral_pretrain", num_nodes=16) >>> print(recipe) )r,r-r/iirA) seq_lengthglobal_batch_sizemicro_batch_sizer)rWrrga2U0*3?)max_lr)rrTdatalogoptimresume) rPartialrrTrrr rrrr"pretrain_performance_optimizations)rWrr,r-rXrYreciperrrpretrain_recipes# rerdcCst|jjsg|j_tjtddd}tjtdd}|jjtt||gd|jj_ d|jj_ d|jj_ d|jj _ |S)a Create a performance-optimized pre-training recipe for Mixtral 8x22B model. This method enables performance optimizations that may not be suitable for all use cases. It builds upon the standard pre-training recipe and adds additional performance enhancements. Args: recipe (run.Partial): Base pre-train recipe to which performance optimizations will be added Returns: run.Partial: Partial configuration for performance-optimized pre-training. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. d)gc_interval_traingc_interval_valF)(overlap_param_gather_with_optimizer_steprAr#T)rTr/rrrrextendrrMr;r5r:rLr1)rdgarbage_collection_callbackmcomm_overlap_callbackrrrrcs,    rclora peft_schemepacked_sequencecCsttd|||||}d|jj_d|jj_|dus|dkr0d|jj_d|jj_d|j j _ |S|dvrLt j t|d d gd d |_d |j j _ |Std|)aH Create a fine-tuning recipe for Mixtral 8x22B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. packed_sequence (Optional[bool]): If true, fine-tuning sequences will be packed into batches up to the given maximum seq_length for better efficiency. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory mixtral_8x22b $ nemo llm finetune --factory "mixtral_8x22b(num_nodes=2, name='my_mixtral_finetune')" Python API usage: >>> recipe = finetune_recipe(name="mixtral_finetune", num_nodes=2) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. zmistralai/Mixtral-8x22B-v0.1r#Nnoner!r"gh㈵>)rmdora linear_qkv linear_projrC)target_modulesdimg-C6?zUnrecognized peft scheme: )r rrTrMr;r5lowerr6r8r`rlrrrr peft ValueError)rWrr,r-rnrordrrrfinetune_recipes")       rz)NrVr#r#rmF)<typingrrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrPnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr &nemo.collections.llm.gpt.model.mixtralr r nemo.collections.llm.peftr -nemo.collections.llm.recipes.finetune_defaultr (nemo.collections.llm.recipes.log.defaultrrr'nemo.collections.llm.recipes.optim.adamr3nemo.lightning.pytorch.callbacks.garbage_collectionr6nemo.lightning.pytorch.callbacks.megatron_comm_overlapr/nemo.lightning.pytorch.callbacks.moe_token_droprnemo.utils.exp_managerrNAMEclifactoryrLightningModulerbfloat16intdtypeboollistrRrTstrrbrercrzrrrrs                 W 40