o wi,@s2ddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZddlmZmZdd lmZdd lmZdd lmZmZmZdd l m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)dZ*ej+j,e*ddej-ej.fddZ/ddej0ddddddddf de1de1deej2dee1de1d e3d!e1d"e1d#e1d$e1d%ee4ej-e dej-ej5fd&d'Z6ej+j,ee*d(dd)dddefd*ee7d+e7d"e1d#e1d,e3d-edej8fd.d/Z9d0ej8dej8fd1d2Z:ej+j,ee*d(  )   3 d8d*ee7d+e7d"e1d#e1d4ee7d5e3dej8fd6d7Z;dS)9)CallableOptionalN)Callback)DistributedDataParallelConfig) lightning)finetunepretrain)MockDataModule)MixtralConfig8x7B MixtralModel) PEFT_STR2CLS)default_finetune_recipe) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing)GarbageCollectionCallback)MegatronCommOverlapCallback)MegatronTokenDropCallback)TimingCallback mixtral_8x7bnamereturncCstjtttdS)ag Factory function to create a Mixtral 8x7B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Mixtral 8x7B model. Examples: CLI usage: $ nemo llm pretrain model=mixtral_8x7b ... Python API usage: >>> model_config = model() >>> print(model_config) )config)runConfigr r rrf/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/recipes/mixtral_8x7b.pymodel(srFi{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelismexpert_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csntjtj|||||||dddtjtddddddd } tjtjdd| |ddd| |tjtjd d | d d d } | S)a Configure the NeMo Lightning Trainer for Mixtral 8x7B model. This function sets up the distributed training strategy optimized for the Mixtral 8x7B model. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. expert_parallelism (int): Degree of expert parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=mixtral_8x7b ... Python API usage: >>> trainer_config = trainer(num_nodes=2, num_gpus_per_node=8) >>> print(trainer_config) T)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collective) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelexpert_model_parallel_sizegradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpur 2 z bf16-mixed) precisionFi) acceleratoraccumulate_grad_batchesr-deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr,r*pluginsstrategyuse_distributed_samplerval_check_interval)rrnlMegatronStrategyrTrainerMegatronMixedPrecision) r#r$r%r&r'r(r)r*r+r,r-rJtrainerrrrrQ;sL*rQ)targetrdefaultdirrperformance_modefnc Csbtj|tt||ttgdtjtddddt||t|ddt dd t d }|r/t |}|S) a Create a pre-training recipe for Mixtral 8x7B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. performance_mode (bool): If true, enables optimizations for maximum performance. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory mixtral_8x7b $ nemo llm pretrain --factory "mixtral_8x7b(num_nodes=8, name='my_mixtral_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="mixtral_8x7b_pretrain", num_nodes=8) >>> print(recipe) )r*r+r-iir ) seq_lengthglobal_batch_sizemicro_batch_sizer)rTrrga2U0*3?)max_lr)rrQdatalogoptimresume) rPartialrrQrrr rrrr"pretrain_performance_optimizations)rTrr*r+rUrVreciperrrpretrain_recipes# rbracCsntjtddd}tjtdd}|jjtt||gd|jj_ d|jj_ d|jj_ d|jj _ d|jj_|S)a Create a performance-optimized pre-training recipe for Mixtral 8x7B model. This method enables performance optimizations that may not be suitable for all use cases. It builds upon the standard pre-training recipe and adds additional performance enhancements. Args: recipe (run.Partial): Base pre-train recipe to which performance optimizations will be added Returns: run.Partial: Partial configuration for performance-optimized pre-training. Note: Use this method with caution and only when you need maximum performance. It may not be suitable for all hardware configurations or use cases. d)gc_interval_traingc_interval_valF)(overlap_param_gather_with_optimizer_stepr r"T)rrrrrQr-extendrrJr9r3r8rIr/r]ruse_precision_aware_optimizer)ragarbage_collection_callbackmcomm_overlap_callbackrrrr`s*     r`lora peft_schemepacked_sequencecCsttd|||||}d|jj_|dus|dkr+d|jj_d|jj_d|jj _ |S|dvrGt j t |dd gd d |_d |jj _ |Std |)aJ Create a fine-tuning recipe for Mixtral 8x7B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. packed_sequence (Optional[bool]): Packing multiple training sequences into one long sequence for training efficiency. Default sequence length is 2048. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory mixtral_8x7b $ nemo llm finetune --factory "mixtral_8x7b(num_nodes=2, name='my_mixtral_finetune')" Python API usage: >>> recipe = finetune_recipe(name="mixtral_8x7b_finetune", num_nodes=2) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. zmistralai/Mixtral-8x7B-v0.1r"Nnoner!gh㈵>)rkdora linear_qkv linear_projr@)target_modulesdimg-C6?zUnrecognized peft scheme: )r rrQrJr9lowerr4r6r]rlrrrr peft ValueError)rTrr*r+rlrmrarrrfinetune_recipes )      rx)NrSr r"rkF)<typingrrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrMnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr &nemo.collections.llm.gpt.model.mixtralr r nemo.collections.llm.peftr -nemo.collections.llm.recipes.finetune_defaultr (nemo.collections.llm.recipes.log.defaultrrr'nemo.collections.llm.recipes.optim.adamr3nemo.lightning.pytorch.callbacks.garbage_collectionr6nemo.lightning.pytorch.callbacks.megatron_comm_overlapr/nemo.lightning.pytorch.callbacks.moe_token_droprnemo.utils.exp_managerrNAMEclifactoryrLightningModulerbfloat16intdtypeboollistrOrQstrr_rbr`rxrrrrs                 T 4.