o }oi?@sddlmZmZddlmZddlZddlm Z m Z ddl m Z ddl mZddlmZmZddlmZddlmZdd lmZdd lmZmZmZdd lmZdd lmZdd l m!Z!dZ"ej#j$e"ddej%ej&fddZ'ej#j$e e"ddddde fdee(de(de)de)dedej*f ddZ+ej#j$e e"d       d$dee(de(de)de)dee(d ee)d!ee,dej*fd"d#Z-dS)%)CallableOptionalN)finetunepretrain)MockDataModule)PackedSequenceSpecs) DeepSeekModelDeepSeekV2LiteConfig) PEFT_STR2CLS)trainer)default_finetune_recipe) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing)DeepEPCallback)TimingCallbackdeepseek_v2_litenamereturncCstt}tjt|dS)ay Factory function to create a DeepSeek-V2-Lite (16B) model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the DeepSeek V2 Lite model. Examples: CLI usage: $ nemo llm pretrain model=deepseek_v2_lite ... Python API usage: >>> model_config = model() >>> print(model_config) )config)runConfigr r)confra/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/deepseek_v2_lite.pymodel$s r)targetrdefaultdirr num_nodesnum_gpus_per_nodefnc Csrtj|ttdd||ttgdtjtddddt||t|ddt d d t d }tt }|jj ||S) a  Create a pre-training recipe for DeepSeek-V2-Lite (16B) model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. performance_mode (bool): If true, enables optimizations for maximum performance. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory deepseek_v2_lite $ nemo llm pretrain --factory "deepseek_v2_lite(num_nodes=4, name='my_deepseek_v2_lite')" Python API usage: >>> recipe = pretrain_recipe(name="deepseek_v2_lite_pretrain", num_nodes=4) >>> print(recipe) r r!)tensor_parallelismexpert_parallelismr#r$ callbacksii) seq_lengthglobal_batch_sizemicro_batch_sizer)r"rrga2U0*3?)max_lr)rr datalogoptimresume)rPartialrr rrrr rrrrr(append)r"rr#r$r%recipedeepep_callbackrrrpretrain_recipe8s$#  r5lora peft_schemer)packed_sequencecCs|durd}|dur d}ttd|||||}|dus!|dkr1d|jj_d|jj_d|jj_ n-|d vrWt t ||_ gd |j _d |jj_d |jj_d |jj_ ntd |||jj_||j_|rxddi|j_t j t|d|j_|S)a Create a fine-tuning recipe for DeepSeek-V2-Lite (16B) model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. seq_length (int): Maximum number of tokens per microbatch. packed_sequence (Optional[bool]): If true, fine-tuning sequences will be packed into batches up to the given maximum seq_length for better efficiency. By default, this value equals performance_mode. performance_mode (bool): If true, enables optimizations for maximum performance. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory deepseek_v2_lite $ nemo llm finetune --factory "deepseek_v2_lite(num_nodes=3, name='my_deepseek_v2_lite_finetune')" Python API usage: >>> recipe = finetune_recipe(name="deepseek_v2_lite_finetune", num_nodes=2) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. Be aware that fine-tuning the DeepSeek-V2 model requires substantial computational resources. Nir zdeepseek-ai/DeepSeek-V2-LitenoneTgh㈵>)r6dora) linear_q_projlinear_kv_down_projlinear_kv_up_proj linear_projFg-C6?zUnrecognized peft scheme: pad_to_max_length)packed_sequence_size)r rlowerr strategytensor_model_parallel_sizesequence_parallelr/rlrrrr pefttarget_modulesuse_distributed_optimizercross_entropy_loss_fusion ValueErrorr)r-dataset_kwargsrpacked_sequence_specs)r"rr#r$r7r)r8r3rrrfinetune_recipeps0-          rN)Nrr r!r6NN).typingrrlightning.pytorchpytorchplnemo_runrnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr-nemo.collections.llm.gpt.data.packed_sequencer'nemo.collections.llm.gpt.model.deepseekrr nemo.collections.llm.peftr %nemo.collections.llm.recipes.deepseekr -nemo.collections.llm.recipes.finetune_defaultr (nemo.collections.llm.recipes.log.defaultr rr'nemo.collections.llm.recipes.optim.adamr'nemo.lightning.pytorch.callbacks.deepeprnemo.utils.exp_managerrNAMEclifactoryrLightningModulerstrintr1r5boolrNrrrrsz           7