o }oi@@sddlmZmZddlmZddlZddlm Z m Z ddl m Z ddl mZddlmZmZddlmZddlmZdd lmZdd lmZmZmZdd lmZdd lmZdd l m!Z!dZ"ej#j$e"ddej%ej&fddZ'ej#j$e e"ddddde fdee(de(de)de)dedej*f ddZ+ej#j$e e"d       d'dee(d!e(de(de)de)d"ee(d#ee)d$ee,dej*fd%d&Z-dS)()CallableOptionalN)finetunepretrain)MockDataModule)PackedSequenceSpecs) DeepSeekModelDeepSeekV2Config) PEFT_STR2CLS)trainer)default_finetune_recipe) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing)DeepEPCallback)TimingCallback deepseek_v2namereturncCstt}tjt|dS)ak Factory function to create a DeepSeek-V2 (236B) model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the DeepSeek V2 model. Examples: CLI usage: $ nemo llm pretrain model=deepseek_v2 ... Python API usage: >>> model_config = model() >>> print(model_config) )config)runConfigr r)confr\/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/deepseek_v2.pymodel$s r)targetrdefaultdirr num_nodesnum_gpus_per_nodefnc Cstj|ttddd||ttgdtjtddddt||t|dd t d d t d }d |jj _ d|jj _ d|jj _tt}|jj||S)a Create a pre-training recipe for DeepSeek-V2 (236B) model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. performance_mode (bool): If true, enables optimizations for maximum performance. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory deepseek_v2 $ nemo llm pretrain --factory "deepseek_v2(num_nodes=16, name='my_deepseek_v2')" Python API usage: >>> recipe = pretrain_recipe(name="deepseek_v2_pretrain", num_nodes=16) >>> print(recipe)  )tensor_parallelismpipeline_parallelismexpert_parallelismr#r$ callbacksii) seq_lengthglobal_batch_sizemicro_batch_sizer)r"rrga2U0*3?)max_lr)rr datalogoptimresumefulluniform)rPartialrr rrrr rrrrrecompute_granularityrecompute_methodrecompute_num_layersrr,append)r"rr#r$r%recipedeepep_callbackrrrpretrain_recipe8s,#     r>deepseek-ai/DeepSeek-V2lora resume_path peft_schemer-packed_sequencec Cs8|durd}|dur|dus|dkrd}n|dvrd}tt||||||}|dus4|dkrDd|jj_d|jj_d |jj_ n<|dvryt t ||_ gd |j _d |jj_d |jj_d |jj_d |jj_d|jj_d |jj_ ntd|||jj_||j_|rddi|j_t j t|d|j_|S)a` Create a fine-tuning recipe for DeepSeek-V2 (236B) model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. resume_path (str): Path to the NeMo checkpoint name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. seq_length (int): Maximum number of tokens per microbatch. packed_sequence (Optional[bool]): If true, fine-tuning sequences will be packed into batches up to the given maximum seq_length for better efficiency. By default, this value equals performance_mode. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory deepseek_v2 $ nemo llm finetune --factory "deepseek_v2(num_nodes=2, name='my_deepseek_v2_finetune')" Python API usage: >>> recipe = finetune_recipe(name="deepseek_v2_finetune", num_nodes=2) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. Be aware that fine-tuning the DeepSeek-V2 model requires substantial computational resources. Ninoner )rAdorar@r'r(gh㈵>)linear_q_down_projlinear_q_up_projlinear_kv_down_projlinear_kv_up_proj linear_projFr&g-C6?zUnrecognized peft scheme: pad_to_max_lengthT)packed_sequence_size)lowerr rr strategypipeline_model_parallel_sizeexpert_model_parallel_sizer3rlrrrr pefttarget_modulesuse_distributed_optimizercross_entropy_loss_fusiontensor_model_parallel_size ValueErrorr-r1dataset_kwargsrpacked_sequence_specs) r"rBrr#r$rCr-rDr<rrrfinetune_recipews8.              r[)Nr?rr@r!rANN).typingrrlightning.pytorchpytorchplnemo_runrnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr-nemo.collections.llm.gpt.data.packed_sequencer'nemo.collections.llm.gpt.model.deepseekrr nemo.collections.llm.peftr %nemo.collections.llm.recipes.deepseekr -nemo.collections.llm.recipes.finetune_defaultr (nemo.collections.llm.recipes.log.defaultr rr'nemo.collections.llm.recipes.optim.adamr'nemo.lightning.pytorch.callbacks.deepeprnemo.utils.exp_managerrNAMEclifactoryrLightningModulerstrintr7r>boolr[rrrrs           >