o }oi/(@srddlmZddlmZddlZddlZddlm Z ddl m Z ddl mZmZddlmZddlmZmZddlmZdd lmZdd lmZmZmZdd lmZdd lm Z m!Z!dd l"m#Z#dZ$ej%j&e$ddej'ej(fddZ)                dAde*de*deej+d ee*d!e*d"e,d#e*d$e*d%e*d&e-d'e*d(e*d)e*d*e*d+e*d,ee.ej'e dej'e j/f"d-d.Z0ej%j&ee$d/dd0ddej1dddddd1ddd2dddd3ddd4d5dd6d7efd8ee-d9e-de*de*deej+d ee*d!e*d"e,d#e*d$e*d%e*d&e-d'e*d:e2d(e*d)e*d*e*d+e*dej3f&d;d<Z4ej%j&ee$d/  0   =dBd8ee-d9e-d#e*d$e*d>ee-dej3f d?d@Z5dS)C)OptionalN)Callback) lightning)finetunepretrain)MockDataModule)StarcoderConfig15BStarcoderModel) PEFT_STR2CLS)default_finetune_recipe) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing) bf16_mixed fp16_mixed)TimingCallback starcoder_15bnamereturncCstjtttdS)aj Factory function to create a Starcoder 15B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Starcoder 15B model. Examples: CLI usage: $ nemo llm pretrain model=starcoder_15b ... Python API usage: >>> model_config = model() >>> print(model_config) )config)runConfigr rrr^/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/starcoder_15b.pymodel$srF{ bf16-mixed tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps precisionaccumulate_grad_batcheslimit_test_batcheslimit_val_batcheslog_every_n_stepsval_check_interval callbackscCsptjtj||||||ddddd }d}| dkrt}n| dkr"t}tjtjd||| | | | ||||d|d}|S) a Configure the NeMo Lightning Trainer for Starcoder 15B models. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. precision (str): Precision configuration, one of fp32, 16-mixed or bf16-mixed. accumulate_grad_batches (int): Number of steps per gradient accumulation. limit_test_batches (int): Limit the number of test batches. limit_val_batches (int): Limit the number of validation batches. log_every_n_steps (int): Log every n steps. val_check_interval (int): Run validation every N steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. T) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_include_optimizerckpt_async_saveckpt_parallel_loadNz16-mixedr"gpuF) acceleratorr5devicesr0r1r2r3r.r,pluginsstrategyuse_distributed_samplerr4)rrnlMegatronStrategyrrTrainer)r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5rDprecision_plugintrainerrrrstarcoder_trainer8sF,rK)targetrdefaultig?iiigiUMu>ga2U0*3?dirrgradient_clip_valcCstj|ttdid|d|d|d|d|d|d|d| d | d | d | d |d |d|d|dttgtjt|||dt||t|ddt | ||||| dt dS)al Create a pre-training recipe for Starcoder 15B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. precision (str): Precision configuration, one of fp32, 16-mixed or bf16-mixed. accumulate_grad_batches (int): Number of steps per gradient accumulation. gradient_clip_val (float): Value for gradient clipping. limit_test_batches (int): Limit the number of test batches. limit_val_batches (int): Limit the number of validation batches. log_every_n_steps (int): Log every n steps. val_check_interval (int): Run validation every N steps. global_batch_size (int): Global batch size. micro_batch_size (int): Micro batch size. seq_length (int): Sequence length. warmup_steps (int): Number of warmup steps. constant_steps (int): Number of constant steps. min_lr (float): Minimum learning rate. max_lr (float): Maximum learning rate. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory starcoder_15b $ nemo llm pretrain --factory "starcoder_15b(num_nodes=1, name='my_starcoder2_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="starcoder2_pretrain", num_nodes=1) >>> print(recipe) Note: This recipe uses a mock dataset, look for the finetune examples to see how to change the dataset. r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5) seq_lengthglobal_batch_sizemicro_batch_sizer)rNrr)r/ warmup_stepsconstant_stepsmin_lrmax_lr clip_grad)rrJdatalogoptimresumeNr) rPartialrrKrrrr rrr )rNrr&r'r(r)r*r+r,r-r.r/r0rOr1r2r3r4rQrRrPrSrTrUrVfnrrrpretrain_recipesjS      r^lora peft_schemecCs|ttd||||}|dus|dkr d|jj_d|jj_|S|dvr7t t ||_ d|jj_|St d|) a@ Create a fine-tuning recipe for Starcoder 15B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory starcoder_15b Python API usage: >>> recipe = finetune_recipe(name="starcoder_15b_finetune", num_nodes=2) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. zbigcode/starcoderNnoner gh㈵>)r_dorag-C6?zUnrecognized peft scheme: )r rlowerrJrDr7rZrlrrrr peft ValueError)rNrr,r-r`reciperrrfinetune_recipes%    rh)rrNNrFrr r!r"rr#r#r$r%N)NrMrr r_)6typingrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrnemorrFnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr(nemo.collections.llm.gpt.model.starcoderrr nemo.collections.llm.peftr -nemo.collections.llm.recipes.finetune_defaultr (nemo.collections.llm.recipes.log.defaultr r r'nemo.collections.llm.recipes.optim.adamr6nemo.collections.llm.recipes.precision.mixed_precisionrrnemo.utils.exp_managerrNAMEclifactoryrLightningModulerintdtypeboolstrlistrHrKbfloat16floatr\r^rhrrrrs4                 T      z