o wiL$@sddlmZddlmZddlZddlZddlm Z ddl m Z ddl mZmZmZmZddlmZmZdedejejfd d Z          d&dededeejdeededededededededed ed!ed"ed#eeeje deje jf"d$d%ZdS)')OptionalN)Callback) lightning)Starcoder2Config3BStarcoder2Config7BStarcoder2Config15BStarcoder2Model) bf16_mixed fp16_mixedversionreturncCsbd}|dkr tt}n|dkrtt}n |dkrtt}|dus*Jd|tjt|dS)a+ A function to create a Starcoder2 models. Args: version (str): The version of the Starcoder2 model to create. one of ["starcoder2_3b", "starcoder2_7b", "starcoder2_15b"]. Returns: run.Config[pl.LightningModule]: Configuration for the Starcoder2 model. N starcoder2_3b starcoder2_7bstarcoder2_15bzInvalid version: )config)runConfigrrrr)r rrd/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/recipes/starcoder2.pystarcoder2_model s    rF{ bf16-mixed tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps precisionaccumulate_grad_batcheslimit_test_batcheslimit_val_batcheslog_every_n_stepsval_check_interval callbackscCsptjtj||||||ddddd }d}| dkrt}n| dkr"t}tjtjd||| | | | ||||d|d}|S) a Configure the NeMo Lightning Trainer for Starcoder2 models. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. precision (str): Precision configuration, one of fp32, 16-mixed or bf16-mixed. accumulate_grad_batches (int): Number of steps per gradient accumulation. limit_test_batches (int): Limit the number of test batches. limit_val_batches (int): Limit the number of validation batches. log_every_n_steps (int): Log every n steps. val_check_interval (int): Run validation every N steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. T) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_include_optimizerckpt_async_saveckpt_parallel_loadNz16-mixedrgpuF) acceleratorr-devicesr(r)r*r+r&r$pluginsstrategyuse_distributed_samplerr,)rrnlMegatronStrategyr r Trainer)rrr r!r"r#r$r%r&r'r(r)r*r+r,r-r<precision_plugintrainerrrrstarcoder2_trainer7sF,rC)rrNNrFrrrrrrrrrN)typingrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrnemorr>)nemo.collections.llm.gpt.model.starcoder2rrrr6nemo.collections.llm.recipes.precision.mixed_precisionr r strrLightningModulerintdtypeboollistr@rCrrrrsx