o wi$@s6ddlmZddlmZddlZddlZddlm Z ddl m Z ddl mZmZmZmZmZmZmZmZmZmZmZmZddlmZmZdedejejfd d Z          d&de!de!deej"dee!de!de#de!de!de!dede!de!d e!d!e!d"e!d#ee$eje deje j%f"d$d%Z&dS)')OptionalN)Callback) lightning) Qwen2Config1P5B Qwen2Config7BQwen2Config72BQwen2Config500M Qwen2ModelQwen25Config1P5BQwen25Config3BQwen25Config7BQwen25Config14BQwen25Config32BQwen25Config72BQwen25Config500M) bf16_mixed fp16_mixedversionreturncCsd}|dkr tt}nc|dkrtt}nY|dkr tt}nO|dkr*tt}nE|dkr4tt}n;|dkr>tt}n1|dkrHtt}n'|d krRtt }n|d kr\tt }n|d krftt }n |d krott }|duszJd |tjt |dS)a A function to create a qwen2 models. Args: version (str): The version of the qwen2 model to create. one of ["qwen2_500m", "qwen2_1p5b", "qwen2_7b", "qwen2_72b"]. Returns: run.Config[pl.LightningModule]: Configuration for the qwen2 model. N qwen2_500m qwen25_500m qwen2_1p5b qwen25_1p5b qwen25_3bqwen2_7b qwen25_7b qwen25_14b qwen25_32b qwen2_72b qwen25_72bzInvalid version: )config)runConfigrrrr r rr r rrrr )rr r#_/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/recipes/qwen2.py qwen2_model(s2            r%F{ bf16-mixed tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps precisionaccumulate_grad_batcheslimit_test_batcheslimit_val_batcheslog_every_n_stepsval_check_interval callbackscCsptjtj||||||ddddd }d}| dkrt}n| dkr"t}tjtjd||| | | | ||||d|d}|S) a Configure the NeMo Lightning Trainer for qwen2 models. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. precision (str): Precision configuration, one of fp32, 16-mixed or bf16-mixed. accumulate_grad_batches (int): Number of steps per gradient accumulation. limit_test_batches (int): Limit the number of test batches. limit_val_batches (int): Limit the number of validation batches. log_every_n_steps (int): Log every n steps. val_check_interval (int): Run validation every N steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. T) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_include_optimizerckpt_async_saveckpt_parallel_loadNz16-mixedr*gpuF) acceleratorr=devicesr8r9r:r;r6r4pluginsstrategyuse_distributed_samplerr<)r!r"nlMegatronStrategyrrTrainer)r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=rLprecision_plugintrainerr#r#r$ qwen2_trainerOsF,rS)r&r'NNr'Fr'r(r)r*r'r+r+r,r-N)'typingrlightning.pytorchpytorchplnemo_runr!torch$lightning.pytorch.callbacks.callbackrnemorrN$nemo.collections.llm.gpt.model.qwen2rrrrr r r r r rrr6nemo.collections.llm.recipes.precision.mixed_precisionrrstrr"LightningModuler%intdtypeboollistrPrSr#r#r#r$sx    8(