o }oi$@s2ddlmZddlmZddlZddlZddlm Z ddl m Z ddl mZmZmZmZmZmZmZmZmZmZmZddlmZmZdedejejfd d Z          d&de de deej!dee de de"de de de dede de d e d!e d"e d#ee#eje deje j$f"d$d%Z%dS)')OptionalN)Callback) lightning) Qwen2Config1P5B Qwen2Config7BQwen2Config72BQwen2Config500M Qwen2ModelQwen25Config1P5BQwen25Config7BQwen25Config14BQwen25Config32BQwen25Config72BQwen25Config500M) bf16_mixed fp16_mixedversionreturncCsd}|dkr tt}nY|dkrtt}nO|dkr tt}nE|dkr*tt}n;|dkr4tt}n1|dkr>tt}n'|dkrHtt}n|d krRtt }n|d kr\tt }n |d krett }|duspJd |tjt |d S)a A function to create a qwen2 models. Args: version (str): The version of the qwen2 model to create. one of ["qwen2_500m", "qwen2_1p5b", "qwen2_7b", "qwen2_72b"]. Returns: run.Config[pl.LightningModule]: Configuration for the qwen2 model. N qwen2_500m qwen25_500m qwen2_1p5b qwen25_1p5bqwen2_7b qwen25_7b qwen25_14b qwen25_32b qwen2_72b qwen25_72bzInvalid version: )config) runConfigrrrr rr r r rrr )rrr!V/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/qwen2.py qwen2_model's.           r#F{ bf16-mixed tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps precisionaccumulate_grad_batcheslimit_test_batcheslimit_val_batcheslog_every_n_stepsval_check_interval callbackscCsptjtj||||||ddddd }d}| dkrt}n| dkr"t}tjtjd||| | | | ||||d|d}|S) a Configure the NeMo Lightning Trainer for qwen2 models. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. precision (str): Precision configuration, one of fp32, 16-mixed or bf16-mixed. accumulate_grad_batches (int): Number of steps per gradient accumulation. limit_test_batches (int): Limit the number of test batches. limit_val_batches (int): Limit the number of validation batches. log_every_n_steps (int): Log every n steps. val_check_interval (int): Run validation every N steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. T) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_include_optimizerckpt_async_saveckpt_parallel_loadNz16-mixedr(gpuF) acceleratorr;devicesr6r7r8r9r4r2pluginsstrategyuse_distributed_samplerr:)rr nlMegatronStrategyrrTrainer)r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;rJprecision_plugintrainerr!r!r" qwen2_trainerLsF,rQ)r$r%NNr%Fr%r&r'r(r%r)r)r*r+N)&typingrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrnemorrL$nemo.collections.llm.gpt.model.qwen2rrrrr r r r r rr6nemo.collections.llm.recipes.precision.mixed_precisionrrstrr LightningModuler#intdtypeboollistrNrQr!r!r!r"sx    4 &