o wi*@sHddlmZddlmZddlZddlZddlm Z ddl m Z ddl m ZddlmZmZmZmZmZmZmZmZmZddlmZmZded ejejfd d Z          d)de de deej!dee de de"de de"de"de de de d ed!e d"e d#e d$e d%e d&ee#eje d ejej$f(d'd(Z%dS)*)OptionalN)Callback)DistributedDataParallelConfig) lightning) Qwen3Config1P7B Qwen3Config4B Qwen3Config8BQwen3Config14BQwen3Config30B_A3BQwen3Config32BQwen3Config235B_A22BQwen3Config600M Qwen3Model) bf16_mixed fp16_mixedversionreturncCsd}|dkr tt}nE|dkrtt}n;|dkr tt}n1|dkr*tt}n'|dkr4tt}n|dkr>tt}n|dkrHtt}n |d krQtt }|dus\Jd |tjt |d S) aZ A function to create a qwen3 models. Args: version (str): The version of the qwen3 model to create. One of ["qwen3_600m", "qwen3_1p7b", "qwen3_4b", "qwen3_8b", "qwen3_14b", "qwen3_32b", "qwen3_30b_a3b", "qwen3_235b_a22b"]. Returns: run.Config[pl.LightningModule]: Configuration for the qwen3 model. N qwen3_600m qwen3_1p7bqwen3_4bqwen3_8b qwen3_14b qwen3_32b qwen3_30b_a3bqwen3_235b_a22bzInvalid version: )config) runConfigr rrrr r r r r)rrr_/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/recipes/qwen3.py qwen3_model&s&         r F{ bf16-mixed tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelismexpert_parallelism'account_for_embedding_in_pipeline_split"account_for_loss_in_pipeline_split num_nodesnum_gpus_per_node max_steps precisionaccumulate_grad_batcheslimit_test_batcheslimit_val_batcheslog_every_n_stepsval_check_interval callbackscCstjtjf|||||||d||ddddtjtdddddddd}d}| dkr,t}n| dkr3t}tjtjd || | |||| | ||d |d }|S) a Configure the NeMo Lightning Trainer for qwen3 models. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. expert_parallelism (Optional[int]): Degree of expert parallelism. account_for_embedding_in_pipeline_split (bool): Whether to treat input embedding layer as a standard transformer layer in the context of partition and placement for pipeline parallelism. account_for_loss_in_pipeline_split (bool): Whether to treat loss layer as a standard transformer layer in the context of partition and placement for pipeline parallelism. account_for_loss_in_pipeline_split (bool): = False, num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. precision (str): Precision configuration, one of fp32, 16-mixed or bf16-mixed. accumulate_grad_batches (int): Number of steps per gradient accumulation. limit_test_batches (int): Limit the number of test batches. limit_val_batches (int): Limit the number of validation batches. log_every_n_steps (int): Log every n steps. val_check_interval (int): Run validation every N steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. r!Toptim_grads_params)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collectivedata_parallel_sharding_strategy)tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelexpert_model_parallel_sizeexpert_tensor_parallel_sizer/r0gradient_as_bucket_viewckpt_include_optimizerckpt_async_saveckpt_parallel_loadddpNz16-mixedr$gpuF) acceleratorr:devicesr5r6r7r8r3r1pluginsstrategyuse_distributed_samplerr9)rrnlMegatronStrategyrrrTrainer)r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:rSprecision_plugintrainerrrr qwen3_trainerGsb5rZ)r!r!NNr!Fr!FFr!r"r#r$r!r%r%r&r'N)&typingrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrU$nemo.collections.llm.gpt.model.qwen3rrrr r r r r r6nemo.collections.llm.recipes.precision.mixed_precisionrrstrrLightningModuler intdtypeboollistrWrZrrrrs     , "