o }oi1(*@sddlmZddlmZddlZddlZddlm Z ddl m Z m Z ddl mZddlmZddlmZddlmZmZmZdd lmZdd lmZmZdd lmZd Zejj ed dej!ej"fddZ#ejj e edddddddddddddddddddddddddd d!e fd"ee$d#e$d$e%d%e%d&eej&d'ee%d(e%d)e'd*ee%d+e%d,e%d-e%d.e$d/e%d0e(d1e%d2e%d3e%d4e%dej)f(d5d6Z*ejj e ed     7 8d=d"ee$d#e$d+e%d,e%d9ee$d:e'dej)fd;d<Z+dS)>)OptionalN) AutoTokenizer)finetunepretrain)MockDataModule) PEFT_STR2CLS)default_finetune_recipe) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing) qwen3_model qwen3_trainer)TimingCallbackqwen3_235b_a22bnamereturncCs ttdS)a Factory function to create a Qwen3 235B-A22B model configuration. This is a MoE (Mixture of Experts) model with 128 experts. Returns: run.Config[pl.LightningModule]: Configuration for the Qwen3 235B-A22B model. Examples: CLI usage: $ nemo llm pretrain model=qwen3_235b_a22b ... Python API usage: >>> model_config = model() >>> print(model_config) )version)r NAMErr`/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/qwen3_235b_a22b.pymodel"s r)targetrdefaultTiz bf16-mixedg? iigiUMu>ga2U0*3?dirrtensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelismexpert_parallelism num_nodesnum_gpus_per_node max_steps precisionaccumulate_grad_batchesgradient_clip_vallimit_test_batcheslimit_val_batcheslog_every_n_stepsval_check_intervalcCstj|ttdid|d|d|d|d|d|d|dd d d d | d | d | d| d| d|d|d|d|dttgtjt|||ttddt||t |ddt | |||||dt dS)a Create a pre-training recipe for Qwen3 235B-A22B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. This model uses Mixture of Experts (MoE) architecture with 128 experts. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. expert_parallelism (Optional[int]): Degree of expert parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. precision (str): Precision configuration, one of fp32, 16-mixed or bf16-mixed. accumulate_grad_batches (int): Number of steps per gradient accumulation. gradient_clip_val (float): Value for gradient clipping. limit_test_batches (int): Limit the number of test batches. limit_val_batches (int): Limit the number of validation batches. log_every_n_steps (int): Log every n steps. val_check_interval (int): Run validation every N steps. global_batch_size (int): Global batch size. micro_batch_size (int): Micro batch size. seq_length (int): Sequence length. warmup_steps (int): Number of warmup steps. constant_steps (int): Number of constant steps. min_lr (float): Minimum learning rate. max_lr (float): Maximum learning rate. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory qwen3_235b_a22b $ nemo llm pretrain --factory "qwen3_235b_a22b(num_nodes=1, name='my_qwen3_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="qwen3_pretrain", num_nodes=1) >>> print(recipe) Note: This recipe uses a mock dataset, look for the finetune examples to see how to change the dataset. r#r$r%r&r'r(r)'account_for_embedding_in_pipeline_splitT"account_for_loss_in_pipeline_splitr*r+r,r-r.r0r1r2r3 callbacksQwen/Qwen3-235B-A22B) seq_lengthglobal_batch_sizemicro_batch_size tokenizerr)r"rr )r- warmup_stepsconstant_stepsmin_lrmax_lr clip_grad)rtrainerdatalogoptimresumeNr) runPartialrrConfigrrrr r r r )r"rr#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r9r:r8r<r=r>r?fnrrrpretrain_recipe6sxV       rJloraF peft_schemepacked_sequencecCsttd|||||}d|jj_d|jj_|dus|dkr?d|jj_d|jj_d|jj_ d|jj_ d|jj_ d|j j _|S|d vrud|jj_d|jj_d|jj_ d|jj_ d|jj_ tt||_d d g|j_d |j j _|Std |)a= Create a fine-tuning recipe for Qwen3 235B-A22B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. This model uses Mixture of Experts (MoE) architecture with 128 experts. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. packed_sequence (Optional[bool]): Packing multiple training sequences into one long sequence for training efficiency. Default sequence length is 2048. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory qwen3_235b_a22b Python API usage: >>> recipe = finetune_recipe(name="qwen3_235b_a22b_finetune", num_nodes=8) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. r7TNnonerrrgh㈵>)rKdora linear_qkv linear_projg-C6?zUnrecognized peft scheme: )rrrAstrategyr4r5lowertensor_model_parallel_sizeexpert_model_parallel_sizepipeline_model_parallel_sizeexpert_tensor_parallel_sizesequence_parallelrDconfiglrrFrHrpefttarget_modules ValueError)r"rr*r+rLrMreciperrrfinetune_recipes0)               r_)NrrrrKF),typingrlightning.pytorchpytorchplnemo_runrFtorch=nemo.collections.common.tokenizers.huggingface.auto_tokenizerrnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockrnemo.collections.llm.peftr-nemo.collections.llm.recipes.finetune_defaultr(nemo.collections.llm.recipes.log.defaultr r r 'nemo.collections.llm.recipes.optim.adamr "nemo.collections.llm.recipes.qwen3r rnemo.utils.exp_managerrrclifactoryrHLightningModulerstrintdtypeboolfloatrGrJr_rrrrs              !