o }oi&@sddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZm Z m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'dZ(ej)j*e(ddej+ej,fddZ-          d2de.de.deej/dee.de.de0de.d e.d!e.d"ee1ej+e dej+ej2fd#d$Z3ej)j*ee(d%dd&ddefd'ee4d(e4de.d e.d)edej5f d*d+Z6ej)j*ee(d%  &   ,  d3d'ee4d(e4de.d e.d-ee4d.ee.d/ee0dej5fd0d1Z7dS)4)CallableOptionalN)Callback)DistributedDataParallelConfig) lightning)finetunepretrain)MockDataModule)PackedSequenceSpecs)Llama32Config1B LlamaModel) PEFT_STR2CLS)default_finetune_recipe) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing) bf16_mixed)TimingCallback llama32_1bnamereturncCstt}d|_tjt|dS)ac Factory function to create a Llama3.2 1B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Llama3.2 1B model. Examples: CLI usage: $ nemo llm pretrain model=llama32_1b ... Python API usage: >>> model_config = model() >>> print(model_config) )config)runConfigr seq_lengthr )confr[/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/llama32_1b.pymodel's r!F{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc Csbtjtj||||||dddtjtddddddd } tjtjdd| |ddd||t| d d d } | S) a Configure the NeMo Lightning Trainer for Llama3.2 1B model. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=llama32_1b ... Python API usage: >>> trainer_config = trainer(num_nodes=1, num_gpus_per_node=1) >>> print(trainer_config) Note: This configuration uses extensive parallelism to handle the large model size efficiently. T)check_for_nan_in_gradgrad_reduce_in_fp32overlap_grad_reduceoverlap_param_gatheraverage_in_collective) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_async_saveckpt_parallel_loadddpgpur"2 Fi) acceleratoraccumulate_grad_batchesr.deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr-r+pluginsstrategyuse_distributed_samplerval_check_interval)rrnlMegatronStrategyrTrainerr) r%r&r'r(r)r*r+r,r-r.rItrainerrrr rO<sJ)rO)targetrdefaultdirrfnc CsVtj|tt||ttgdtjtddddt||t|ddt dd t d }|S) a Create a pre-training recipe for Llama3.2 1B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory llama32_1b $ nemo llm pretrain --factory "llama32_1b(num_nodes=1, name='my_1b_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="llama32_1b_pretrain", num_nodes=1) >>> print(recipe) Note: This recipe is optimized for the large 8B model and requires significant computational resources. )r+r,r.rir")rglobal_batch_sizemicro_batch_sizer)rRrrga2U0*3?)max_lr)r!rOdatalogoptimresume) rPartialr!rOrrr rrrr)rRrr+r,rSreciperrr pretrain_recipes$ r]lora peft_schemerpacked_sequencecCs|dur |rdnd}ttd|||||}|dus|dkr*d|jj_d|jj_n/|dvrRt t ||_ d |j _ d |j _d |jj_d |jj_d |jj_ntd |||jj_||j_|rsddi|j_t j t|d|j_|S)a& Create a fine-tuning recipe for Llama3.2 1B model. This function sets up a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. The recipe uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, unless peft_scheme is set to None. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. peft_scheme (Optional[str]): Name of the peft scheme to use for fine-tuning. Allowed values: 'lora'/'dora'/'none'/None. seq_length (int): Maximum number of tokens per microbatch. packed_sequence (Optional[bool]): If true, fine-tuning sequences will be packed into batches up to the given maximum seq_length for better efficiency. Returns: run.Partial: Partial configuration for fine-tuning. Examples: CLI usage: $ nemo llm finetune --factory llama32_1b Python API usage: >>> recipe = finetune_recipe(name="llama32_1b_finetune", num_nodes=1) >>> print(recipe) Note: This recipe uses the SQuAD dataset for fine-tuning. Niizmeta-llama/Llama-3.2-1Bnoner"gh㈵>)r^dorar#Fg-C6?zUnrecognized peft scheme: pad_to_max_lengthT)packed_sequence_size)rr!lowerrOrIr4rYrlrrrr peftdimalphause_distributed_optimizercross_entropy_loss_fusion ValueErrorrrWdataset_kwargsr packed_sequence_specs)rRrr+r,r_rr`r\rrr finetune_recipes,,         rp) r"r"NNr"Fr"r#r$N)NrQr"r#r^NN)8typingrrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrLnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr -nemo.collections.llm.gpt.data.packed_sequencer $nemo.collections.llm.gpt.model.llamar r nemo.collections.llm.peftr -nemo.collections.llm.recipes.finetune_defaultr(nemo.collections.llm.recipes.log.defaultrrr'nemo.collections.llm.recipes.optim.adamr6nemo.collections.llm.recipes.precision.mixed_precisionrnemo.utils.exp_managerrNAMEclifactoryrLightningModuler!intdtypeboollistrNrOstrr[r]rprrrr s               R 4