o }oi @sddlmZmZmZddlmZddlZddl Z ddl m Z ddl mZddlmZddlmZmZddlmZddlmZddlmZdd lmZdd lmZerZdd lmZm Z eZ!    d-dej"ej#de$dee$de$de%de%de&dee!dej'fddZ(d d e j)dd dd dddddf ddZ*   d.dee$de$deej"d d!eej"d"dej"e j+f d#d$Z,d%e$dej"e j-fd&d'Z.ej/j0d(d)dej"efd*d(Z1ej/j0d+d)dej"efd,d+Z2dS)/) TYPE_CHECKINGAnyOptionalN)llm)PackedSequenceSpecs)DoRALoRA)tensorboard_logger),distributed_fused_adam_with_cosine_annealing) bf16_mixed)PEFT)TimingCallback)TensorBoardLogger WandbLoggerdefaultFmodel resume_pathdirname num_nodesnum_gpus_per_nodepacked_sequence tokenizerreturnc Cs~|rtjtjdddtddd}n tjtjdddd}tjtj|t||d|t||t |d d t d d d ddt ||d} | S)a Create a default fine-tuning recipe for any model. This function sets up a template for a complete configuration for fine-tuning, including model, trainer, data, logging, optimization, and resumption settings. Args: model (run.Config[pl.LightningModule]): Configuration for a NeMo model. resume_path (str): Path to the Huggingface model or pretrained distributed checkpoint for resume dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the fine-tuning run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. packed_sequence (bool): Whether to use packed sequence. tokenizer (Optional[TokenizerType]): Tokenizer setting to be applied. Can be 'data' or 'model' or an instance of TokenizerSpec. Returns: run.Partial: Partial configuration for fine-tuning. See usages of this recipe for further details. irr)packed_sequence_size) seq_lengthglobal_batch_sizemicro_batch_sizepacked_sequence_specs)rrr)rrr)rrr g-C6?r2g\(\?)max_lrmin_lr warmup_steps adam_beta2)rtrainerdatalogoptimresumer) runConfigrSquadDataModulerPartialfinetunedefault_finetune_trainerdefault_finetune_logr r nemo_resume) rrrrrrrr datamodulereciper7a/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/finetune_default.pydefault_finetune_recipe%s. r9ic CsTtjtj||||||ddd } tjtjdd|| | d||t| d| ttgd} | S)a; Create a default fine-tuning trainer for any model. This function sets up a template for strategy and trainer. Args: See docstrings of MegatronStrategy and Trainer. Returns: run.Config: Config for a finetuning trainer. See usages of this in recipes for further details. Tlog_all)tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewckpt_load_strictnessgpurF) acceleratoraccumulate_grad_batchesdeviceslimit_test_batcheslimit_val_batcheslog_every_n_steps max_stepsrpluginsstrategyuse_distributed_samplerval_check_interval callbacks)r-r.nlMegatronStrategyTrainerr r )tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelismrrrKrHrIrOrMr(r7r7r8r2`s8  r2r r wandb_loggerrcCs.tjtjddddd}tjtj|||||dS)aI Create a default fine-tuning logger for any model. This function sets up a template for ModelCheckpoint and NeMoLogger. Args: See docstrings of ModelCheckpoint and NeMoLogger. Returns: run.Config: Config for a finetuning NeMoLogger. See usages of this in recipes for further details. linkr#z6{model_name}--{val_loss:.2f}-{step}-{consumed_samples}) save_last save_top_kevery_n_train_stepsfilename)ckptr tensorboardwandblog_dir)r-r.rQModelCheckpoint NeMoLogger)rrr rZrar7r7r8r3sr3model_idcCs"tjtjtjtjd|ddS)a Configure automatic resumption from a NeMo checkpoint converted from Huggingface for https://huggingface.co/{model_id}. This NeMo checkpoint should be converted from Huggingface beforehand, using nemo.collections.llm.import_ckpt. When converting the checkpoint, the NeMo checkpoint will be saved in NEMO_HOME (set to ~/.cache/nemo by default). This function sets up the configuration to resume training from path nemo://{model_id}. This translates to the full path {NEMO_HOME}/models/{model_id}. Args: model_id (str): Path to the Huggingface model or pretrained distributed checkpoint for resume Returns: run.Config[nl.AutoResume]: Configuration for resuming from NeMo checkpoint. znemo://)path)restore_config)r-r.rQ AutoResume RestoreConfig)rgr7r7r8r4sr4lorar"cC ttS)a@ Factory function to create a LoRA configuration. Returns: run.Config[PEFT]: Configuration for the LoRA class. Examples: CLI usage: $ nemo llm finetune -f llama3_8b peft=lora Python API usage: >>> lora_config = lora() >>> print(lora_config) )r-r.rr7r7r7r8rl doracCrm)a@ Factory function to create a DoRA configuration. Returns: run.Config[PEFT]: Configuration for the DoRA class. Examples: CLI usage: $ nemo llm finetune -f llama3_8b peft=dora Python API usage: >>> dora_config = dora() >>> print(dora_config) )r-r.rr7r7r7r8rorn)NrrrFr)NrNN)3typingrrrlightning.pytorchpytorchplnemo_runr-torchnemo.lightning lightningrQnemo.collectionsr-nemo.collections.llm.gpt.data.packed_sequencernemo.collections.llm.peftrr(nemo.collections.llm.recipes.log.defaultr 'nemo.collections.llm.recipes.optim.adamr 6nemo.collections.llm.recipes.precision.mixed_precisionr nemo.lightning.pytorch.callbacksr nemo.utils.exp_managerr lightning.pytorch.loggersrr TokenizerTyper.LightningModulestrintboolr0r9bfloat16r2rfr3rjr4clifactoryrlror7r7r7r8s           < <    &