o }oiE@sddlZddlmZddlmZddlmZmZmZm Z ddl m Z ddl ZddlmZddlmZddlmZmZmZmZmZddlmZmZddlmZeZejj d d      d*d e j!d e j"d edeeeej#efdeeeej#efdeedeedee eeefdefddZ$ejj d d    d+d e j!d e j"d edeeeej#efdeeeej#efdeedefddZ%ejj d d     d,d e j!d e j"d edeeeej#efdeeeej#efdeedee eeefdefddZ&ejj d d      d*d e j!d e j"d edeeeej#efdeeeej#efdeedeedee eeefdefddZ'ddZ(ejj dd d d!dZ)d e j!d e j"deddfd"d#Z*d e j!d e j"d edeedeedeedeedee eeefdefd$d%Z+d&d'Z,     d*d e j!d e j"d edeedeedeedeedee eeefddfd(d)Z-dS)-N)deepcopy)Path)AnyCallableOptionalUnion) Annotated) AutoResume NeMoLoggerOptimizerModuleTrainer-configure_no_restart_validation_training_loop)PEFTModelTransform)loggingspeechlm) namespacemodeldatatrainerlogresumeoptim tokenizermodel_transformreturnc C*t||||||||d}||||jS)aY Trains a model using the specified data and trainer, with optional tokenizer, source, and export. Args: model (pl.LightningModule): The model to be trained. data (pl.LightningDataModule): The data module containing training data. trainer (Trainer): The trainer instance configured with a MegatronStrategy. log (NeMoLogger): A nemologger instance. resume (Optional[Union[AutoResume, Resume]]): Resume training from a checkpoint. optim (Optional[OptimizerModule]): The optimizer module to be used. If not provided, the default optimizer from the model will be used. tokenizer (Optional[TokenizerType]): Tokenizer setting to be applied. Can be 'data' or 'model' or an instance of TokenizerSpec. export (Optional[str]): Filename to save the exported checkpoint after training. model_transform (Optional[Union[Callable[[nn.Module], nn.Module], PEFT]]): A model transform to be applied. Returns ------- Path: The directory path where training artifacts are saved. Examples -------- >>> from nemo.collections import llm >>> from nemo import lightning as nl >>> model = llm.MistralModel() >>> data = llm.SquadDataModule(seq_length=4096, global_batch_size=16, micro_batch_size=2) >>> precision = nl.MegatronMixedPrecision(precision="bf16-mixed") >>> trainer = nl.Trainer(strategy=nl.MegatronStrategy(tensor_model_parallel_size=2), plugins=precision) >>> llm.train(model, data, trainer, tokenizer="data") PosixPath('/path/to/log_dir') rrrrrrrr)_setupfitexp_dir rrrrrrrr app_stater#Q/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/speechlm/api.pytrain&s+ r%c Cs*t||||||dt||||||ddS)a Pretrains a model using the specified data and trainer, with optional logging, resuming, and optimization. This function is a wrapper around the `train` function, specifically configured for pretraining tasks. Note, by default it will use the tokenizer from the model. Args: model (pl.LightningModule): The model to be pretrained. data (pl.LightningDataModule): The data module containing pretraining data. trainer (Trainer): The trainer instance configured with a MegatronStrategy. log (NeMoLogger): A nemologger instance. resume (Optional[AutoResume]): Resume training from a checkpoint. optim (Optional[OptimizerModule]): The optimizer module to be used. If not provided, the default optimizer from the model will be used. Returns: Path: The directory path where pretraining artifacts are saved. Examples: >>> from nemo.collections import llm >>> from nemo import lightning as nl >>> model = llm.MistralModel() >>> data = llm.PretrainingDataModule(paths=[...], seq_length=4096, global_batch_size=16, micro_batch_size=2) >>> precision = nl.MegatronMixedPrecision(precision="bf16-mixed") >>> trainer = nl.Trainer(strategy=nl.MegatronStrategy(tensor_model_parallel_size=2), plugins=precision) >>> llm.pretrain(model, data, trainer) PosixPath('/path/to/log_dir') )rrrr)rrrrrrr_validate_configr%)rrrrrrr#r#r$pretrainas%r(peftc Cs.t|||||||dt||||||d|dS)a Finetunes a model using the specified data and trainer, with optional logging, resuming, and PEFT. Note, by default it will use the tokenizer from the model. Args: model (pl.LightningModule): The model to be finetuned. data (pl.LightningDataModule): The data module containing finetuning data. trainer (Trainer): The trainer instance configured with a MegatronStrategy. log (NeMoLogger): A nemologger instance. resume (Optional[AutoResume]): Resume training from a checkpoint. optim (Optional[OptimizerModule]): The optimizer module to be used. If not provided, the default optimizer from the model will be used. peft (Optional[PEFT]): A PEFT (Parameter-Efficient Fine-Tuning) configuration to be applied. Returns: Path: The directory path where finetuning artifacts are saved. Examples: >>> from nemo.collections import llm >>> from nemo import lightning as nl >>> model = llm.MistralModel() >>> data = llm.SquadDataModule(seq_length=4096, global_batch_size=16, micro_batch_size=2) >>> precision = nl.MegatronMixedPrecision(precision="bf16-mixed") >>> trainer = nl.Trainer(strategy=nl.MegatronStrategy(tensor_model_parallel_size=2), plugins=precision) >>> llm.finetune(model, data, trainer, peft=llm.peft.LoRA()]) PosixPath('/path/to/log_dir') )rrrrrrr&)rrrrrrr)r#r#r$finetunes'r*c Cr)a Validates a model using the specified data and trainer, with optional logging, resuming, and model transformations. Args: model (pl.LightningModule): The model to be validated. data (pl.LightningDataModule): The data module containing validation data. trainer (Trainer): The trainer instance configured with a MegatronStrategy. log (NeMoLogger): A nemologger instance. resume (Optional[AutoResume]): Resume from a checkpoint for validation. optim (Optional[OptimizerModule]): The optimizer module to be used. If not provided, the default optimizer from the model will be used. tokenizer (Optional[TokenizerType]): Tokenizer setting to be applied. Can be 'data' or 'model' or an instance of TokenizerSpec. model_transform (Optional[Union[Callable[[nn.Module], nn.Module], PEFT]]): A model transform to be applied. Returns: Path: The directory path where validation artifacts are saved. Examples: >>> from nemo.collections import llm >>> from nemo import lightning as nl >>> model = llm.MistralModel() >>> data = llm.SquadDataModule(seq_length=4096, global_batch_size=16, micro_batch_size=2) >>> precision = nl.MegatronMixedPrecision(precision="bf16-mixed") >>> trainer = nl.Trainer(strategy=nl.MegatronStrategy(tensor_model_parallel_size=2), plugins=precision) >>> llm.validate(model, data, trainer, tokenizer="data") PosixPath('/path/to/log_dir') r)rvalidater r!r#r#r$r+s' r+cCtd)z( Evaluates NeMo SpeechLM model. 'This function will be implemented laterNotImplementedErrorr#r#r#r$evaluatesr0generate)namercCr,)z3 Generates text using a NeMo Speech model. r-r.r#r#r#r$r1scCs|dkr t|d|jdS|dkrt|d|jdSz!ddlm}t||r5t|d|t|d|WdStd|tyFtdw)Nrrrr) TokenizerSpecz2Expected TokenizerSpec or 'data' or 'model', got: zTokenizerSpec is not available) _set_with_ior1nemo.collections.common.tokenizers.tokenizer_specr3 isinstance ValueError ImportError)rrrr3r#r#r$_use_tokenizer s    r9c Cst||pt}|rt|tr|jrtdd|j_|j|t |ddt t ddd} |dur6||||r=| ||rEt ||||rMt |d|t |ddrqtdd|jDsqt|trj|j|| S|jt| S) Nz8Disabling try_restore_best_ckpt restoration for adaptersFresume_if_exists__io__)r: task_configrcss|]}t|tVqdS)N)r6r).0cbr#r#r$ @sz_setup..)r r r6rckptrinfotry_restore_best_ckptsetupgetattrr%connectr9r4any callbacksrappend) rrrrrrrr_logr"r#r#r$rs2           rcCs@t|||t|drt|drt|j|t|jdSdSdS)Nr;)setattrhasattrr;r)objattrvaluer#r#r$r4Is r4cCst|drUt|jdddksJt|jdddksJ|jjdks#J|jjdks+J|jjdks3J|jjdks;Jt|jdrTt|jdddurT|jj|jjksTJn t |j t j r`Jdt|drl|j dkslJt|drx|jdksxJt|dr|jdksJt|drt|dr|j|j dksJd t |j t j r|j jdksJ|j jdksJ|j jdksJ|j|j|j j|j j|j jdksJd |j|j |j|j|j j|j j|j jdksJd |j jdkr|j jd krtd d|j _|j jdkr|j jdusJdn |j jdur*tdd|j _|j jdur:tdd|j _|j jdkr_t|dr_|jjdur_|jj|j jddks_Jd|j jdkrt|dr|jjduswJd|jj|j jdksJddSdSdSdS)Nconfig seq_lengthrmax_position_embeddingszExpected model.config to existmicro_batch_sizeglobal_batch_sizezGGlobal batch size must be divisible by micro batch size in data module.z[Number of GPUs must be divisible by the product of all parallelism sizes for data parallel.z]Global batch size must be divisible by the product of micro batch size and data parallel sizeTzKDisabling sequence parallelism because tensor model parallelism is disabledFzCpipeline_dtype must be set if pipeline model parallelism is enabledzUDisabling virtual pipeline parallelism because pipeline model parallelism is disabledzMSetting pipeline dtype to None because pipeline model parallelism is disabledz[Sequence length must be divisible by 2 * context parallel size if context parallel is used.zs.        : 0 3 6  +