o wiUM@sddlmZddlmZddlmZmZddlmZ ddl Z ddl Z ddl mZddlmZddlmZddlmZdd lmZdd lmZmZdd lmZdd lmZmZdd l m!Z!ddl"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7dZ8ddZ9de:de;defddZ? !   ! " ! # # $ % &  " ' ' " ' (dnd)ed?ZCdod@ee:BdBfdAdBZDdd d!d!d d!d d!d!dCdDdEd"d&d#dd'd'ddd"d#d!ddgd"d"d'd'd'd(d&d"ddddFef&dGe:dHefdfdgZJe jGjHee8dhedide jEfdjdkZKe jGjHee8dhedid]e:dBde jEfdldmZLdS)p)asdict)Path)LiteralOptionalN)Callback)DistributedDataParallelConfig) lightning)llm) TokenizerSpec)finetunepretrain)PreTrainingDataModule) Evo2Datasetparse_dataset_config)MockDataModule) default_logtensorboard_logger wandb_logger),distributed_fused_adam_with_cosine_annealing) bf16_mixedbf16_with_fp8_mixed)/userbuffers_bf16_h100_h8192_tp4_mbs1_seqlen8192)get_nmt_tokenizer) callbacks)FLOPsMeasurementCallback)MegatronCommOverlapCallback)TimingCallback hyena_testcCstjtddS)a Creates and returns a configuration for initializing a tokenizer. The configuration is set up to use the `get_nmt_tokenizer` function with the specified library type as 'byte-level'. Returns: run.Config: A configuration object for the tokenizer setup. z byte-level)library)runConfigrr!r!d/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/recipes/hyena_base.pytokenizer_recipe1s r# model_sizetp_comm_overlap seq_lengthreturncCs|dkrtj}n?|dkrtj}n7|dkrtj}n/|dkr tj}n'|dkr(tj}n|dkr0tj}n|dkr8tj}n|dkr@tj}nt d |t j tj t j |||d t d S) aX Factory function to create a striped hyena model Returns: run.Config[pl.LightningModule]: Configuration for the striped hyena model. Examples: CLI usage: $ nemo llm pretrain model=hyena_test ... Python API usage: >>> model_config = model() >>> print(model_config) test1b7bznv-7b40bznv-40b7b_arc_longcontext40b_arc_longcontextzUnsupported model size: )r&r%)config tokenizer)r HyenaTestConfig Hyena1bConfig Hyena7bConfigHyenaNV7bConfigHyena40bConfigHyenaNV40bConfigHyena7bARCLongContextConfigHyena40bARCLongContextConfigNotImplementedErrorrr HyenaModelr/)r$r%r&cfg_clsr!r!r" model_recipeAs4r;Fd2 T torch_disttensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_stepsval_check_intervallimit_test_batcheslimit_val_batcheslog_every_n_stepsrfp8grad_reduce_in_fp32align_param_gatherno_aligned_megatron_ddpckpt_async_savesave_ckpt_format)rBzarrcCs|r tjtd||d}n tjtd|ddd|dd}tjtj||||||d||d|d }|r4t}d|_nt}tjtjdd| |||||d| | | | d }|S) a: Configure the NeMo Lightning Trainer for Striped Hyena model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Number of tensor replicas for vector parallelism. pipeline_parallelism (int): Number of pipeline segments for model pipeline parallelism. pipeline_parallelism_type ([type]): Type of pipeline parallelism to apply. Support 'interleaved','split'. virtual_pipelien_parallelism (int): Number of virtual pipeline stages for interleaving smaller sub-microbatches to reduce the computational graph bubbles caused by pipeline parallel (if pipeline parallel>1 is in use) context_parallelism (int): Number of context parallel blocks for processing sub-attention matrices in a block parallel fassion. sequence_parallelism (bool): Whether to use the sequence_parallelism improvement on the base tensor parallelism. This will allow for more layers to be parallelized when using tensor parallelism. num_nodes (int): Number of nodes for the distributed setting. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps before training terminates. val_check_interval (int): Interval between val check runs. limit_test_batches (int): Maximum number of batches over which to run test check runs on the dev set data. limit_val_batches (int): Maximum number of batches over which to run val check runs on the dev set data. log_every_n_steps (int): Log to the endpoint every n steps. callbacks (list[run.Config[Callback]]): A list of nemo/lightning callbacks to execute during training. fp8 (bool): Whether to use fp8 precision for computations. grad_reduce_in_fp32 (bool): Boolean indicating whether to reduce the gradient weight in the FP32 format rather than the default bf16. align_param_gather (bool): Optimization for faster train step timing potentially through aligning parameter gather operations. no_aligned_megatron_ddp (bool): Skip the aligned megatron DDP optimizations. ckpt_async_save (bool): Bool indicating whether to use asynchronous checkpoint saving. save_ckpt_format (Literal['torch_dist', 'zarr']): The checkpoint save method. The default torch_dist may result in larger checkpoints currently, but is the preferred option in the long run. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. T)check_for_nan_in_gradrQrR)rWrQoverlap_grad_reduceoverlap_param_gatheraverage_in_collectiverRuse_distributed_optimizer) tensor_model_parallel_sizepipeline_model_parallel_sizepipeline_dtype$virtual_pipeline_model_parallel_sizecontext_parallel_sizesequence_parallelgradient_as_bucket_viewrTrUckpt_parallel_loadddpgpur=F) acceleratoraccumulate_grad_batchesrdevicesrKrIpluginsstrategyuse_distributed_samplerrLrMrNrO) rr rnlMegatronStrategyrfp8_amax_history_lenrTrainer)rCrDrErFrGrHrIrJrKrLrMrNrOrrPrQrRrSrTrUrdrkmixed_precision_cfgtrainerr!r!r"trainer_recipensh; rs config_pathcCstjt|dS)a Creates a configuration for a blended dataset by utilizing the `run.Config` function. Args: config_path (Path | str | None, optional): The path to the dataset configuration file. Can be a `Path` object, a string, or `None`. Defaults to `None`. Returns: run.Config: A configuration object initialized with the dataset parsing function and the provided dataset configuration path. )dataset_config_path)rr r)rtr!r!r"blended_dataset_config_recipes rvi ir*defaultdataset_configglobal_batch_sizemicro_batch_sizegrad_acc_batchestensor_parallel_sizer]r`seed#use_megatron_comm_overlap_llama3_8kworkersdirenable_preemptiontflops_callback gc_intervalnsys_profilingnsys_start_step nsys_end_step nsys_ranksra ckpt_format)rVrBrestore_optimizer_from_ckpt resume_path wandb_project wandb_namenamec&.KsBt| | | d}'|stjt| ||td}(ntjtt|t| ||| | td }(i})|"durY|#durQd| d|d|d|d |d |d |d | d |d|}#t|"|#d|)d<|!rftjt j |!d| d}*nd}*tjt j ddd||*d}+tt g},| r|, tjt| tdd|d|dkr|, tjtj||d|r|dur|}|, tjtj|||dd|r|, ttj|rtttt|'|(d}-|, |-tj|%|'td3id|d|d|d|d|d |d!|d"|d#|d$|d%|,d&|d'|d(|d)|d*|d+||(td3||$t|$d,d-|)td.d/d0d1|+d2S)4a  Create a pre-training recipe for a striped hyena model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dataset_config (str): a string specifying the path to the dataset config file (json schema) global_batch_size (int | None): global training batch size. If left None this will be inferred from cluster and model parallelism configurations. micro_batch_size (int): micro batch size per active device. num_nodes (int): number of nodes to use num_gpus_per_node (int): number of num_gpus_per_node per node to use. grad_acc_batches (int): number of training batches the gradients will be accumulated before performing each optimizer update. tensor_parallel_size (int): Number of tensor parallel splits. Typically between 1 and 8 to keep parallelism within a single node. pipeline_model_parallel_size (int): Pipeline model parallel size, splits the model by layer. context_parallel_size (int): Context model parallel size, how many splits to make across the sequence dimension to be processed in parallel similar to the strategy described in the ring attention paper. seq_length (int): The desired sequence length to train this model on. seed (int): Random seed to use for initialization model_size (str): model size to load use_megatron_comm_overlap_llama3_8k (bool): If using TP, this controls advanced overlap communications which can improve performance during pretraining. workers (int): Number of workers to use for per-device batch creation. val_check_interval (int): How often the model evaluates during training. dir (str): Directory to save logs and checkpoints enable_preemption (bool): Enable preemption when training on slurm, captures timeout signals and attempts to save a final checkpoint. align_param_gather (bool): Optimization for faster train step timing potentially through aligning parameter gather operations. tflops_callback (bool): Enable tflops callbacks for reporting training speed and device utilization. gc_interval (int): How often to run GC operations throughout training (default is auto) nsys_profiling (bool): Enable nsys profiling from NeMo repo. max_steps (int): Maximum number of steps the training model should take. nsys_start_step (int): Step for when NSYS will start collecting logs nsys_end_step (int): Step for when NSYS will stop collecting logs. nsys_ranks (list[int]): Ranks for processing nsys logs. Defaults to [0] if not specified. no_aligned_megatron_ddp (bool): Disables aligned megatron ddp optimizations. fp8 (bool): Whether to use fp8 precision for computations. grad_reduce_in_fp32 (bool): Boolean indicating whether to reduce the gradient weight in the FP32 format rather than the default bf16. ckpt_async_save (bool): Bool indicating whether to use asynchronous checkpoint saving. save_ckpt_format (Literal['torch_dist', 'zarr']): The checkpoint save method. The default torch_dist may result in larger checkpoints currently, but is the preferred option in the long run. resume_path (str): If specified starting weights will be loaded from this checkpoint rather than being randomly initialized. restore_optimizer_from_ckpt (bool): when loading checkpoint, try to load the optimizer. wandb_project (str): if set, logging to wandb will happen wandb_name (str): override default name for the wandb log. Returns: run.Partial: Partial configuration for pre-training. )r$r&r%)r&ryrzr/)paths dataset_clsr&rzryr} num_workersr/Nz heyna-size-z-TPz-PPz-CPz-GBSz-MBSz-GRFP32z-ALIGNz-NODESz-FP8)projectrrT)pathload_model_stateload_optim_stateF)resume_if_existsresume_ignore_no_checkpointresume_past_endresume_from_directoryrestore_config)r%tp_comm_overlap_cfgwgrad_deferral_limit(overlap_param_gather_with_optimizer_steprRr)gc_interval_traingc_interval_val) start_stepend_stepranks gen_shapehyenarKrIrCrDrGrHrJrLrMrNrrPrQrRrSrTrUr)rrrga2U0*3?giUMu>i )max_lrmin_lr warmup_steps)modelrrdatalogoptimresumer!)r;rr rr#r rvrrrm RestoreConfig AutoResumerappendrr nl_callbacksGarbageCollectionCallback NsysCallbackPreemptionCallbackrrPartialrsrrr).rxryrzrIrJr{r|r]r`r&r}r$r~rrLrrrRrrrrKrrrrSrQrPrTrarrNrrrrrfnkwargs model_run_cfg data_run_cfg extra_loggers restore_cfg nemo_resumerflop_meas_callbackr!r!r"pretrain_recipe_creaters0`            rrcCstS)z Creates and returns a tokenizer configuration. Returns: run.Config[TokenizerSpec]: A configuration object for the tokenizer. )r#r!r!r!r"r/sr/cCs td||S)z Factory function to create a Hyena model configuration. Returns: run.Config[pl.LightningModule]: Configuration for a Hyena model. r()r;)r%r&r!r!r"rs r)targetr)rcOst|i|S)aP Create a pre-training recipe for a Hyena model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. r)rargsrr!r!r"pretrain_recipesrcOs$|dusJdt|||d|S) Nz&resume_path None, invalid for finetune)rrr)rrrrr!r!r"finetune_recipesr)r<r=NNr=Fr=r<r>r>r?r@rANFTTFTrB)N)M dataclassesrpathlibrtypingrrlightning.pytorchpytorchplnemo_runrtorch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrnemorrmnemo.collectionsr 1nemo.collections.common.tokenizers.tokenizer_specr nemo.collections.llm.apir r nemo.collections.llm.gpt.datar ,nemo.collections.llm.gpt.data.megatron.hyenarr"nemo.collections.llm.gpt.data.mockr(nemo.collections.llm.recipes.log.defaultrrr'nemo.collections.llm.recipes.optim.adamr6nemo.collections.llm.recipes.precision.mixed_precisionrr;nemo.collections.llm.recipes.tp_overlap_configs.userbuffersr3nemo.collections.nlp.modules.common.tokenizer_utilsrnemo.lightning.pytorchrr/nemo.lightning.pytorch.callbacks.flops_callbackr6nemo.lightning.pytorch.callbacks.megatron_comm_overlaprnemo.utils.exp_managerrNAMEr#strboolintr LightningModuler;dtypelistrprsrvrrclifactoryr/rrrr!r!r!r"s                 ".       v      !"#$%(  i