o }oi V@sPUddlZddlmZddlZddlZddlmZmZddlm Z m Z ddl m Z ddl mZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZddl m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)dej*j+_,ej-ej.ej/ej0dZ1e2e3eej4fe5d<ddZ6ddZ7e8dkr e7dSdS)N)Type)LearningRateMonitorRichModelSummary)TensorBoardLogger WandbLogger)DistributedDataParallelConfig)OptimizerConfig) lightning)llm)MockDataModulePreTrainingDataModule)get_nmt_tokenizer) NeMoLogger) callbacks)ModelCheckpoint)MegatronCommOverlapCallback)CosineAnnealingScheduler)MegatronOptimizerModule) RestoreConfig)TimingCallbackT)4B8B47B56B model_optionscCsvtjdtjd}|}|jddtddd|jdd d d |jd d d d |jdtdd|jdtddd|jdtddd|jdtddd|jdtddd|jdtddd|jdtddd|jdd d d!d"|jd#td$d%d|jd&tdd'd|jd(tdd)d|jd*tdd+d|jd,d-d.d/d0d1|jd2d d3d |jd4d d5d |jd6tdd7d|jd8tdd9d|jd:tdd;d|jdtd?d|jd@d d dAd"|jdBd d dCd"|jdDd d dEd"|jdFtgdGdHdIdJ|jdKd d dL|jdMttt dNdOdJ|jdPtd.dQdR|jdStd dTdR|jdUtdVd dTdW|jdXtdYdZd|jd[tdd d\dW|jd]tdd^d|jd_t d`dad|jdbd dcd |jddtdedfd|jdgtdhdid|jdjtdkdld|jdmd d dnd"|jdod d dL|jdptdqdrgdqdsdJ|jdtd d dud"|jdvd d dwd"|jdxtdkdyd|jdztd{d|jd|td}d|jd~d d dd"|jdt ddd|jdt ddd|jdtddd|jdd d dd"|jdtd dkdd|jdtd ddR|jdtdd dkgdd|jdt ddd|jdt dd|jdd d dd"|jdd d dd"|jdtddd| S)z'Parse arguments for NMH model training.z'Train a nemotronh model using NeMo 2.0.) descriptionformatter_classz-dz--dataset-configNzCPath to the blended / weighted training dataset configuration YAML.)typedefaulthelpz --mock-data store_truezZTrain with Mock data (for testing/debugging), either set this or provide a dataset config.)actionrz--sftzSFT with SQUAD.z --dataset-dirzAbsolute path to the dataset directory. Defaults to using the absolute or relative paths (dataset_prefix) specified in the dataset config YAML.)rrz --num-nodesz3Number of nodes to use for training, defaults to 1.z --devicesz5Number of devices to use for training, defaults to 1.z --seq-lengthi zTraining sequence lengthz--tensor-parallel-sizez+Order of tensor parallelism. Defaults to 1.z--pipeline-model-parallel-sizez-Order of pipeline parallelism. Defaults to 1.z--context-parallel-sizez,Order of context parallelism. Defaults to 1.z --no-wandbFzDisable Wandb logging)r!rrz--wandb-project nemotronhzWandb project namez--wandb-run-idzWandb run identifierz --wandb-groupz3A unique string shared by all runs in a given groupz--wandb-job-typezA unique string representing a type of run, which is useful when you're grouping runs together into larger experiments using group.z--disable-checkpointing store_falseTcreate_checkpoint_callbackz,Disable creating a ModelCheckpoint callback.)r!rdestrz--sequence-parallelz#Set to enable sequence parallelism.z--fp8zSet to enable FP8z--micro-batch-sizez,Micro-batch size for data-parallel training.z--global-batch-sizez\Global batch size for training. If set to None, infer it from the TP, CP, and PP parameters.z--grad-acc-batchesz/Number of batches to accumulate gradients over.z --max-stepsz*Number of training optimizer update steps.z--val-check-intervalzFNumber of steps between validation measurements and model checkpoints.z--grad-reduce-in-fp32zGradient reduce in FP32.z --fp8-wgradz?Faster option that is maybe less accurate (TBD) when using fp8.z--no-aligned-megatron-ddpz'Do not do aligned gradient updates etc.z--tp-comm-overlap-backend)ncclmpigloor'z4TP communication backend to use. Defaults to 'nccl'.)rchoicesrrz--align-param-gather)r!rz --model-sizerzModel architecture to usez--experiment-dirz4Directory to write model checkpoints and results to.)rrequiredrz --vocab-filezPath to tokenizer vocab file.z--hf-tokenizer-nameznvidia/Nemotron-H-8B-Base-8K)rrr+rz--limit-val-batcheszNumber of validation stepsz--log-every-n-stepsz Number of steps between logging.z --ckpt-dirzUDirectory to restore an initial checkpoint from. Use this for supervised fine-tuning.z--wdg{Gz?zWeight decay for optimizer.z--restore-optimizer-from-ckptzCRestore optimizer state from initial checkpoint. Defaults to False.z--seedizSet random seed for training.z --workersz*Number of workers to use for data loading.z --gc-intervalrzeSet to a value > 0 if you want to synchronize garbage collection, will do gc every gc-interval steps.z--enable-preemptionzTEnable preemption hooks. If enabled this will save a checkpoint whenver slurm exits.z--ckpt-async-savez --ckpt-format torch_distzarrzSpecify checkpoint format to use. Defaults to 'torch_dist', as 'zarr' is deprecated. Only use if resuming training from a zarr checkpoint.z--cross-entropy-loss-fusionzjUse the faster, but maybe less accurate fused form of cross entropy, which also has bf16 grads internally.z--no-fp32-residual-connectionzWIf set, turn off fp32 residual connections which may be faster but may impact accuracy.z--debug-ddp-parity-freqz:Set to value > 0 to debug DDP weight parity between ranks.z--hybrid-override-patternz]Override the hybrid override pattern in the config (specifies mamba layer ordering and type).z --num-layerszHIf set, override the number of layers specified in the requested config.z--log-parameters-and-shapesz8Log training parameters shapes and dtypes for debugging.z--lrga2U0*3?zLearning rate.z--min-lrgiUMu>z&Min learning rate in cosine annealing.z--warmup-stepsi z*Number of warmup steps in cosine annealingz--nsys-profilingaSEnable targeted `nsys` profiling on the training loop for a defined step range. To actually get profiling output you must run the whole program with `nsys`. For example: `nsys profile -s none -o output_report_name -t cuda,nvtx --force-overwrite true --capture-range=cudaProfilerApi --capture-range-end=stop [regular python command here]`z--nsys-start-stepz%Start nsys profiling after this step.)rr+rrz--nsys-end-stepz#End nsys profiling after this step.z --nsys-ranks+z&Enable nsys profiling for these ranks.)rnargsr+rrz --clip-gradg?zGGrad clip value. Note that when using DDP this may need to be inflated.z--seq-len-interpolation-factorzAdjusts the linear scaling of ROPE (Rotary Position Embedding) for context extension. Set this factor relative to your base context length e.g., for an original context length of 8192 and an extended context length of 524288, use 524288/8192 = 64.z--overlap-param-gatherzOverlap the parameter gather with the optimizer step. This is currently disabled due to a NeMo bug when using DDP. Making this an option defaulting to False is a temporary solution until the bug is fixed.z--overlap-grad-reducez4Overlap the gradient reduce with the optimizer step.z --bucket-sizei@zDDP bucket size.) argparseArgumentParserArgumentDefaultsHelpFormatteradd_mutually_exclusive_group add_argumentstrintsortedrkeysfloat parse_args)parser data_groupr?b/home/ubuntu/.local/lib/python3.10/site-packages/tests/collections/llm/gpt/model/test_nemotronh.pyr<0s   r<cCst}|jrtdd|jdd}ntd|jdd}|j}|jr+t|j|j||j |d}n%|j r@t j |j|j|j||j ddid }nt |j|j|j||j|j |d }|j|jrWd nd|j|j d }|jrh|j|d <|jrp|j|d<|jtvr}td|jt|jd?i|}t j||jd}tddttg}|jrt|j|j ddddd}|!||j"r|!t#$|j%dkr|!t#j&|j%d|j'r|!t#(|!t)|j*|j+d|j,dkr|!t#j-|j,|j,d|j.r |j/dur|j0} n|j/} |!t#j1|j2| |j3ddg} i} |j4sQ|j5rQt6d|jd|j7d|j8d|j9d|d |jd!|j:d"|jd#|j;|j<|j=|j>|j5|j d$} | !| | | d%<t?d&d'} | | d(<| !| t@d?d)|j i| }tAd|jB|jC|jDd*}tEjF||j7|j8|j9tGjH|jdd|jI|jJd+d, }tEjK|jL|j;|j0d-|| ||jM|jNdd tEjOd.tGjH|jD|j:rd/ndd0d1|j:rd2nd3d4d4d5 |j|jd6}|jP|dd7tEjQddd |j |jRrtS|jRd|jTd8ndd9}|P||tUd:|jVd;d<|jW|jXddd=}tY|j0|jZ|j[d>}t\||}|]||^||dS)@z"Main function to run NMH training.tiktokenTiktokenTokenizerT)library model_name vocab_fileuse_fast huggingface)rCrDrF) seq_lengthmicro_batch_sizeglobal_batch_size num_workers tokenizerpad_to_max_length)rHrIrJrLrKdataset_kwargs)pathsrHrIrJseedrKrLF)rHdistribute_saved_activationscross_entropy_loss_fusionfp32_residual_connectionhybrid_override_pattern num_layerszInvalid model size: )rL) max_depth)every_n_train_stepsdirpath save_top_kalways_save_contextsave_optim_on_train_endsave_context_on_train_endr)interval) bucket_sizetp_comm_bootstrap_backend)gc_interval_traingc_interval_valN) start_stepend_stepranks gen_shapeznemotronh-size-z-TPz-PPz-CPz-GBSz-MBSFP8z-SEQLENz-NODES)namegroupjob_typeidprojectsave_dirwandbdummy)rn tensorboardlog_dir)check_for_nan_in_gradoverlap_grad_reduceoverlap_param_gathergrad_reduce_in_fp32log_all) ddptensor_model_parallel_sizepipeline_model_parallel_sizecontext_parallel_sizepipeline_dtypesequence_parallelckpt_load_optimizerckpt_save_optimizerckpt_async_savesave_ckpt_formatckpt_load_strictnessgpuz bf16-mixedhybrid tensorwiser"max most_recent) precision params_dtypervfp8 fp8_recipefp8_amax_history_lenfp8_amax_compute_algonum_layers_at_start_in_bf16num_layers_at_end_in_bf16)devices num_nodes max_steps acceleratorstrategyloggerrlog_every_n_stepslimit_val_batchesnum_sanity_val_stepsuse_distributed_samplerpluginsval_check_intervalenable_checkpointing)resume_if_exists)pathload_model_stateload_optim_state)rresume_ignore_no_checkpointresume_past_endresume_from_directoryrestore_configadamg?gffffff?) optimizerlr adam_beta1 adam_beta2 weight_decay clip_graduse_distributed_optimizerbf16)r warmup_stepsmin_lrr?)_r<rEr hf_tokenizer_namerJ mock_datar rHrIworkerssftr SquadDataModuler dataset_dirrPr}rRno_fp32_residual_connectionrTrU model_sizer ValueError MambaModelrLrrrr%rrexperiment_dirappendenable_preemption nl_callbacksPreemptionCallbackdebug_ddp_parity_freqDdpParityCheckerlog_parameters_and_shapesParameterDebuggerrr`tp_comm_overlap_backend gc_intervalGarbageCollectionCallbacknsys_profiling nsys_end_stepr NsysCallbacknsys_start_step nsys_ranksno_wandb wandb_projectrtensor_parallel_sizerzr{rr wandb_groupwandb_job_type wandb_run_idrrrrtrurvnlMegatronStrategytorchbfloat16r ckpt_formatTrainerrrrMegatronMixedPrecisionsetup AutoResumeckpt_dirrrestore_optimizer_from_ckptrrwdrrrrrconnectfit)argsrLrJdataconfig_modifiers_init mamba_configmodelrcheckpoint_callbackrloggersnemo_logger_kwargs wandb_logger tb_logger nemo_loggerrxrtrainerresume opt_configschedoptr?r?r@mainFs                  r__main__)9r2typingrr torch._dynamolightning.pytorch.callbacksrrlightning.pytorch.loggersrrmegatron.core.distributedrmegatron.core.optimizerrnemor rnemo.collectionsr nemo.collections.llm.gpt.datar r 3nemo.collections.nlp.modules.common.tokenizer_utilsr nemo.lightningrnemo.lightning.pytorchrr nemo.lightning.pytorch.callbacksr6nemo.lightning.pytorch.callbacks.megatron_comm_overlaprnemo.lightning.pytorch.optimr%nemo.lightning.pytorch.optim.megatronr'nemo.lightning.pytorch.strategies.utilsrnemo.utils.exp_managerr_dynamoconfigsuppress_errorsNemotronHConfig4BNemotronHConfig8BNemotronHConfig47BNemotronHConfig56Brdictr7 SSMConfig__annotations__r<r__name__r?r?r?r@sF                o