o wi-@s,ddlZddlmZddlmZddZdS)N)get_nemorun_home)DEFAULT_NEMO_HOMEc Cs(tjdd}|jddtddd|jdd td dd|jd d tgd ddd|jddtdtdtd|jddtddddddg}|jddtd|ddd|jdd td!d"gd#dd!d$d%}|jd&d'tgd(|dd)d$|jd*d+d,d-d.|jd/d0d1d-d.|jd2d3td4dd5d|jd6d7d8d-d.|jd9d:d;d-d.|jddd5d|jd?d@tdAdd5d|jdBdCtdDdd5d|jdEdFdGdHgdIdHdJ|jdKdLtdMd5dNdOdPdQtg}|jdRdStd|tdN|jdTdUdVdd-dW|jdXdYtdZdd5d|jd[d\td]dd5d|jd^d_td`dd5d|jdadbtdcdd5d|jdddetdfdd5d|jdgdhdidjdkd5dldd5dm|jdndotdd5dp|jdqdrtdd5dp|jdsdttdudd5d|jdvdwtdxddyd|jdzd{td|dd}dd~d}|jddd|dd5d|jddd|dd5d|jddd|dd5d|jddd|dd5d|jddd|dd5d|jddtddd5d|jddtddd5d|jdtddd5ddd}|jdddd5tddd5d|jdd|ddgd|jddd-dd|jdddd-ddd|jdd|dd5d|jddd|dd5d|S)z Command line arguments correspong to Slurm cluster and NeMo2.0 for running pre-training and fine-tuning experiments. z/NeMo2.0 Performance Pretraining and Fine-Tuning) descriptionz-az --accountz#Slurm account to use for experimentT)typehelprequiredz-pz --partitionz%Slurm partition to use for experimentz-gz--gpu)h100b200gb200zTarget gpu type.)rchoicesrrz-lz --log_dirz6Directory for logging experiment results. Defaults to F)rrrdefaultz-tz --time_limitzUMaximum time limit to run experiment for. Defaults to 30 minutes (format- 'HH:MM:SS')z00:30:00zaNeMo container to use for experiment. Defaults to latest dev container- 'nvcr.io/nvidia/nemo:dev'zBMake sure your NGC credentials are accessible in your environment.z-iz--container_image znvcr.io/nvidia/nemo:devz-cz--compute_dtypebf16fp8z9Compute precision. Options- bf16 or fp8. Defaults to bf16)rr rrr zFP8 recipe. Options- ds (per-tensor delayed scaling), cs (per-tensor current scaling), mxfp8, ss (subchannel scaling). Defaults to dsz-frz --fp8_recipe)dscsmxfp8ssrz-enz --enable_nsysz)Enable Nsys profiling. Diabled by default store_true)ractionz-emz--enable_memory_profilez1Enable memory usage profiling. Diabled by defaultz-mpz--memory_profile_out_pathz+Path to the output file of memory profilingNz-tbz --tensorboardz/Enable tensorboard logging. Disabled by defaultz-wdz--wandbz)Enable wandb logging. Disabled by defaultz-wdkz --wandb_keyz6wandb key. Needed for wandb logger projetion to serverz-wdpz--wandb_prj_namezwandb project namez-wdjz--wandb_job_namezwandb job namez-fz --finetuningsftloraz,Finetuning scheme to use. Defaults to 'lora')r rr z-hfz --hf_tokenzWHuggingFace token. Defaults to None. Required for accessing tokenizers and checkpoints.)rrr z]Sets env var `NEMO_HOME` (on compute node using sbatch script)- directory where NeMo searcheszjfor models and checkpoints. This saves a lot of time (especially for bigger models) if checkpoints alreadyzPexist here. Missing files will be downloaded here from HuggingFace. Defaults to z-nhz --nemo_homez-dz--dryrunzGIf true, prints sbatch script to terminal without launching experiment.)rrrz-tpz--tensor_parallel_sizez?Intra-layer model parallelism. Splits tensors across GPU ranks.z-ppz--pipeline_parallel_sizezJInter-layer model parallelism. Splits transformer layers across GPU ranks.z-cpz--context_parallel_sizez?Splits network input along sequence dimension across GPU ranks.z-vpz --virtual_pipeline_parallel_sizez]Number of virtual blocks per pipeline model parallel rank is the virtual model parallel size.z-epz--expert_parallel_sizez;Distributes Moe Experts across sub data parallel dimension.z-etz--expert_tensor_parallel_sizecSs|durt|SdS)N)int)xr`/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/scripts/performance/argument_parser.pysz parse_cli_args..?zIntra-layer tensor model parallelsm for expert layer. Splits tensors across GPU ranks. Use -et/--expert_tensor_parallel_size for None or -et/--expert_tensor_parallel_size )rnargsconstrrr z-mbz--micro_batch_size)rrr z-gbz--global_batch_sizez-ngz --num_gpuszNumber of gpus.z-gnz--gpus_per_nodez&Number of gpus per node. Defaults to 8z-msz --max_stepsz&Number of train steps. Defaults to 100dcSs.|dvrdS|dvrdStd|)N)true1tyesyT)false0fnonFz$Invalid value for boolean argument: )lower ValueErrorargrrrbool_args  z parse_cli_args..bool_argz-cgz --cuda_graphsz'Enable CUDA graphs. Disabled by default)rrrr z-fsdpz--use_mcore_fsdpz6Enable Megatron Core (Mcore) FSDP. Disabled by defaultz-fsdp_dbz--use_fsdp_double_bufferz.Enable FSDP double buffer. Disabled by defaultz-ubrz--use_user_buffer_registrationz4Enable user buffer registration. Disabled by defaultz-sharpz --use_sharpz!Enable sharp. Disabled by defaultz-rlz--recompute_layerszNumber of Transformer layers to recompute, where all the intermediate activations of a Transformer layer are computed. Defaults to Nonez-olz--activation_offload_layerszKNumber of Transformer layers to offload to the CPU memory. Defaults to Nonez--nccl_communicator_config_pathz*Path to NCCL communicator config yaml filecSs |dS)N,)splitr/rrrlist_of_stringsas z'parse_cli_args..list_of_stringsz-rmz--recompute_modules*z{List of modules to perform selective activation recompute. Users can provide 0 or any number of arguments. Defaults to None)rr rrrr z-cmz--custom_mountsz Comma separated string of mountsz--use_hf_tokenizerz\Use HuggingFace tokenizer. Disabled by default. Null tokenizer will be used if not provided.)rrrz-dcdfrz#--dump_config_diff_from_base_recipezs