o }oiz] @sddlZddlZddlZddlmZmZz ddlmZdZ Wn e e fy+dZ Ynwz ddl m Z ddlmZmZmZmZmZmZmZmZmZdZWn e e fyYdZYnwzddlmZmZmZmZmZdZWn(e e fye d dd l!m"Zdd l#mZmZmZdd l#m$ZdZYnw                dde%fddZ&ddZ'ddZ(    dddZ)dS)N)AppStatelogging)set_logging_levelTF)tensor_parallel) RankGenerator get_pipeline_model_parallel_rankset_expert_model_parallel_rank$set_expert_model_parallel_world_size set_pipeline_model_parallel_rank&set_pipeline_model_parallel_split_rank&set_pipeline_model_parallel_world_sizeset_tensor_model_parallel_rank$set_tensor_model_parallel_world_size)!ConstantNumMicroBatchesCalculatorget_current_global_batch_sizeget_micro_batch_sizeget_num_microbatches init_num_microbatches_calculatorzCMegatron num_microbatches_calculator not found, using Apex version.)ConstantNumMicroBatches)rrr)setup_microbatch_calculatoruse_gloo_process_groupscCs>t}||_||_||_||_||_||_||_||_| |_ | |_ | |_ | |_ ||_ ||_||_||_||_||_||_t||||||| ||| | |d \|_|_|_|_|_|_|_|_t|jt|jt|jt |jt!|j|j t"|jt#|jt$j%j&|d|durt'||r| durt(rddl)m*}|durt+||| |j|ddn`t,|t-rt.|ksJt/| ksJt0|| |jksJn@t1dddl2m*}|durt+||| |j|ddn&t,|t-r t.|ksJt/| ksJt0|| |jks Jnt1dd |_3t4rt5|dSdS) z)Initialize model parallel groups in NeMo.) world_sizeranktensor_model_parallel_size_pipeline_model_parallel_size_%virtual_pipeline_model_parallel_size_#pipeline_model_parallel_split_rank_context_parallel_size_expert_model_parallel_size_expert_tensor_parallel_size_#encoder_tensor_model_parallel_size_%encoder_pipeline_model_parallel_size_use_tp_pp_dp_mapping)use_te_rng_trackerNr)#_GLOBAL_NUM_MICROBATCHES_CALCULATORF)rglobal_batch_sizemicro_batch_sizedata_parallel_sizerampup_batch_sizedecrease_batch_size_if_neededz*Microbatch calculator already initialized.T)6r global_rankr local_rankr%expert_model_parallel_sizetensor_model_parallel_sizepipeline_model_parallel_size$virtual_pipeline_model_parallel_sizecontext_parallel_size"encoder_tensor_model_parallel_size$encoder_pipeline_model_parallel_size$pipeline_model_parallel_comm_backenduse_fp8 use_sharpinit_mpi_proc_groupexpert_tensor_parallel_size#num_distributed_optimizer_instancesnccl_communicator_config_pathrfake_initialize_model_paralleltensor_model_parallel_rankpipeline_model_parallel_rankexpert_model_parallel_rankexpert_tensor_parallel_rankmodel_parallel_sizer*"pipeline_model_parallel_split_rank$virtual_pipeline_model_parallel_rankrr r rr r r rrandominitialize_rng_tracker_set_random_seedMCORE_MB_CALCULATOR)megatron.core.num_microbatches_calculatorr'r isinstancerrrr Exception(apex.transformer.pipeline_parallel.utils_is_megatron_initialized HAVE_APEXr)rr-r.r0r/r:r1r2rCr6r3r4r5r)r(r+r7r9seedapex_transformer_log_levelr%r&r;r<r8r app_stater'rRP/home/ubuntu/.local/lib/python3.10/site-packages/nemo/lightning/megatron_init.py"initialize_model_parallel_for_nemoMs              rTcCsl|dur/|dkr/|dt}t|tj|t|tjdkr-t |dSdSt d |)z$Set random seed for reproducability.Nrdz'Seed ({}) should be a positive integer.) rrErOnptorch manual_seedcuda device_countrmodel_parallel_cuda_manual_seed ValueErrorformat)seed_rOrRrRrSrGs   rGcCsftjdkr1tjdtjdtjdtjdtjdtjdtj ddSdS)z%Set PyTorch JIT layer fusion options.z1.10.0a0+0aef44cTFN) rW __version___C_jit_set_profiling_executor_jit_set_profiling_mode_jit_override_can_fuse_on_cpu_jit_override_can_fuse_on_gpu_jit_set_texpr_fuser_enabled_jit_set_nvfuser_enabled%_debug_set_autodiff_subgraph_inliningrRrRrRrSset_jit_fusion_optionss       rhc 2st||} t||} | | }t||}| durd}n| }| dkr'| dkr'| }n| }|dkr;|dks3J|| ks;Jd|||}| | |}||}||dksjJd|d| d| d|d|d |d |||}||}||}|||ks~Jd}|durd}|dkrt|d |||| rd nd ddndt| d || || rd nd |d|dur| }||| }||}||dkrtd|d|dt|||| d | rd nd |d| r| d ks||ksJdddksJddddd6fdd }g}|dD]}||vr)t|}td|d|q|dD]}||||vrH|} td|d| q/||}!td|td|d |!g}"|d!D]}|"|||vr|}#td|d"|#qh|#|}$td#|"td|d$|$g}%|d%D]}|%|||vrtd|d&t|qtd'|%g}&d}'|d(D]}|&|||vr|}'td|d)|'q|'|}(td*|&td|d+|(d})|dur!|d kr!|d,d-d.D]}||vrt||})qd}*|durC|d krC|d(d-d.D]}||vrAt||}*q3g}+g},d}-d}.d}/|dD]L}|+|||vrj|}-td|d/|-t |d kr|d|d0g}0|,|0n |}0|,t|0||0vrt|0}.td|d1|.qQ|-|}1|.dur|.|}/td2|+td|d3|1td4|+td|d5|/|(|1|)|*||||fS)7a Fake initialize model data parallel groups so that we can instantiate model parallel models before DDP is initialized. This is needed because PTL execution flow is init model, init trainer -> call trainer.fit(model). DDP is initialized during .fit. This function is taken from megatron.core.parallel_state and modified so that the distributed groups are not created. We only need the tensor parallel and pipeline parallel ranks to instantiate the model. Arguments: tensor_model_parallel_size: number of GPUs used to parallelize model tensor. pipeline_model_parallel_size: number of GPUs used to parallelize model pipeline. context_parallel_size: number of GPUs used to parallelize tokens of each input. Let's say we have a total of 16 GPUs denoted by g0 ... g15 and we use 2 GPUs to parallelize the model tensor, and 4 GPUs to parallelize the model pipeline. The present function will create 8 tensor model-parallel groups, 4 pipeline model-parallel groups and 8 data-parallel groups as: 8 data_parallel groups: [g0, g2], [g1, g3], [g4, g6], [g5, g7], [g8, g10], [g9, g11], [g12, g14], [g13, g15] 8 tensor model-parallel groups: [g0, g1], [g2, g3], [g4, g5], [g6, g7], [g8, g9], [g10, g11], [g12, g13], [g14, g15] 4 pipeline model-parallel groups: [g0, g4, g8, g12], [g1, g5, g9, g13], [g2, g6, g10, g14], [g3, g7, g11, g15] Note that for efficiency, the caller should make sure adjacent ranks are on the same DGX box. For example if we are using 2 DGX-1 boxes with a total of 16 GPUs, rank 0 to 7 belong to the first box and ranks 8 to 15 belong to the second box. Nrz9We do not support encoders with more TP than the decoder.z world_size: zM must be divisible by total world_size: (decoder_)tensor_model_parallel_size z* * (decoder_)pipeline_model_parallel_size z# * (decoder_)context_parallel_size z& + encoder_tensor_model_parallel_size z( * encoder_pipeline_model_parallel_size z * context_parallel_size rztp-cp-ep-pp-dpztp-cp-ep-dp-pp)tpepdpppcporder rank_offsetzdecoder world_size (zB) is not divisible by expert_tensor_model_pipeline_parallel size ()zmWhen not using pp-last rank ordering, the data parallel size of the attention and moe layers must be the samerlzaPipeline parallel groups are expected to be the same for Non-Expert and Expert part, but got z and Fc;sddlm} |rj|fi|}n j|fi|}dur+|D]}|Vq#dSj|fi|}|dkrKt|||D] \}}||Vq?dS|dkrjt|t|ksYJt||D] \}}||Vq^dS|D]}|Vql|D]}|VqtdS)Nr)cyclerltp-pp) itertoolsrq get_ranksziplen) group_type is_expertkwargsrqd_ranksxe_ranksydecoder_rank_generatorencoder_rank_generatorexpert_decoder_rank_generatorrRrSgenerator_wrappers2   z9fake_initialize_model_parallel..generator_wrapperrkzRank z has data parallel group : zdp-cpz< has combined group of data parallel and context parallel : z>All data parallel group ranks with context parallel combined: zRanks z has data parallel rank: rmz has context parallel group: z"All context parallel group ranks: z has context parallel rank: rrz has model parallel group: z All model parallel group ranks: riz" has tensor model parallel group: z'All tensor model parallel group ranks: z! has tensor model parallel rank: rjT)rxz$ has pipeline model parallel group: z has embedding group: z)All pipeline model parallel group ranks: z" has pipeline model parallel rank zAll embedding group ranks: z has embedding rank: )F) minr RuntimeErrorrtlistrinfoappendindexrv)2rrrrrrr!r"r r#r$r%r0r1rBr3r5r4encoder_model_sizedecoder_model_sizetotal_model_sizer*encoder_world_sizedecoder_world_sizerD*expert_tensor_model_pipeline_parallel_sizeexpert_data_parallel_sizer%all_data_parallel_group_ranks_with_cpranksdata_parallel_group ranks_with_cpdata_parallel_group_with_cpdata_parallel_rank all_context_parallel_group_rankscontext_parallel_groupcontext_parallel_rankall_model_parallel_group_ranks%all_tensor_model_parallel_group_rankstensor_model_parallel_groupr>r@rA'all_pipeline_model_parallel_group_ranksall_embedding_group_rankspipeline_model_parallel_groupembedding_groupembedding_rankembedding_ranksr?rRr~rSr=s~ -                  #                           r=)rrNrNNNrrrNNNFFrrFFrNFT)NNrNrrrF)*rEnumpyrVrW nemo.utilsrrapex.transformer.log_utilrrN ImportErrorModuleNotFoundError megatron.corermegatron.core.parallel_staterrrr r r r r rHAVE_MEGATRON_CORErIrrrrrrHwarningapex.transformer.microbatchesrrLrboolrTrGrhr=rRrRrRrSs  ,