o wi"c @sddlZddlZddlZddlmZmZz ddlmZdZ Wn e e fy+dZ Ynwzddl m Z ddlmZmZmZmZmZmZmZmZdZWn e e fyWdZYnwzddlmZmZmZmZmZdZWn*e e fyed e rdd l m!Zdd l"mZmZmZdd l"m#ZdZYnw                dde$fddZ%ddZ&ddZ'    dddZ(dS)N)AppStatelogging)set_logging_levelTF)tensor_parallel) RankGenerator get_pipeline_model_parallel_rankset_expert_model_parallel_rank$set_expert_model_parallel_world_size set_pipeline_model_parallel_rank&set_pipeline_model_parallel_world_sizeset_tensor_model_parallel_rank$set_tensor_model_parallel_world_size)!ConstantNumMicroBatchesCalculatorget_current_global_batch_sizeget_micro_batch_sizeget_num_microbatches init_num_microbatches_calculatorzCMegatron num_microbatches_calculator not found, using Apex version.)ConstantNumMicroBatches)rrr)setup_microbatch_calculatoruse_gloo_process_groupscCsr|dus |dks Jd| dkr| dks| |ksJdt}||_||_||_||_||_||_||_||_| |_ | |_ | |_ | |_ ||_ ||_||_||_||_||_||_t||||||| ||| | |d \|_|_|_|_|_|_|_|_t|jt|jt|jt |jt!|j|j t"|jt#j$j%|d|durt&||r+| dur+t'rddl(m)}|durt*||| |j|dd nct+|t,rt-|ksJt.| ksJt/|| |jksJnCt0d ddl1m)}|durt*||| |j|dd n(t+|t,r't-|ksJt.| ksJt/|| |jks&Jnt0d d |_2t3r7t4|dSdS) z)Initialize model parallel groups in NeMo.Nr1pipeline_model_parallel_split_rank is deprecated.pencoder_pipeline_model_parallel_size is temporarily unavailable. We are working on a refactoring to add it back.) world_sizeranktensor_model_parallel_size_pipeline_model_parallel_size_%virtual_pipeline_model_parallel_size_#pipeline_model_parallel_split_rank_context_parallel_size_expert_model_parallel_size_expert_tensor_parallel_size_#encoder_tensor_model_parallel_size_%encoder_pipeline_model_parallel_size_use_tp_pp_dp_mapping)use_te_rng_tracker)#_GLOBAL_NUM_MICROBATCHES_CALCULATORF)rglobal_batch_sizemicro_batch_sizedata_parallel_sizerampup_batch_sizedecrease_batch_size_if_neededz*Microbatch calculator already initialized.T)5r global_rankr local_rankr&expert_model_parallel_sizetensor_model_parallel_sizepipeline_model_parallel_size$virtual_pipeline_model_parallel_sizecontext_parallel_size"encoder_tensor_model_parallel_size$encoder_pipeline_model_parallel_size$pipeline_model_parallel_comm_backenduse_fp8 use_sharpinit_mpi_proc_groupexpert_tensor_parallel_size#num_distributed_optimizer_instancesnccl_communicator_config_pathrfake_initialize_model_paralleltensor_model_parallel_rankpipeline_model_parallel_rankexpert_model_parallel_rankexpert_tensor_parallel_rankmodel_parallel_sizer+"pipeline_model_parallel_split_rank$virtual_pipeline_model_parallel_rankr r r rr r rrandominitialize_rng_tracker_set_random_seedMCORE_MB_CALCULATOR)megatron.core.num_microbatches_calculatorr(r isinstancerrrr Exception(apex.transformer.pipeline_parallel.utils_is_megatron_initialized HAVE_APEXr)rr.r/r1r0r;r2r3rDr7r4r5r6r*r)r,r8r:seedapex_transformer_log_levelr&r'r<r=r9r app_stater(rSY/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/lightning/megatron_init.py"initialize_model_parallel_for_nemoNs              rUcCsl|dur/|dkr/|dt}t|tj|t|tjdkr-t |dSdSt d |)z$Set random seed for reproducability.Nrdz'Seed ({}) should be a positive integer.) rrFrPnptorch manual_seedcuda device_countrmodel_parallel_cuda_manual_seed ValueErrorformat)seed_rPrSrSrTrHs   rHcCsftjdkr1tjdtjdtjdtjdtjdtjdtj ddSdS)z%Set PyTorch JIT layer fusion options.z1.10.0a0+0aef44cTFN) rX __version___C_jit_set_profiling_executor_jit_set_profiling_mode_jit_override_can_fuse_on_cpu_jit_override_can_fuse_on_gpu_jit_set_texpr_fuser_enabled_jit_set_nvfuser_enabled%_debug_set_autodiff_subgraph_inliningrSrSrSrTset_jit_fusion_optionss       ric 4sX|dusJd| dkr| dks| |ksJdt||} t||} | | }t||}| dur2d}n| }| dkr?| dkr?| }n| }|dkrS|dksKJ|| ksSJd|||}| | |}||}||dksJd|d| d| d |d |d |d |||}||}||}|||ksJd}|durd}|dkrt|d |||| rdndddndt| d || || rdnd|d|dur| }||| }||}||dkrtd|d|dt|||| d | rdnd|d| r| d ks||ksJdddksJddddd<fdd }g}|dD]}||vrDt|}td|d|q.|dD]}||||vrc|} td|d| qJ||}!td |td!|d"|!g}"|d#D]}|"|||vr|}#td|d$|#q|#|}$td%|"td!|d&|$g}%|d'D]}|%|||vrtd|d(t|qtd)|%g}&d}'|d*D]}|&|||vr|}'td|d+|'q|'|}(td,|&td|d-|(d})|durV|d krVg}*|d.d/d0D]}|*|||vrAt||})q.td1|*td|d2|)d}+|dur|d krg},|d*d/d0D]}|,|||vr}t||}+qjtd3|,td|d4|+g}-g}.d}/d}0d}1|dD]L}|-|||vr|}/td|d5|/t |d kr|d|d6g}2|.|2n |}2|.t|2||2vrt|2}0td|d7|0q|/|}3|0dur|0|}1td8|-td|d9|3td:|-td|d;|1|(|3|)|+||||fS)=a Fake initialize model data parallel groups so that we can instantiate model parallel models before DDP is initialized. This is needed because PTL execution flow is init model, init trainer -> call trainer.fit(model). DDP is initialized during .fit. This function is taken from megatron.core.parallel_state and modified so that the distributed groups are not created. We only need the tensor parallel and pipeline parallel ranks to instantiate the model. Arguments: tensor_model_parallel_size: number of GPUs used to parallelize model tensor. pipeline_model_parallel_size: number of GPUs used to parallelize model pipeline. context_parallel_size: number of GPUs used to parallelize tokens of each input. Let's say we have a total of 16 GPUs denoted by g0 ... g15 and we use 2 GPUs to parallelize the model tensor, and 4 GPUs to parallelize the model pipeline. The present function will create 8 tensor model-parallel groups, 4 pipeline model-parallel groups and 8 data-parallel groups as: 8 data_parallel groups: [g0, g2], [g1, g3], [g4, g6], [g5, g7], [g8, g10], [g9, g11], [g12, g14], [g13, g15] 8 tensor model-parallel groups: [g0, g1], [g2, g3], [g4, g5], [g6, g7], [g8, g9], [g10, g11], [g12, g13], [g14, g15] 4 pipeline model-parallel groups: [g0, g4, g8, g12], [g1, g5, g9, g13], [g2, g6, g10, g14], [g3, g7, g11, g15] Note that for efficiency, the caller should make sure adjacent ranks are on the same DGX box. For example if we are using 2 DGX-1 boxes with a total of 16 GPUs, rank 0 to 7 belong to the first box and ranks 8 to 15 belong to the second box. Nrrrz9We do not support encoders with more TP than the decoder.z world_size: zM must be divisible by total world_size: (decoder_)tensor_model_parallel_size z* * (decoder_)pipeline_model_parallel_size z# * (decoder_)context_parallel_size z& + encoder_tensor_model_parallel_size z( * encoder_pipeline_model_parallel_size z * context_parallel_size rztp-cp-ep-pp-dpztp-cp-ep-dp-pp)tpepdpppcporder rank_offsetzdecoder world_size (zB) is not divisible by expert_tensor_model_pipeline_parallel size ()zmWhen not using pp-last rank ordering, the data parallel size of the attention and moe layers must be the samermzaPipeline parallel groups are expected to be the same for Non-Expert and Expert part, but got z and Fc;sddlm} |rj|fi|}n j|fi|}dur+|D]}|Vq#dSj|fi|}|dkrKt|||D] \}}||Vq?dS|dkrjt|t|ksYJt||D] \}}||Vq^dS|D]}|Vql|D]}|VqtdS)Nr)cyclermtp-pp) itertoolsrr get_ranksziplen) group_type is_expertkwargsrrd_ranksxe_ranksydecoder_rank_generatorencoder_rank_generatorexpert_decoder_rank_generatorrSrTgenerator_wrappers2   z9fake_initialize_model_parallel..generator_wrapperrlzRank z has data parallel group : zdp-cpz< has combined group of data parallel and context parallel : z>All data parallel group ranks with context parallel combined: zRanks z has data parallel rank: rnz has context parallel group: z"All context parallel group ranks: z has context parallel rank: rsz has model parallel group: z All model parallel group ranks: rjz" has tensor model parallel group: z'All tensor model parallel group ranks: z! has tensor model parallel rank: rkT)ryz'All expert model parallel group ranks: z! has expert model parallel rank: z(All expert tensor parallel group ranks: z" has expert tensor parallel rank: z$ has pipeline model parallel group: z has embedding group: z)All pipeline model parallel group ranks: z" has pipeline model parallel rank zAll embedding group ranks: z has embedding rank: )F) minr RuntimeErrorrulistrinfoappendindexrw)4rrrrr rr"r#r!r$r%r&r1r2rCr4r6r5encoder_model_sizedecoder_model_sizetotal_model_sizer+encoder_world_sizedecoder_world_sizerE*expert_tensor_model_pipeline_parallel_sizeexpert_data_parallel_sizer%all_data_parallel_group_ranks_with_cpranksdata_parallel_group ranks_with_cpdata_parallel_group_with_cpdata_parallel_rank all_context_parallel_group_rankscontext_parallel_groupcontext_parallel_rankall_model_parallel_group_ranks%all_tensor_model_parallel_group_rankstensor_model_parallel_groupr?rAall_expert_model_parallel_ranksrB all_expert_tensor_parallel_ranks'all_pipeline_model_parallel_group_ranksall_embedding_group_rankspipeline_model_parallel_groupembedding_groupembedding_rankembedding_ranksr@rSrrTr>s,                    #                             r>)rrNrNNNrrrNNNFFrrFFrNFT)NNrNrrrF))rFnumpyrWrX nemo.utilsrrapex.transformer.log_utilrrO ImportErrorModuleNotFoundError megatron.corermegatron.core.parallel_staterrrr r r r r HAVE_MEGATRON_CORErJrrrrrrIwarningapex.transformer.microbatchesrrMrboolrUrHrir>rSrSrSrTs  (