o }oiJ@sdZddlZddlZddlmZmZmZmZmZm Z ddl Z ddl m Z ddl m Z mZz"ddlmZddlmZddlmZdd lmZdd lmZd ZWn eefy[d ZYnwzdd lmZmZddlm Z d Z!Wn eefy{d Z!Ynwde fddZ"Gddde#Z$  dVde j de j de%de j de%de%de%fddZ&ddZ'dd Z(d!d"Z)d#d$Z*d%d&Z+d'd(Z,e j-j.d)d*Z/d+d,Z0ze j1Z2Wne j-j.Z2Ye2d-d.Z3e j-j.d/d0Z4d1d2Z5 dWd3d4Z6d5d6Z7d7d8Z8d9d:Z9d;d<Z:d=d>Z;d?d@ZdEdFZ?dGe e j@jAee j@jAfdHeeBe j@jCffdIdJZDdGe e j@jAee j@jAfdHeeeBee j@jCffdKdLZEdWdMee%fdNdOZF dWdPe eee j fdQeGdMee%dHefdRdSZHdTdUZIdS)XzUtilities for models.N)DictIteratorListOptionalTupleUnion)Tensor)logging logging_mode)MixedFusedRMSNorm)FusedLayerNorm) AttnMaskType) FastLayerNorm) listify_modelTF)parallel_statetensor_parallel)1linear_with_grad_accumulation_and_async_allreduceinputcCs|td|S)zr Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs gZd;?)torchsigmoid)rrf/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/nlp/modules/common/megatron/utils.pyApproxGELUActivation1srcs(eZdZdZfddZddZZS)ApexGuardDefaultszQ This class can be used to replace missing classes when apex is missing. cstdSN)super__init__)self __class__rrr=szApexGuardDefaults.__init__cCsdSrr)ritemrrr __getattr__@szApexGuardDefaults.__getattr__)__name__ __module__ __qualname____doc__rr! __classcell__rrrrr8s rinput_word_embeddings_weightparallel_outputbias%async_tensor_model_parallel_allreducesequence_parallelgradient_accumulation_fusionc CsXtdk}|o |o | }|s|r|} nt|} t| |||||d} |r'| St| S)aLanguage Model logits using word embedding weights. Args: input_ (torch.Tensor): [b, s, h] word_embeddings_weight (torch.Tensor): [(padded) vocab size, h] parallel_output (bool): False will gather logits from tensor model parallel region bias (torch.Tensor, optional): bias tensor. Defaults to None. async_tensor_model_parallel_allreduce (bool, optional): Defaults to False. sequence_parallel (bool, optional): If True will use sequence parallelism. Defaults to False. gradient_accumulation_fusioa (bool, optional): If True fuse gradient accumulation to WGRAD GEMM Returns: torch.Tensor: [b, s, (padded) vocab size] )rweightr*r-allreduce_dgradr,)r$get_tensor_model_parallel_world_sizer$copy_to_tensor_model_parallel_regionr(gather_from_tensor_model_parallel_region) r'r(r)r*r+r,r-tensor_model_parallelr0input_parallellogits_parallelrrrparallel_lm_logitsDs    r7cfdd}|S)z!Init method based on N(0, sigma).ctjjj|ddSN)meanstdrnninitnormal_tensorsigmarrinit_|z!init_method_normal..init_r)rErFrrDrinit_method_normalys rHcr8)Ncstjjj|dS)N)a)rr?r@kaiming_uniform_rBvalrrrFsz*init_method_kaiming_uniform..init_rrLrFrrKrinit_method_kaiming_uniform rNcr8)Ncstjj|Sr)rr?r@ constant_rBrKrrrFsz init_method_const..init_rrMrrKrinit_method_constrOrQcs"|td|fdd}|S)z3Init method based on N(0, sigma/sqrt(2*num_layers).g@cr9r:r>rBr=rrrFrGz(scaled_init_method_normal..init_)mathsqrt)rE num_layersrFrrRrscaled_init_method_normals rVcCs||d|S)Ng) masked_fill_)attention_scoresattention_maskrrrattention_mask_funcs rZcCsPtj||}||jt|jWd|S1s!wY|S)z/Simple linear layer with weight initialization.N)rr?Linearr/no_gradr*zero_)rowscolumns init_methodlayerrrrget_linear_layers    rbcCs*d|dtd|dd||S)zOpenAI's gelu implementation.?g?gQ63E?gHm?)rtanhxrrr gelu_impls*rgcCst|Sr)rgrerrr openai_gelusrhcCsttjj|dS)N)rpowr? functionalrelurerrr squared_relusrmcCs4|dt|dj|jdt|j|jdS)Nrcg:?dtype)rerftoro ones_likererrrerf_gelus4rscCsBtdd|D}tjj|td|tjjtd}|S)z*Reduce a tensor of losses across all GPUs.cSsg|] }|dqS)r.)clonedetachview).0lossrrr z=average_losses_across_data_parallel_group..)group)rcat distributed all_reducerget_data_parallel_groupget_world_size)lossesaveraged_lossesrrr)average_losses_across_data_parallel_groups rc Csf|\}}|r |}nd}d} |r%ttj|||f|jd|d||} tj|tj|jd} |r9d| ||k<tj|tj|jd} | d |d} |rR| } |sV|rt |D]M} | | || |kf} |rl| } d}t | dD]0}| |}|rd| | d|ddd|df<|r| | |ddf|d|8<|d}qvqZ|r| dk} | | | fS)z4Build masks and position id for left to right model.r.Ndevicerorr;rrc) sizertrilonesrrvfloatarangelong unsqueezerepeatrtrange)data eod_tokenreset_position_idsreset_attention_mask eod_mask_losscompute_attention_maskmicro_batch_size seq_lengthatt_mask_batchrY loss_mask position_idsb eod_index prev_indexjirrrget_ltor_masks_and_position_idssD    $  rcCs|durdS|d}|S)Nr.)r) attn_maskextended_attention_maskrrrattn_mask_postprocess s rcCsdd|DS)NcSg|]}t|qSr)r)rwrrrrryz3enc_dec_extended_attention_mask..r)attention_mask_listrrrenc_dec_extended_attention_masksrcCs6|d}tj|tj|jd}|d|}|S)Nr.rr)rrrrrr expand_asrt) token_idsrrrrrbuild_position_idss rcCs0|dddddf|dddddf}|S)| Returns a 3-dimensional (3-D) attention mask :param source_block: 2-D array :param target_block: 2-D array Nr source_mask target_maskmaskrrrmake_attention_mask_3d"s,rcCst||k||kS)rr) source_block target_blockpad_idrrr make_inference_attention_mask_3d,srcCsH|j\}}tj||jd}|d|dddfkd}||||}|S)Nrr)shaperrrexpand)blockbatchlengthr history_maskrrrmake_inference_history_mask_3d6s rcCst||}|dkSz Returns a 3D joint attention mask for Megatron given two 2D masks :param source_mask - True for non-masked, else masked [batch, src length] :param target_mask - True for non-masked, else masked [batch, tgt length] rcrrrrrbuild_attention_mask_3d_padding>s rcCs"t|}t||}||}|dkSr)rr)rr causal_maskrrrrbuild_attention_mask_3d_causalIs rcCs>|tjkr t||}|S|tjkrt||}|Std|)a Returns a 3D attention mask for Megatron given two 2D masks :param source_mask - < 0.5 for non-masked, else masked [batch, src length] :param target_mask - < 0.5 for non-masked, else masked [batch, tgt length] :param attn_mask_type - AttnMaskType enum z,Unsupported attention mask attn_mask_type = )r paddingrcausalr ValueError)rrattn_mask_typerrrrbuild_attention_mask_3dVs   rmodelreturnc Cst|}gdd}gdd}gddd}dd}|D]U}|D]N}t|tttfr<|dttd d|j q |j D],\}} | d urJqA| d rX|d| gqA|| re|d| gqA|d| gqAq q|||g} t td d| S) zDivide params into with-weight-decay and without-weight-decay groups. Layernorms and biases will have no weight decay but the rest will. Fparams is_expertTr;)r weight_decayrcSt|dd SN allreduceTgetattrparamrrrsz:get_params_for_weight_decay_optimization..rcSs|duSrr)prrrrysNr*cSt|ddkSNrrlengrrrr)rmodules isinstancer rr extendlistfilter _parametersvaluesitemsendswithtuple) rrweight_decay_paramsweight_decay_expert_paramsno_weight_decay_paramsrmodulemodule_namer param_groupsrrr(get_params_for_weight_decay_optimizationgs.      rc st|}gdd}gdd}dd|D]$}|dttfdd|7<|dtt|7<q||g}ttdd|S) z Use all params for weight decay.FrTcSrrrrrrrrrz>get_all_params_for_weight_decay_optimization..rcs | Srrrerrrrs cSrrrrrrrrr)rrr parametersr)rrrrrrrrr,get_all_params_for_weight_decay_optimizations  & renforce_divisible_batchcsHt||rtdksJdfddtdtDS)z. Split a list into equal sized chunks r$Issue with batch size configuration!csg|] }||qSrrrwr chunk_sizeinputsrrryzsplit_list..)rr)r num_chunksrrrr split_lists  rrnum_microbatchesc st|trdd|D}t|dkrtjd|tjddd|D}dd|D}d d|D}t|radd j ddkrat d dd j dd d fddDdd j ddkrddj dfddDt|dkrfddt D}n.t|}fdd|D}ddDdd|D|fddt D}dd|D}n|dj ddksJdg|D]l}t |rt j|ddqt|tr>t|dt jr5fdd|Dg} t D]| fddt tDqt| } | qt|q|durI|qt dt|fddt D}t|S)a Split a batch into k microbatches, where the batch size is divisible by k. Batch could be a dictionary of tensors or a list of tensors. A dictionary batch could also have items of List type, as long as the length of that list is the same as the batch size. cSs$g|]\}}t|tjtfs|qSrrrrrrwkvrrrry$z(get_iterator_k_split..rzjOnly support splitting torch.Tensor and List[torch.Tensor]. Discarding the following keys from the batch: )modecSs&i|]\}}t|tjtfr||qSrrrrrr s&z(get_iterator_k_split..cSs"i|] \}}t|tjr||qSr)rrrrrrrrs"cSs i|] \}}t|tr||qSr)rrrrrrr r.z0Issue with batch size configuration: batch size z is not divisible by !cs g|] }tj|dddqS)r.rdimr tensor_splitrwr rrrryrcsg|] }fdd|DqS)csg|]}|dqSrr)rwrrrrry3get_iterator_k_split...rrr rrryrzcs*g|]fddttDqS)cs$g|]}|d|fqSrrr)rr split_batchrrryrrrrrw)rrrrrycsg|] }t|ddqS)r.)r)rr )rrrrryscSsg|]}|dqSrrr rrrryrcs*g|]fddttDqS)cs g|] }||fqSrrr)all_keysall_split_batchrrrryrrrr)rrrrryrcSrr)dictrwelemrrrryrrrcsg|] }tj|ddqS)rrrrr rrryrcsg|]}|qSrrr)mbi split_tensorsrrryr NzUnsupported item type: csg|] fddDqS)cs g|] }|dur |n|qSrrrrrrryrrrr)rrrrys)rrrrr warningr ONCErrrrr is_tensorappendrrrrtype itertoolschain) rrr discard_items tensor_items list_items microbatchessplit_list_batchr split_tupler) rrrrrrrrrrget_iterator_k_splitsn           &     r*cCsVtr)t|tjr)|jjdkrt}n|jjdkr t}nt|j |dS|S)Ncudacpurn) ris_autocast_enabledrrrr!get_autocast_gpu_dtypeget_autocast_cpu_dtypeNotImplementedErrorrq)rCrorrr_cast_if_autocast_enableds      r1)NFFF)T)Jr%r"rStypingrrrrrrrr nemo.utilsr r apex.normalizationr #apex.normalization.fused_layer_normr apex.transformer.enumsr "apex.transformer.layers.layer_normr3apex.transformer.pipeline_parallel.schedules.commonr HAVE_APEX ImportErrorModuleNotFoundError megatron.corerr$megatron.core.tensor_parallel.layersrHAVE_MEGATRON_CORErobjectrboolr7rHrNrQrVrZrbjitscriptrgrhcompile jit_fuserrmrsrrrrrrrrrrrr?Modulestr Parameterrrrintr*r1rrrrs         5        =       $   S