o TiC@sddlZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z m Z mZddlmZddlmZdd lmZd d lmZdd lmZdd lmZGddde ZGdddeZGdddejZGdddejZ dS)N)Function)DeepSpeedSelfAttention)DeepSpeedInferenceConfig) SoftmaxOpVectorMatMulOp GELUGemmOp)BiasResidualOp)EinsumSecSmEcmOp) LayerNormOp)TopKGate)comm)MoEResMatmulOpcsreZdZdZ                          dfd d Zed dZeddZZS)DeepSpeedMoEInferenceConfigaInitialize the DeepSpeed Transformer Config. Arguments: hidden_size: The hidden size of the transformer layer intermediate_size: The intermediate size of the feed-forward part of transformer layer heads: The number of heads in the self-attention of the transformer layer num_hidden_layers: The number of transformer layers layer_norm_eps: The epsilon value for the layer norm local_rank: Optional: The rank of GPU running the transformer kernel, it is not required to use if the model already set the current device, otherwise need to set it so that the transformer kernel can work on the right device mp_size (optional): This argument is mainly used to create the parameters on the kernel side using model-parallel architecture. If the client model already takes care of this, there is no need to pass this argument. fp16: Enable half-precision computation bf16: Enable bf16 floating point computation pre_layer_norm: Select between Pre-LN or Post-LN transformer architecture stochastic_mode: Enable for high performance, please note that this flag has some level of non-determinism and can produce different results on different runs. However, we have seen that by enabling it, the pretraining tasks such as BERT are not affected and can obtain a high accuracy level. On the other hand, for the downstream tasks, such as fine-tuning, we recommend to turn it off in order to be able to reproduce the same result through the regular kernel execution. scale_attention: If true, both q and k are scaled by 1/sqrt(attention_heads) before attention computation. return_tuple: if True, returns the transformer output as a tuple, otherwise returns as a tensor -q=rFT?Nstandardcstt|||dkr |nd|||||||| | | | | ||||||_||_||_||_||_||_||_ ||_ ||_ ||_ ||_ dS)Nrr )superr__init__ moe_expertskcapacity_factoreval_capacity_factor min_capacitynoisy_gate_policy drop_tokensuse_rtsglobal_expertsmlp_typescale_attn_by_inverse_layer_idx)self hidden_sizeintermediate_sizeheadsnum_hidden_layerslayer_norm_eps local_rankmp_sizefp16bf16q_int8pre_layer_normstochastic_modescale_attentiontriangular_maskinglocal_attention window_size return_tuplerr rrrrrrrr!r" __class__e/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/ops/transformer/inference/moe_inference.pyr1s&  z$DeepSpeedMoEInferenceConfig.__init__cCs&t}|D] \}}||j|<q|SN)ritems__dict__)cls json_objectconfigkeyvaluer7r7r8 from_dict_s z%DeepSpeedMoEInferenceConfig.from_dictcCsFt|ddd }|}Wdn1swY|t|S)Nrzutf-8)encoding)openreadrAjsonloads)r< json_filereadertextr7r7r8from_json_filefs z*DeepSpeedMoEInferenceConfig.from_json_file)rrrrrrrFFFTFTTFrTrrrrrrNTFrF) __name__ __module__ __qualname____doc__r classmethodrArK __classcell__r7r7r5r8rsF. rc@s$eZdZeddZeddZdS)DeepSpeedMLPFunctionc Cs|jr | ||||j|d|d| |j}| |||d|| }n | |||||j|j| }| dur@tj| ddkr@tj|| | d||S)Ngroupr)rVasync_op)r-epsilonr.distget_world_size all_reduce)ctxinputinter_winter_br>output_boutput_wq_scalesq_groups merge_countmp_grouprWgelu_gemm_funcvector_matmul_func intermediateoutputr7r7r8forwardoszDeepSpeedMLPFunction.forwardcCstd)NzYou are running with DeepSpeed Inference mode. Please switch to Training mode for running backward!) RuntimeError)r\ grad_outputr7r7r8backward}szDeepSpeedMLPFunction.backwardN)rLrMrN staticmethodrjrmr7r7r7r8rRms   rRcs(eZdZdfdd Zd ddZZS) DeepSpeedMoEMLPNrFcstt|||_tt|jj|_ tt|jj|_ |jj |dur)dnt j |d}tt|jj||_tt||_tt||jj|_tt|jj|_||_|re|dn||_tt||_||_t|j|_t|j|_dS)NrrUrS)rrorr>nn ParametertorchTensorr$attn_nwattn_nbr%rYrZr^r_rar`rbrcintmathlog2rdrerrfrrg)r#r>rbrcrdmlp_extra_groupingre interm_sizer5r7r8rs  zDeepSpeedMoEMLP.__init__cCs8t||j|j|j|j|j|j|j|j |j ||j |j Sr9) rRapplyr^r_r>r`rarbrcrdrerfrg)r#r]rWr7r7r8rjszDeepSpeedMoEMLP.forward)NrrFN)F)rLrMrNrrjrQr7r7r5r8rosrocs~eZdZdZdZ       dfdd Zdd Zd d Zd d ZddZ ddZ            dddZ Z S)DeepSpeedMoEInferencea@Initialize the DeepSpeed MoE Transformer Layer. Arguments: layer_id: The layer index starting from 0, e.g. if model has 24 transformer layers, layer_id will be 0,1,2...23 when each layer object is instantiated config: An object of DeepSpeedInferenceConfig mp_group: Model parallelism group initialized on the modeling side. quantize_scales: This argument groups all the layers' scales used for quantization quantize_groups: Number of groups used for quantizing the model merge_count: Shows the number of model-parallel checkpoints merged before running inference. We use this argument to control the quantization scale for the model parameters if a bigger quantize-grouping than 1 is used. mlp_extra_grouping: This flag is used to show a 2x higher number of groups used for the MLP part of a Transformer layer. We use this feature for quantization to reduce the convergence impact for specific downstream tasks. rNrFc stt||_tj|j_|jjtjksJdtjd7_t|j||_ t t |jj |_t t |jj |_t t |jj |_t t |jj |_jdkr~t||_t t |jj d|_t|j|_t|j|_d_t fddt|jjD|_t|jj |jj |jj!|jj"|jj#|jj$|jj%|jj&|jj'|j( |_)||_(||_*|_+t,d|jj-t.|j|_/t0|j|_1t2|j|_3t4|j|_5dS)NzEDeepSpeed MoE Transformer Inference not yet tested for bfloat supportrresidualrSc3s"|] }tVqdSr9)ro).0ir>expert_mp_grouprdryquantize_groupsquantize_scalesr7r8 s  z1DeepSpeedMoEInference.__init__..z.DeepSpeed MoE Transformer Inference config is )6rr|rr>layer_iddtyperrbfloat16r attentionrprqrsr$rtrunorm_wnorm_br!rores_mlpres_coefr coef_funcrrgr* ModuleListrangermlpr r rrrrrrrep_groupmoe_gatererprintr;r bias_residual_funcr ds_layernormr einsum_sec_sm_ecmrmoe_res_matmul) r#r>rerrrrrdryr5rr8rsD             zDeepSpeedMoEInference.__init__c Cs,|||j|}||tddddd|S)NrFr)rgrrrrempty)r#inprWr7r7r8 res_coef_funcsz#DeepSpeedMoEInference.res_coef_funccCsF||d|jjd\}}}}||||d|jj}||fS)Nr)rviewr>r$rtype_as)r#attention_output_combined_weights dispatch_maskdispatched_attentionr7r7r8moe_gate_einsums z%DeepSpeedMoEInference.moe_gate_einsumcCs||jj|jj|jjd|jj}|j|jjdd}tj|jj|djdf|djdd|j |j d}t |t t |jD]\}}|j||d|jd|jd||<qA|S)NrrdimrrSrdevice)reshaper>r rr$chunkrrrshaperrziprlenrr)r#dispatched_inputchunksexpert_outputsrexpertr7r7r8 expert_execs$  z!DeepSpeedMoEInference.expert_execcCs6tj|jddkrt|}tj|||jd|S|S)NrUr)rYrZrrr empty_likeall_to_all_single)r#rrr7r7r8 _alltoalls  zDeepSpeedMoEInference._alltoallcCs:t|||jdd|d|jd}||jS)Nrr)rrmatmulrrr)r#r expert_outputrcombined_outputr7r7r8scale_expert_output s  z)DeepSpeedMoEInference.scale_expert_outputc Cs|p|p| }|dur |n|}|j}|jjtjtjfvr$|tjkr$|}t||||||| | | |j |j }|rI|dd\}}}||f}n| rV|dd\}}}}n|d}||jj }| ||j |j|jj}|jjdkr|j|dd}|j|dd}|jdurtj|jd}tj|||j|jd}tj|||jd|jd g|d dR}||\}}||}||}||}||||}|jdur|j |j!dtj|jddd tj"|jd}|jjdkr|#||||$||td }|jj%s| ||j |j |jj}||jkr|&|}Wdn 1s*wY|r6||f}|jj'rGt(|t)urD|S|fS|S) NrrTr r}T)rWrUrrrr)*rr>rrfloat16int8floathalfno_gradrrrattn_obrrtrurXr!rrrrYrZrnumelrall_gather_into_tensorrsizerrrrsplitrget_rankrrr.tor4typetuple) r#r] input_maskattention_mask head_mask layer_past get_key_value get_presentencoder_outputenc_dec_attn_maskencoder_hidden_statesencoder_attention_mask use_cacheoutput_attentions input_typerp_keyp_valuepresentsrcontext_output residual_add res_mlp_out res_coef_out world_size gather_bufferrrrrrrir7r7r8rjsl               3 zDeepSpeedMoEInference.forward)NNNNrrF) NNNNFFNNNNFF) rLrMrNrOrrrrrrrrjrQr7r7r5r8r|s:4 r|)!rFrwrrtorch.autogradrtorch.nnrp ds_attentionrr>r op_bindingrrrop_binding.bias_residualr op_binding.einsum_sec_sm_ecmr op_binding.layer_normr moe.sharded_moer deepspeedrrYop_binding.moe_res_matmulrrrRModuleror|r7r7r7r8s$          W