o Ti!@s6ddlZddlmZddlmZGdddeeZdS)N)HybridEngineContainer)MegatronContainerc@s>eZdZdejfddZd ddZdejfdd Zd d ZdS) HybridMegatronContainerxc Cs|jd|j}|j|f|dd}|j|}t|jdkr)t|dnd}tj||j|d|d\}}}t|jdkrk|jtj | d|jd| d|jd| d|jdfdd |jdS|jtj | d| d| dfdd |jdS)z Internal helper for accepting the head-contiguous weight matrix and chunking the query, key, and value components. rrNdim) shapenum_attention_headssizeviewlentorchsplitdatacopy_catreshape) selfrattention_head_size new_x_shapex_1div_dimqkvro/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/module_inject/containers/features/hybrid_megatron.py _align_qkvs " 6:z"HybridMegatronContainer._align_qkvreturnNct|jdrDddlm}ddlm|j|jg}fdd|D}||||j||jWddS1s=wYdS||j||jdS)a Overrides the HybridEngineContainer implementation. The alternative layout of the QKV matrix for Megatron is such that each head's Q, K, and V are sequential in memory. This is different from the default layout in which all of the Qs are sequential, followed by all of the Ks, and then all of the Vs. Here, we take the default layout and transform it to the inference layout. ds_idrGatheredParametersZeroParamStatusc&g|]}t|dr|jjkr|qSr$hasattr ds_status NOT_AVAILABLE.0paramr'rr , zCHybridMegatronContainer.transform_for_inference..N)r,qkvwdeepspeed.runtime.zeror&+deepspeed.runtime.zero.partition_parametersr(qkvbr!rr& param_listnon_active_paramsrr'r transform_for_inference     " z/HybridMegatronContainer.transform_for_inferencecstj||jdddd}|djd|j}|j|f|ddfdd|D\}}}t|jdkrN|jtj|||fd d d |jd dS|jtj|||fd d d dS) zm Internal helper for taking contiguous QKV and partitioning it for contiguous heads. rr r rNcsg|]}|jqSr)r)r0rrrr r2=sz:HybridMegatronContainer._partition_qkv..rr) rrr r rrrrrr)rrq_k_vrrrrrr=r _partition_qkv5s.&z&HybridMegatronContainer._partition_qkvcr#)a Overrides the HybridEngineContainer implementation. The alternative layout of the QKV matrix for Megatron is such that each head's Q, K, and V are sequential in memory. This is different from the default layout in which all of the Qs are sequential, followed by all of the Ks, and then all of the Vs. This function takes the inference format and reverts it back to the default format. r$rr%r'cr)r*r+r/r'rr r2Pr3zBHybridMegatronContainer.transform_for_training..N)r,r4r5r&r6r(r7r@r8rr'r transform_for_trainingCr<z.HybridMegatronContainer.transform_for_training)r"N) __name__ __module__ __qualname__rTensorr!r;r@rArrrr r s   r)r hybrid_enginermegatronrrrrrr s