o Ti*@s"ddlmZddlmZddlmZddlmZmZm Z m Z m Z m Z ddl mZmZmZmZmZmZmZmZmZmZmZmZmZmZde ded efd d Zd eded efd dZdeded efddZde ded efddZde ded efddZ de ded efddZ!de ded efddZ"dS))RaggedInferenceEngineConfig) NormTypeEnum) ConfigBundle)DSEmbeddingsConfigDSLinearConfig DSMoEConfig DSNormConfigDSSelfAttentionConfigDSUnembedConfig)DSEmbeddingBaseDSEmbeddingRegistry DSLinearBaseDSLinearRegistry DSMoEBase DSMoERegistryDSPostNormBaseDSPostNormRegistry DSPreNormBaseDSPreNormRegistryDSSelfAttentionBaseDSSelfAttentionRegistry DSUnembedBaseDSUnembedRegistryattention_config engine_configreturncCtd|d}t|S)a Choose an appropriate attention implementation based on the given configurations. This method is currently a stub, but as more implementations may be developed we can centralize the logic for choosing between them here. Arguments: attention_config (DSSelfAttentionConfig): Configuration for the attention module. engine_config (RaggedInferenceEngineConfig): Configuration for the inference engine. Returns: An attention module implementing the given configuration. dense_blocked_attentionnameconfig)rrinstantiate_config)rrr!r#]/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/modules/heuristics.pyinstantiate_attention$s  r% embed_configcCr)a Choose an appropriate embedding implementation based on the given configurations. This method is currently a stub, but as more implementations may be developed we can centralize the logic for choosing between them here. Arguments: embed_config (DSEmbeddingsConfig): Configuration for the embedding module. engine_config (RaggedInferenceEngineConfig): Configuration for the inference engine. Returns: An embedding module implementing the given configuration. ragged_embeddingr)rr r")r&rr!r#r#r$instantiate_embed8  r( linear_configcCs|jj}|durtd|d}n>|dkrFddl}|js tdt|jdo+|jj du}|r2td|j dj d kr?td td |d}ntd |t |S) a Choose an appropriate linear implementation based on the given configurations. This method is currently a stub, but as more implementations may be developed we can centralize the logic for choosing between them here. Arguments: linear_config (DSLinearConfig): Configuration for the linear module. engine_config (RaggedInferenceEngineConfig): Configuration for the inference engine. Returns: A linear module implementing the given configuration. Nblas_fp_linearrwf6af16z.WF6AF16 quantization is only supported on CUDAhipz5WF6AF16 quantization is only supported on NVIDIA GPUsz>WF6AF16 quantization is only supported on Ampere architecturesquantized_wf6af16_linearzUnsupported quantization mode: ) quantizationquantization_modertorchcuda is_available ValueErrorhasattrversionr.get_device_propertiesmajorrr")r*rr2r!r3is_rocm_pytorchr#r#r$instantiate_linearKs  r< moe_configcCs.d}|dkr d|ji}td||d}t|S)a Choose an appropriate MoE implementation based on the given configurations. This method is currently a stub, but as more implementations may be developed we can centralize the logic for choosing between them here. Arguments: moe_config (DSMoEConfig): Configuration for the MoE module. engine_config (RaggedInferenceEngineConfig): Configuration for the inference engine. Returns: A MoE module implementing the given configuration. cutlass_multi_gemm_moe weight_dtype)r r!implementation_config) input_dtyperrr")r=rmoe_typer@r!r#r#r$instantiate_moens rC norm_configcCr)a Choose an appropriate post-norm implementation based on the given configurations. This method is currently a stub, but as more implementations may be developed we can centralize the logic for choosing between them here. Arguments: norm_config (DSNormConfig): Configuration for the post-norm module. engine_config (RaggedInferenceEngineConfig): Configuration for the inference engine. Returns: A post-norm module implementing the given configuration. cuda_post_lnr)rrr")rDrr!r#r#r$instantiate_post_normr)rFcCs@t|jtjkr d}n t|jtjkrd}t||d}t|S)a Choose an appropriate pre-norm implementation based on the given configurations. Currently, this will select between two CUDA implementations, one for LayerNorm and one for RMSNorm. Arguments: norm_config (DSNormConfig): Configuration for the pre-norm module. engine_config (RaggedInferenceEngineConfig): Configuration for the inference engine. Returns: A pre-norm module implementing the given configuration. cuda_pre_ln cuda_pre_rmsr)rtype LayerNormRMSNormrrr")rDr module_namer!r#r#r$instantiate_pre_norms    rMunembed_configcCr)a Choose an appropriate unembedding implementation based on the given configurations. This method is currently a stub, but as more implementations may be developed we can centralize the logic for choosing between them here. Arguments: unembed_config (DSUnembedConfig): Configuration for the unembed module. engine_config (RaggedInferenceEngineConfig): Configuration for the inference engine. Returns: An unembed module implementing the given configuration. ragged_unembedr)rrr")rNrr!r#r#r$instantiate_unembedr)rPN)# config_v2rinference_utilsrmodule_registryrmodules.configsrrrr r r modules.interfacesr r rrrrrrrrrrrrr%r(r<rCrFrMrPr#r#r#r$s    @ #