o Ti#@sddlmZmZmZddlZddlmZddlm Z ddl m Z ddl m Z mZddlTddlTddlTddlmZd d lmZmZd d lmZmZGd ddeZdS))IterableOptionalTupleN) empty_from)RaggedInferenceEngineConfig)ActivationType DtypeEnum)*)RaggedBatchWrapper)DSModelImplementationConfigMPType)MixtralNonTransformerContainerMixtralTransformerContainerc seZdZUdZeeed< eeeed< e de fddZ e de fddZ e de fd d Z e de fd d Ze de fd dZe de fddZe de fddZe de fddZe defddZe defddZe defddZe defddZe deefddZ e de fdd Ze de fd!d"Ze defd#d$Z d%e!d&e"d'e#dd(ffd)d* Z$d+e%de&j'fd,d-Z(d.e d/e&j'd0e&j'd1e%de)e&j'e&j'ff d2d3Z*d0e&j'd1e%de&j'fd4d5Z+d6e%de&j'fd7d8Z,Z-S)9MixtralInferenceModelz< Inference model implementation for Mixtral models. _non_transformer _transformerreturncC|jjSN)_configmax_position_embeddingsselfrn/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/model_implementations/mixtral/model.pymax_sequence_length-z)MixtralInferenceModel.max_sequence_lengthcCrr)rnum_hidden_layersrrrr num_layers5rz MixtralInferenceModel.num_layerscCrr)r hidden_sizerrrr model_dim9rzMixtralInferenceModel.model_dimcCrr)r vocab_sizerrrrr$=rz MixtralInferenceModel.vocab_sizecCs |j|jSr)r#n_headsrrrr head_sizeAs zMixtralInferenceModel.head_sizecCrr)rnum_attention_headsrrrrr%ErzMixtralInferenceModel.n_headscCrr)rintermediate_sizerrrrintermediate_dimIrz&MixtralInferenceModel.intermediate_dimcCrr)rnum_key_value_headsrrrr n_heads_kvMrz MixtralInferenceModel.n_heads_kvcCs0|jjtjkr tjS|jjtjkrtjStd)Nz Only fp16 and bf16 are supported) r torch_dtypetorchfloat16r fp16bfloat16bf16NotImplementedErrorrrrractivation_dtypeQs z&MixtralInferenceModel.activation_dtypecCsT|jj}|dkr tjS|dkrtjS|dkrtjS|dkr"tjStd|d)Ngelurelugegelusiluz Activation z not supported)r hidden_actlowerrGEGLUReGLUSiGLUr2)r activationrrrmlp_activation_fnZs z'MixtralInferenceModel.mlp_activation_fncCtjSr) NormTypeEnumRMSNormrrrr norm_typehzMixtralInferenceModel.norm_typecCr?r)PositionalEmbeddingType rotate_halfrrrrpositional_embedding_typelrCz/MixtralInferenceModel.positional_embedding_typecCst|jjdS)zG The positional embedding configuration for the model. ) theta_base)RotateHalfConfigr rope_thetarrrrpositional_embedding_configpsz1MixtralInferenceModel.positional_embedding_configcCrr)rnum_local_expertsrrrr n_experts{rzMixtralInferenceModel.n_expertscCrr)rnum_experts_per_tokrrrrn_top_krzMixtralInferenceModel.n_top_kcCsdS)NTrrrrrnormalize_expert_scoressz-MixtralInferenceModel.normalize_expert_scoresconfig engine_config base_mp_groupNcsRt||||||||||d|_ dS)a  Base implementation for initialization. By default, this will initialize the traditional components of a transformer model: - Embedding - QKV projection - Self attention - Attention output projection - Feed forward network - Normalization - Unembedding Arguments: config (DSModelImplementationConfig): Model-specific configuration. No assumptions should be made about this config that are not closely tied to the specific model implementation. engine_config (RaggedInferenceEngineConfig): Engine configuration. base_mp_group (MPType): Base communication group for Tensor-parallel inference. N) super__init__make_norm_layermake_qkv_layermake_attn_layermake_attn_out_layermake_moe_layermake_embedding_layermake_unembedding_layer_kv_cache_config)rrPrQrR __class__rrrTs zMixtralInferenceModel.__init__ ragged_batchcCs<|||jj}|jd|jkrtd|jd|j|S)z Performs the embedding lookup prior to running the transformer of the model. Arguments: ragged_batch (RaggedBatchWrapper): The batch to embed. Returns: torch.Tensor: The embedded batch. zEmbedding output shape z does not match model_dim )embedrword_embshaper# ValueError)rr_rarrr_forward_embeds z$MixtralInferenceModel._forward_embed layer_idxresidual hidden_statesragged_batch_infocCs|j|}|j|}|||j}||||}|||j}|jdkr-t j ||j d| |||j \}}||||j|j|j}|jdkrPt j ||j d||jdkrl|j|d}| |||j\}}||fS||||fS)aL Executes one (slightly offset) layer of the transformer. This implementation does a peak-ahead optimization to fuse the layer norm of the next layer into the current layer. Arguments: layer_idx (int): The index of the layer to execute. residual (torch.Tensor): The residual tensor from the previous layer. hidden_states (torch.Tensor): The hidden states from the previous layer. This is the hidden states after pre normalization. ragged_batch_info (RaggedBatchWrapper): The batch metadata. rgroup)r state_manager get_cacheqkvqkv_wattnattn_out attn_out_wtp_sizedist all_reduce_base_mp_groupnormmlp_norm_gammamoemoe_gate moe_mlp_1 moe_mlp_2r!attn_norm_gammaadd_)rrfrgrhri cur_paramskv_cache next_paramsrrr_forward_transformers&     z*MixtralInferenceModel._forward_transformercCs|j||jj||jjd}|jdkrKt|j|j|jd|jdf}t|j|jd|j f}t j |||j d| |ddd|jd|j |S|S)z Performs unembedding of the hidden states to logits. This will only sample the final token of each sequence. )gammarrrjr )unembedr word_unembed final_normrsr _comm_logitsrc_return_logitsr$rtall_gather_into_tensorrvcopy_permutereshape)rrhrilogits comm_buffer full_logitsrrr_forward_unembeds  $z&MixtralInferenceModel._forward_unembed wrapped_batchcCsX||}|j|d|jdjdd\}}t|jD] }|||||\}}q|||S)Nr)beta)rerwrr}ranger!rr)rrrgrhrfrrrforwards  zMixtralInferenceModel.forward).__name__ __module__ __qualname____doc__rr__annotations__rrpropertyintrr!r#r$r&r%r)r+r r3rr>r@rBrDrFrHrJrLrNboolrOr rrrTr r-Tensorrerrrr __classcell__rrr]rrsn     +r)typingrrrr-deepspeed.commcommrt allocatorr config_v2rinference_utilsrr model_implementationsmodules.configsmodules.interfacesraggedr inference_model_baser r containerrrDSMoETransformerModelBaserrrrrs