o Ti@sddlmZmZmZddlZddlmZddlm Z ddl m Z m Z ddl TddlTddlTddlmZdd lmZmZGd d d eZdS) )IterableOptionalTupleN) empty_from)ActivationType DtypeEnum)*)RaggedBatchWrapper)MistralNonTransformerContainerMistralTransformerContainerc @seZdZUdZeeed< eeeed< e de fddZ e de fddZ e de fd d Z e de fd d Ze de fd dZe de fddZe de fddZe de fddZe defddZe defddZe defddZe defddZe deefddZ dedejfd d!Z d"e d#ejd$ejd%ede!ejejff d&d'Z"d$ejd%edejfd(d)Z#d*edejfd+d,Z$d-S).MistralInferenceModelzP Inference model implementation for ragged batching for Mistral models. _non_transformer _transformerreturncC|jjSN)_configmax_seq_lengthselfrn/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/model_implementations/mistral/model.pymax_sequence_length(z)MistralInferenceModel.max_sequence_lengthcCrr)rnum_hidden_layersrrrr num_layers0rz MistralInferenceModel.num_layerscCrr)r hidden_sizerrrr model_dim4rzMistralInferenceModel.model_dimcCrr)r vocab_sizerrrrr 8rz MistralInferenceModel.vocab_sizecCs |j|jSr)rn_headsrrrr head_size<s zMistralInferenceModel.head_sizecCrr)rnum_attention_headsrrrrr!@rzMistralInferenceModel.n_headscCrr)rintermediate_sizerrrrintermediate_dimDrz&MistralInferenceModel.intermediate_dimcCrr)rnum_key_value_headsrrrr n_heads_kvHrz MistralInferenceModel.n_heads_kvcCs0|jjtjkr tjS|jjtjkrtjStd)Nz Only fp16 and bf16 are supported) r torch_dtypetorchfloat16rfp16bfloat16bf16NotImplementedErrorrrrractivation_dtypeLs z&MistralInferenceModel.activation_dtypecCsT|jj}|dkr tjS|dkrtjS|dkrtjS|dkr"tjStd|d)Ngelurelugegelusiluz Activation z not supported)r hidden_actlowerrGEGLUReGLUSiGLUr.)r activationrrrmlp_activation_fnUs z'MistralInferenceModel.mlp_activation_fncCtjSr) NormTypeEnumRMSNormrrrr norm_typeczMistralInferenceModel.norm_typecCr;r)PositionalEmbeddingType rotate_halfrrrrpositional_embedding_typegr?z/MistralInferenceModel.positional_embedding_typecCst|jjdS)N) theta_base)RotateHalfConfigr rope_thetarrrrpositional_embedding_configksz1MistralInferenceModel.positional_embedding_config ragged_batchcCs<|||jj}|jd|jkrtd|jd|j|S)z Performs the embedding lookup prior to running the transformer of the model. Arguments: ragged_batch (RaggedBatchWrapper): The batch to embed. Returns: torch.Tensor: The embedded batch. zEmbedding output shape z does not match model_dim )embedrword_embshaper ValueError)rrGrIrrr_forward_embedss z$MistralInferenceModel._forward_embed layer_idxresidual hidden_statesragged_batch_infocCs|j|}|j|}|j||jdd}||||}|j||jdd}|jdkr1t j ||j d|j |||j dd\}}|j||jdd}|j||jdd}|jdkr\t j ||j d||jdkrz|j|d}|j |||jdd\}}||fS||||fS)aL Executes one (slightly offset) layer of the transformer. This implementation does a peak-ahead optimization to fuse the layer norm of the next layer into the current layer. Arguments: layer_idx (int): The index of the layer to execute. residual (torch.Tensor): The residual tensor from the previous layer. hidden_states (torch.Tensor): The hidden states from the previous layer. This is the hidden states after pre normalization. ragged_batch_info (RaggedBatchWrapper): The batch metadata. N)br groupbeta)r state_manager get_cacheqkvqkv_wattnattn_out attn_out_wtp_sizedist all_reduce_base_mp_groupnormmlp_norm_gammamlp_1mlp_1_wmlp_2mlp_2_wrattn_norm_gammaadd_)rrNrOrPrQ cur_paramskv_cache next_paramsrrr_forward_transformers$     z*MistralInferenceModel._forward_transformercCs|j||jj||jjd}|jdkrKt|j|j|jd|jdf}t|j|jd|j f}t j |||j d| |ddd|jd|j |S|S)z Performs unembedding of the hidden states to logits. This will only sample the final token of each sequence. )gammar rrS)unembedr word_unembed final_normr^r _comm_logitsrK_return_logitsr r_all_gather_into_tensorracopy_permutereshape)rrPrQlogits comm_buffer full_logitsrrr_forward_unembeds  $z&MistralInferenceModel._forward_unembed wrapped_batchcCsX||}|j|d|jdjdd\}}t|jD] }|||||\}}q|||S)NrrU)rMrbrrhrangerrmr|)rr}rOrPrNrrrforwards  zMistralInferenceModel.forwardN)%__name__ __module__ __qualname____doc__rr __annotations__rr propertyintrrrr r"r!r%r'rr/rr:r<r>r@rBrDrFr r)TensorrMrrmr|rrrrrrsV    ,r)typingrrrr)deepspeed.commcommr_ allocatorrinference_utilsrrmodel_implementationsmodules.configsmodules.interfacesraggedr containerr r DSTransformerModelBaserrrrrs