o Ti@sddlmZmZmZddlZddlmZddlm Z ddl m Z m Z ddl TddlTddlTddlmZd d lmZmZGd d d eZdS) )IterableOptionalTupleN) empty_from)ActivationType DtypeEnum)*)RaggedBatchWrapper)Phi3NonTransformerContainerPhi3TransformerContainerc @seZdZUdZeeed< eeeed< e de fddZ e de fddZ e de fd d Z e de fd d Ze de fd dZe de fddZe de fddZe de fddZe defddZe defddZe defddZe defddZe deefddZ dedejfd d!Z d"e d#ejd$ejd%ede!ejejff d&d'Z"d$ejd%edejfd(d)Z#d*edejfd+d,Z$d-S).Phi3InferenceModelzP Inference model implementation for ragged batching for Llama-2 models. _non_transformer _transformerreturncC|jjSN)_configmax_seq_lengthselfrk/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/model_implementations/phi3/model.pymax_sequence_length(z&Phi3InferenceModel.max_sequence_lengthcCrr)rnum_hidden_layersrrrr num_layers0rzPhi3InferenceModel.num_layerscCrr)r hidden_sizerrrr model_dim4rzPhi3InferenceModel.model_dimcCrr)r vocab_sizerrrrr!8rzPhi3InferenceModel.vocab_sizecCs |j|jSr)r n_headsrrrr head_size<s zPhi3InferenceModel.head_sizecCrr)rnum_attention_headsrrrrr"@rzPhi3InferenceModel.n_headscCrr)rintermediate_sizerrrrintermediate_dimDrz#Phi3InferenceModel.intermediate_dimcCrr)rnum_key_value_headsrrrr n_heads_kvHrzPhi3InferenceModel.n_heads_kvcCs0|jjtjkr tjS|jjtjkrtjStd)Nz Only fp16 and bf16 are supported) r torch_dtypetorchfloat16rfp16bfloat16bf16NotImplementedErrorrrrractivation_dtypeLs z#Phi3InferenceModel.activation_dtypecCsT|jj}|dkr tjS|dkrtjS|dkrtjS|dkr"tjStd|d)Ngelurelugegelusiluz Activation z not supported)r hidden_actlowerrGEGLUReGLUSiGLUr/)r activationrrrmlp_activation_fnUs z$Phi3InferenceModel.mlp_activation_fncCtjSr) NormTypeEnumRMSNormrrrr norm_typeczPhi3InferenceModel.norm_typecCr<r)PositionalEmbeddingType rotate_halfrrrrpositional_embedding_typegr@z,Phi3InferenceModel.positional_embedding_typecCst|jjdS)N) theta_base)RotateHalfConfigr rope_thetarrrrpositional_embedding_configksz.Phi3InferenceModel.positional_embedding_config ragged_batchcCs<|||jj}|jd|jkrtd|jd|j|S)z Performs the embedding lookup prior to running the transformer of the model. Arguments: ragged_batch (RaggedBatchWrapper): The batch to embed. Returns: torch.Tensor: The embedded batch. zEmbedding output shape z does not match model_dim )embedrword_embshaper ValueError)rrHrJrrr_forward_embedss z!Phi3InferenceModel._forward_embed layer_idxresidual hidden_statesragged_batch_infocCs|j|}|j|}|j||jdd}||||}|j||jdd}|jdkr1t j ||j d|j |||j dd\}}|j||jdd}|j||jdd}|jdkr\t j ||j d||jdkrz|j|d}|j |||jdd\}}||fS||||fS)aL Executes one (slightly offset) layer of the transformer. This implementation does a peak-ahead optimization to fuse the layer norm of the next layer into the current layer. Arguments: layer_idx (int): The index of the layer to execute. residual (torch.Tensor): The residual tensor from the previous layer. hidden_states (torch.Tensor): The hidden states from the previous layer. This is the hidden states after pre normalization. ragged_batch_info (RaggedBatchWrapper): The batch metadata. N)br group)beta)r state_manager get_cacheqkvqkv_wattnattn_out attn_out_wtp_sizedist all_reduce_base_mp_groupnormmlp_norm_gammamlp_1mlp_1_wmlp_2mlp_2_wrattn_norm_gammaadd_)rrOrPrQrR cur_paramskv_cache next_paramsrrr_forward_transformer_layers$    z-Phi3InferenceModel._forward_transformer_layercCs|j||jj||jjd}|jdkrKt|j|j|jd|jdf}t|j|jd|j f}t j |||j d| |ddd|jd|j |S|S)z Performs unembedding of the hidden states to logits. This will only sample the final token of each sequence. )gammar rrTr )unembedrword_unembed_wfinal_norm_gammar^r _comm_logitsrL_return_logitsr!r_all_gather_into_tensorracopy_permutereshape)rrQrRlogits comm_buffer full_logitsrrr_forward_unembeds  $z#Phi3InferenceModel._forward_unembed wrapped_batchcCsX||}|j|d|jdjdd\}}t|jD] }|||||\}}q|||S)Nr)rnrV)rNrbrrhrangerrmr{)rr|rPrQrOrrrforwards   zPhi3InferenceModel.forwardN)%__name__ __module__ __qualname____doc__rr __annotations__rrpropertyintrrr r!r#r"r&r(rr0rr;r=r?rArCrErGr r*TensorrNrrmr{r~rrrrrsV    )r)typingrrrr*deepspeed.commcommr_ allocatorrinference_utilsrrmodules.configsmodules.interfacesraggedr containersr rDSTransformerModelBaserrrrrs