o TiS@sddlmZmZmZddlZddlmZddlm Z ddl m Z m Z ddl TddlTddlTddlmZd d lmZmZGd d d eZdS) )IterableOptionalTupleN) empty_from)ActivationType DtypeEnum)*)RaggedBatchWrapper)PhiNonTransformerContainerPhiTransformerContainerc @seZdZUdZeeed< eeeed< e de fddZ e de fddZ e de fd d Z e de fd d Ze de fd dZe de fddZe de fddZe de fddZe defddZe defddZe defddZe defddZe deefddZ dedejfd d!Z d"e d#ejd$ejd%ede!ejejff d&d'Z"d$ejd%edejfd(d)Z#d*edejfd+d,Z$d-S).PhiInferenceModelzP Inference model implementation for ragged batching for Llama-2 models. _non_transformer _transformerreturncC|jjSN)_configmax_seq_lengthselfrj/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/model_implementations/phi/model.pymax_sequence_length(z%PhiInferenceModel.max_sequence_lengthcCrr)rnum_hidden_layersrrrr num_layers0rzPhiInferenceModel.num_layerscCrr)r hidden_sizerrrr model_dim4rzPhiInferenceModel.model_dimcCrr)r vocab_sizerrrrr!8rzPhiInferenceModel.vocab_sizecCs |j|jSr)r n_headsrrrr head_size<s zPhiInferenceModel.head_sizecCrr)rnum_attention_headsrrrrr"@rzPhiInferenceModel.n_headscCrr)rintermediate_sizerrrrintermediate_dimDrz"PhiInferenceModel.intermediate_dimcCrr)rnum_key_value_headsrrrr n_heads_kvHrzPhiInferenceModel.n_heads_kvcCs0|jjtjkr tjS|jjtjkrtjStd)Nz Only fp16 and bf16 are supported) r torch_dtypetorchfloat16rfp16bfloat16bf16NotImplementedErrorrrrractivation_dtypeLs z"PhiInferenceModel.activation_dtypecCtjSr)rGELUrrrrmlp_activation_fnUz#PhiInferenceModel.mlp_activation_fncCr1r) NormTypeEnum LayerNormrrrr norm_typeYr4zPhiInferenceModel.norm_typecCr1r)PositionalEmbeddingType rotate_halfrrrrpositional_embedding_type]r4z+PhiInferenceModel.positional_embedding_typecCs"t|jj|j}t||jjdS)N) rotate_dim theta_base)intrpartial_rotary_factorr#RotateHalfConfig rope_theta)r rotary_dimrrrpositional_embedding_configasz-PhiInferenceModel.positional_embedding_config ragged_batchcCs<|||jj}|jd|jkrtd|jd|j|S)z Performs the embedding lookup prior to running the transformer of the model. Arguments: ragged_batch (RaggedBatchWrapper): The batch to embed. Returns: torch.Tensor: The embedded batch. zEmbedding output shape z does not match model_dim )embedrword_embshaper ValueError)rrCrErrr_forward_embedjs z PhiInferenceModel._forward_embed layer_idxresidual hidden_statesragged_batch_infocCs|j|}|j|}|}|j||j|jd}||||}|j||j|j d} |} |j | |j |j d} |j | |j|jd} | | |jdkrPtj| |jd||jdkro|j|d} |j|| | j| jd\}} || fS|| || fS)aL Executes one (slightly offset) layer of the transformer. This implementation does a peak-ahead optimization to fuse the layer norm of the next layer into the current layer. Arguments: layer_idx (int): The index of the layer to execute. residual (torch.Tensor): The residual tensor from the previous layer. hidden_states (torch.Tensor): The hidden states from the previous layer. This is the hidden states after pre normalization. ragged_batch_info (RaggedBatchWrapper): The batch metadata. )br group)beta)r state_manager get_cacheqkvqkv_wqkv_battnattn_out attn_out_w attn_out_bmlp_1mlp_1_wmlp_1_bmlp_2mlp_2_wmlp_2_badd_tp_sizedist all_reduce_base_mp_grouprnormln_gammaln_beta)rrJrKrLrM cur_paramskv_cache attn_ln_outattn_hidden_stateattention_output mlp_ln_outmlp_hidden_state mlp_output next_paramsrrr_forward_transformer_layer{s$    z,PhiInferenceModel._forward_transformer_layercCs|j||jj||jj|jj|jjd}|jdkrQt|j|j|j d|j df}t|j |j d|j f}t j |||jd||ddd|j d|j |S|S)z Performs unembedding of the hidden states to logits. This will only sample the final token of each sequence. )biasgammarQr rrOr )unembedrword_unembed_wword_unembed_bfinal_norm_gammafinal_norm_betarbr _comm_logitsrG_return_logitsr!rcall_gather_into_tensorrecopy_permutereshape)rrLrMlogits comm_buffer full_logitsrrr_forward_unembeds  $z"PhiInferenceModel._forward_unembed wrapped_batchcCs`||}|j|d|jdj|jdjd\}}t|jD] }|||||\}}q|||S)Nr)rtrQ) rIrfrrgrhrangerrrr)rrrKrLrJrrrforwards      zPhiInferenceModel.forwardN)%__name__ __module__ __qualname____doc__rr __annotations__rrpropertyr=rrr r!r#r"r&r(rr0rr3r5r7r8r:r?rBr r*TensorrIrrrrrrrrrrsV   (r)typingrrrr*deepspeed.commcommrc allocatorrinference_utilsrrmodules.configsmodules.interfacesraggedr containersr rDSTransformerModelBaserrrrrs