o TiX@sddlmZmZmZddlZddlmZddlm Z ddl m Z m Z ddl TddlTddlTddlmZd d lmZmZGd d d eZdS) )IterableOptionalTupleN) empty_from)ActivationType DtypeEnum)*)RaggedBatchWrapper)FalconNonTransformerContainerFalconTransformerContainerc @seZdZUdZeeed< eeeed< e de fddZ e de fddZ e de fd d Z e de fd d Ze de fd dZe de fddZe de fddZe de fddZe defddZe defddZe defddZe defddZe defddZ dedejfd d!Z d"e d#ejd$ejd%ede!ejejff d&d'Z"d$ejd%edejfd(d)Z#d*edejfd+d,Z$d-S).FalconInferenceModelzP Inference model implementation for ragged batching for Llama-2 models. _non_transformer _transformerreturncC|jjSN)_configmax_seq_lengthselfrm/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/model_implementations/falcon/model.pymax_sequence_length(z(FalconInferenceModel.max_sequence_lengthcCrr)rnum_hidden_layersrrrr num_layers0rzFalconInferenceModel.num_layerscCrrr hidden_sizerrrr model_dim4rzFalconInferenceModel.model_dimcCrr)r vocab_sizerrrrr"8rzFalconInferenceModel.vocab_sizecCs |j|jSr)r!n_headsrrrr head_size< zFalconInferenceModel.head_sizecCrr)rnum_attention_headsrrrrr#@rzFalconInferenceModel.n_headscCs d|jjS)Nrrrrrintermediate_dimDr%z%FalconInferenceModel.intermediate_dimcCs|jjs|jjs |jjSdS)Nr )rnew_decoder_architecture multi_query num_kv_headsrrrr n_heads_kvHs  zFalconInferenceModel.n_heads_kvcCs0|jjtjkr tjS|jjtjkrtjStd)Nz Only fp16 and bf16 are supported) r torch_dtypetorchfloat16rfp16bfloat16bf16NotImplementedErrorrrrractivation_dtypeMs z%FalconInferenceModel.activation_dtypecCtjSr)rGELUrrrrmlp_activation_fnVz&FalconInferenceModel.mlp_activation_fncCr5r) NormTypeEnum LayerNormrrrr norm_typeZr8zFalconInferenceModel.norm_typecCr5r)PositionalEmbeddingType rotate_halfrrrrpositional_embedding_type^r8z.FalconInferenceModel.positional_embedding_typecCstS)zG The positional embedding configuration for the model. )RotateHalfConfigrrrrpositional_embedding_configbsz0FalconInferenceModel.positional_embedding_config ragged_batchcCs<|||jj}|jd|jkrtd|jd|j|S)z Performs the embedding lookup prior to running the transformer of the model. Arguments: ragged_batch (RaggedBatchWrapper): The batch to embed. Returns: torch.Tensor: The embedded batch. zEmbedding output shape z does not match model_dim )embedrword_embshaper! ValueError)rrArCrrr_forward_embedms z#FalconInferenceModel._forward_embed layer_idxresidual hidden_statesragged_batch_infocCs|jjsJd|j|}|j|}|}|j||jdd}||||}|j||j dd} |jj r@|j |d|j |j d\}} n|} |j| |jdd} |j| |jdd} | | |jdkrftj| |jd||jdkr|j|d} |j || | j| jd\}} || fS|| || fS)aL Executes one (slightly offset) layer of the transformer. This implementation does a peak-ahead optimization to fuse the layer norm of the next layer into the current layer. Arguments: layer_idx (int): The index of the layer to execute. residual (torch.Tensor): The residual tensor from the previous layer. hidden_states (torch.Tensor): The hidden states from the previous layer. This is the hidden states after pre normalization. ragged_batch_info (RaggedBatchWrapper): The batch metadata. z3Only parallel attention implementation is supportedN)bgammabetar group)rO)config parallel_attnr state_manager get_cacheqkvqkv_wattnattn_out attn_out_wr)norm ln_mlp_gamma ln_mlp_betamlp_1mlp_1_wmlp_2mlp_2_wadd_tp_sizedist all_reduce_base_mp_groupr ln_attn_gamma ln_attn_beta)rrHrIrJrK cur_paramskv_cache attn_ln_outattn_hidden_stateattention_output mlp_ln_outmlp_hidden_state mlp_output next_paramsrrr_forward_transformer_layer~s:       z/FalconInferenceModel._forward_transformer_layercCs|j||jj||jj|jjd}|jdkrNt|j|j|jd|jdf}t|j |jd|j f}t j |||j d||ddd|jd|j |S|S)z Performs unembedding of the hidden states to logits. This will only sample the final token of each sequence. rMr rrPr )unembedr word_unembedfinal_norm_gammafinal_norm_betarcr _comm_logitsrE_return_logitsr"rdall_gather_into_tensorrfcopy_permutereshape)rrJrKlogits comm_buffer full_logitsrrr_forward_unembeds  $z%FalconInferenceModel._forward_unembed wrapped_batchcCs`||}|j|d|jdj|jdjd\}}t|jD] }|||||\}}q|||S)NrrM) rGr[rrgrhrangerrrr)rrrIrJrHrrrforwards      zFalconInferenceModel.forwardN)%__name__ __module__ __qualname____doc__rr __annotations__rrpropertyintrrr!r"r$r#r(r,rr4rr7r9r;r<r>r?r@r r.TensorrGrrrrrrrrrrsV   4r)typingrrrr.deepspeed.commcommrd allocatorrinference_utilsrrmodules.configsmodules.interfacesraggedr containerr rDSTransformerModelBaserrrrrs