o Ti@sddlmZmZmZddlZddlmZddlm Z ddl m Z m Z ddl TddlTddlTddlmZdd lmZd d lmZmZGd d d eZdS))IterableOptionalTupleN) empty_from)ActivationType DtypeEnum)*) heuristics)RaggedBatchWrapper)QwenNonTransformerContainerQwenTransformerContainerc @seZdZUdZeeed< eeeed< e de fddZ e de fddZ e de fd d Z e de fd d Ze de fd dZe de fddZe de fddZe de fddZe defddZe defddZe defddZe defddZe deefddZd0d d!Z d"edej fd#d$Z!d%e d&ej d'ej d(ede"ej ej ff d)d*Z#d'ej d(edej fd+d,Z$d-edej fd.d/Z%dS)1QwenInferenceModelzP Inference model implementation for ragged batching for Llama-2 models. _non_transformer _transformerreturncC|jjSN)_configmax_seq_lengthselfrk/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/model_implementations/qwen/model.pymax_sequence_length)z&QwenInferenceModel.max_sequence_lengthcCrr)rnum_hidden_layersrrrr num_layers1rzQwenInferenceModel.num_layerscCrr)r hidden_sizerrrr model_dim5rzQwenInferenceModel.model_dimcCrr)r vocab_sizerrrrr"9rzQwenInferenceModel.vocab_sizecCs |j|jSr)r!n_headsrrrr head_size= zQwenInferenceModel.head_sizecCrr)rnum_attention_headsrrrrr#ArzQwenInferenceModel.n_headscCs |jjdS)Nr )rintermediate_sizerrrrintermediate_dimEr%z#QwenInferenceModel.intermediate_dimcCs|jj|jjSr)rr kv_channelsrrrr n_heads_kvIszQwenInferenceModel.n_heads_kvcCsB|jj|jjdk}|rtjS|jjrtjS|jjrtjStd)Nrz Only fp16 and bf16 are supported)rbf16fp16rNotImplementedError)rautoset_precisionrrractivation_dtypeMsz#QwenInferenceModel.activation_dtypecCtjSr)rSiGLUrrrrmlp_activation_fn[z$QwenInferenceModel.mlp_activation_fncCr0r) NormTypeEnumRMSNormrrrr norm_type_r3zQwenInferenceModel.norm_typecCr0r)PositionalEmbeddingType rotate_halfrrrrpositional_embedding_typecr3z,QwenInferenceModel.positional_embedding_typecCst|jjdS)N) theta_base)RotateHalfConfigrrotary_emb_baserrrrpositional_embedding_configgsz.QwenInferenceModel.positional_embedding_configNc Cs>t|jjj|j|j|j|j|j|jjd}t ||j|_ dS)a Instantiates the normalization layer for the model. This sets the `self.norm` attribute. TODO(cmikeh2): In the future we'll distinguish between the different norm objects, but for now we'll just use the same one for all of them. ) max_tokenstypechannelsresidual_dtype input_dtype output_dtypeepsN) DSNormConfig_engine_config state_managermax_ragged_batch_sizer6r!r/rlayer_norm_epsilonr instantiate_pre_normnorm)r norm_configrrrmake_norm_layerks z"QwenInferenceModel.make_norm_layer ragged_batchcCs<|||jj}|jd|jkrtd|jd|j|S)z Performs the embedding lookup prior to running the transformer of the model. Arguments: ragged_batch (RaggedBatchWrapper): The batch to embed. Returns: torch.Tensor: The embedded batch. zEmbedding output shape z does not match model_dim )embedrword_embshaper! ValueError)rrNrPrrr_forward_embeds z!QwenInferenceModel._forward_embed layer_idxresidual hidden_statesragged_batch_infocCs|j|}|j|}|j||j|jd}||||}|j||jdd}|j dkr2t j ||j d|j |||jdd\}}|j||jdd}|j||jdd}|j dkr]t j ||j d||jdkr{|j|d}|j |||jdd\}}||fS||||fS)aL Executes one (slightly offset) layer of the transformer. This implementation does a peak-ahead optimization to fuse the layer norm of the next layer into the current layer. Arguments: layer_idx (int): The index of the layer to execute. residual (torch.Tensor): The residual tensor from the previous layer. hidden_states (torch.Tensor): The hidden states from the previous layer. This is the hidden states after pre normalization. ragged_batch_info (RaggedBatchWrapper): The batch metadata. )bNr groupbeta)rrG get_cacheqkvqkv_wqkv_battnattn_out attn_out_wtp_sizedist all_reduce_base_mp_grouprKmlp_norm_gammamlp_1mlp_1_wmlp_2mlp_2_wrattn_norm_gammaadd_)rrUrVrWrX cur_paramskv_cache next_paramsrrr_forward_transformer_layers$     z-QwenInferenceModel._forward_transformer_layercCs|j||jj||jjd}|jdkrKt|j|j|jd|jdf}t|j|jd|j f}t j |||j d| |ddd|jd|j |S|S)z Performs unembedding of the hidden states to logits. This will only sample the final token of each sequence. )gammar rrZr )unembedr word_unembed final_normrer _comm_logitsrR_return_logitsr"rfall_gather_into_tensorrhcopy_permutereshape)rrWrXlogits comm_buffer full_logitsrrr_forward_unembeds  $z#QwenInferenceModel._forward_unembed wrapped_batchcCsX||}|j|d|jdjdd\}}t|jD] }|||||\}}q|||S)Nrr\)rTrKrrnrangerrsr)rrrVrWrUrrrforwards   zQwenInferenceModel.forward)rN)&__name__ __module__ __qualname____doc__rr__annotations__rrpropertyintrrr!r"r$r#r(r*rr/rr2r4r6r7r9r;r=rMr torchTensorrTrrsrrrrrrrsX     ,r)typingrrrrdeepspeed.commcommrf allocatorrinference_utilsrrmodules.configsmodules.interfacesmodulesr raggedr containerrrDSTransformerModelBaserrrrrs