o ۾iU<@s.ddlmZddlZddlmZddlmZddlmZddl m Z m Z ddl m Z ddlmZddlmZmZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZmZddl m!Z!ddl"m#Z#m$Z$m%Z%m&Z&e e'Z(GdddeZeddddddGdddej)Z*GdddeZ+dS))IterableN) LlamaConfig)support_torch_compile) VllmConfigget_current_vllm_config) init_logger)RMSNorm)QKVParallelLinearReplicatedLinear)LogitsProcessor)QuantizationConfig)ParallelLMHeadVocabParallelEmbedding)default_weight_loadermaybe_remap_kv_scale_name)LlamaDecoderLayerLlamaForCausalLM) NestedTensors)AutoWeightsLoaderget_draft_quant_config maybe_prefixprocess_eagle_weightc seZdZ   ddedededBdeddf fd d ZdededBfd d Z d e j de e j e j ffddZ d e j de e j e j ffddZde j de j d e j de j dBde e j e j ff ddZZS)rNr vllm_configprefixconfig layer_idxreturnc stj|||d|p|jj}||}|dkrd|jn|j}t|dd}t||jj |jj |jj ||t |dd|j_ t|j|jd|_||_t|d drT|j|_dS|j|_dS) N)rrrattention_biasFqkv_proj)bias quant_configrepsnorm_before_residual)super__init__ model_config hf_configget_quant_config hidden_sizegetattrr self_attnhead_dimtotal_num_headstotal_num_kv_headsrr!r rms_norm_eps hidden_normr_norm_before_residual_residual_norm_norm_after_residual)selfrrrrr#qkv_input_sizeqkv_bias __class__[/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llama_eagle3.pyr('s&        zLlamaDecoderLayer.__init__cCst|S)z8Use drafter's quantization config instead of verifier's.)r)r7rr<r<r=r+Msz"LlamaDecoderLayer.get_quant_config hidden_statescCs||}|}||fSNr3r7r>residualr<r<r=r4Qs z'LlamaDecoderLayer._norm_before_residualcCs|}||}||fSr?r@rAr<r<r=r6Xs z&LlamaDecoderLayer._norm_after_residual positionsembedsrBcCsx|jdkr||}|j|d\}}tj||gdd}n|||\}}|j||d}|||\}}||}||fS)Nr)r>)dim)rCr>)rinput_layernormr5torchcatr.post_attention_layernormmlp)r7rCrDr>rBr<r<r=forward_s   zLlamaDecoderLayer.forward)rNr)__name__ __module__ __qualname__rstrrintr(r r+rHTensortupler4r6rL __classcell__r<r<r:r=r&sJ&  rrE) input_idsrCr> input_embeds)dynamic_arg_dimsc seZdZddddedededdffd d Zd ejdejfd d Z dd ejdejdejdejdBde ejejff ddZ de e eejfde efddZZS) LlamaModelrr)start_layer_idrrrYrrNc st|jjj_jj_t|_t jdd}|dur*d|vr*|d_ nd_ t t jjjj tdd_tfddtjjD_j r}tjdrcjjd }njj d }t|jj d |jjjtd d d _tjj jjd _dS)N eagle_configuse_aux_hidden_stateT embed_tokensrc s.g|]}ttd|j|dqS)zlayers.)rrr)rrr).0rcurrent_vllm_configrr7rYr<r= sz'LlamaModel.__init__..target_hidden_sizeFfc) input_size output_sizer" params_dtyper#r return_biasr$)r'r(speculative_configdraft_model_configr*r vocab_sizerr#r-r[rrr,rr\nn ModuleListrangenum_hidden_layerslayershasattrrbr r)dtyperdrr2norm)r7rrYrrZ fc_input_sizer:r_r=r(sH          zLlamaModel.__init__rUcCs ||Sr?)r\)r7rUr<r<r=embed_input_idss zLlamaModel.embed_input_idsrCr>rVcCsf|dur ||}|jd|jdksJd}|jD] }|||||d\}}q|||\}}||fS)NrE)rCrDr>rB)rushaperprs)r7rUrCr>rVrBlayerhidden_prenormr<r<r=rLs   zLlamaModel.forwardweightsc Cs*gd}t|}t}|D]\}}d|vr|dd}|jdurK|j|}rK||}t|dt} |dkr<|n|d}| ||| |qd|vsSd|vr]t ||}|dur]q|D]\} } } | |vriq_|| | }||}|j } | ||| n||}t|dt} | ||| |q|S)N)) .qkv_projz.q_projq)rzz.k_projk)rzz.v_projv) .gate_up_projz .gate_projr)r~z.up_projrz midlayer.z layers.0. weight_loaderrscale zero_point) dictnamed_parameterssetreplacer#get_cache_scaler-rrFaddrr) r7rystacked_params_mapping params_dict loaded_paramsname loaded_weight scale_nameparamr param_name weight_nameshard_idr<r<r= load_weightssB             zLlamaModel.load_weightsr?)rMrNrOrrQrPr(rHrRrurSrLrrrrTr<r<r:r=rX}s4 : ,rXc @seZdZdddedefddZ  ddejd edBd ejdBd ejfd d Z ddejdejdejdejdBd e ejejff ddZ dejd ejdBfddZ dejd ejfddZ dee eejffddZdS)Eagle3LlamaForCausalLMrr]rrcCstj||jjj|_t|jdddur t|jdd}||j_|j |j }||j_ t |d|d|_t|jdd}t|jj|jjt|dd|_t|jj|d |_tjtj|jjtjd d d |_|jj|_|jr|jd td|jjrvdnd|jjd ddSdS)Ndraft_vocab_sizerkmodel)rrrY logit_scaleg?lm_headr])r)rrF) requires_grad mask_hiddenrrc) persistent)rlModuler(rirjr*rr-rr)get_num_layersparallel_configtarget_layer_countrXrr r,rrr logits_processor ParameterrHzeroslongdraft_id_to_target_idparallel_draftinguse_parallel_draftingregister_bufferr[)r7rrbase_vocab_sizetarget_layer_numrr<r<r=r( sL    zEagle3LlamaForCausalLM.__init__NrUmultimodal_embeddings is_multimodalrcCs |j|Sr?)rru)r7rUrrr<r<r=ru8s z&Eagle3LlamaForCausalLM.embed_input_idsrCr> inputs_embedscCs|||||Sr?)r)r7rUrCr>rr<r<r=rL@szEagle3LlamaForCausalLM.forwardcCs||j|}|jdur$|jd|jjks"Jd|jjd|j|Stj|jj|j d}||j}| |jd|jjft d}||dd|f<|S)Nrz"Expected logits to have shape (*, z ), but got )devicerz-inf) rrrrvrrkrHarangerrnew_fullfloat)r7r>logitsbasetargets logits_newr<r<r=compute_logitsIs(  z%Eagle3LlamaForCausalLM.compute_logitscCs|jjs|S|j|Sr?)rr[rd)r7r>r<r<r=combine_hidden_statesas z,Eagle3LlamaForCausalLM.combine_hidden_statesryc Cs i}d}d}d}|D]G\}}d|vrq d|vr |dd}d}n"d|vr:|js-tdq |j|dd d}q d |vrBd |}d |vrHd}|||<t||q |s[|jr[td dg}|se| d|sl| d |j j su| dt |d|d} | |dS)NFt2dd2trTrzmmask_hidden found in weights but model is not configured for parallel drafting. Skipping loading mask_hidden.rrErzmodel.r\zzmask_hidden not found in weights but model is configured for parallel drafting. Please provide mask_hidden in the weights.zfc.) skip_prefixes skip_substrs)rrloggerwarningrcopy_viewr ValueErrorappendrr[rritems) r7ry model_weightsincludes_draft_id_mappingincludes_embed_tokensincludes_mask_hiddenrrrloaderr<r<r=rjsT       z#Eagle3LlamaForCausalLM.load_weights)NNr?)rMrNrOrrPr(rHrRrrurSrLrrrrr<r<r<r=r sH1      r),collections.abcrrHtorch.nnrl transformersrvllm.compilation.decoratorsr vllm.configrr vllm.loggerr$vllm.model_executor.layers.layernormr!vllm.model_executor.layers.linearr r +vllm.model_executor.layers.logits_processorr 3vllm.model_executor.layers.quantization.base_configr 3vllm.model_executor.layers.vocab_parallel_embeddingr r-vllm.model_executor.model_loader.weight_utilsrr vllm.model_executor.models.llamarrvllm.multimodal.inputsrutilsrrrrrMrrrXrr<r<r<r=s8         W