o ۾i<@sdZddlmZddlmZddlmZddlmZddl Z ddl m Z ddl m Z dd l mZdd lmZmZdd lmZmZdd lmZdd lmZddlmZddlmZddlmZmZm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1m2Z2m3Z3m4Z4m5Z5ee6Z7ede8dBde8dBde j9fddZ:Gdd d e j9Z;Gd!d"d"e j9ZGd'd(d(e j9e.e/Z?dS))z?Inference-only Gemma model compatible with HuggingFace weights.)Iterable)cache)islice)AnyN)nn) GemmaConfig)support_torch_compile) CacheConfig VllmConfig) get_pp_group$get_tensor_model_parallel_world_size) init_logger) GeluAndMul) Attention) GemmaRMSNorm)MergedColumnParallelLinearQKVParallelLinearRowParallelLinear)LogitsProcessor)QuantizationConfig)get_rope)VocabParallelEmbedding)default_weight_loader)IntermediateTensors) SupportsLoRA SupportsPP)AutoWeightsLoaderis_pp_missing_parameter'make_empty_intermediate_tensors_factory make_layers maybe_prefix hidden_acthidden_activationreturncCs\|dur|durtd||tddS|dkrtddS|dkr&tddStd|d) Na~Gemma's activation function was incorrectly set to exact GeLU in the config JSON file when it was initially released. Changing the activation function to approximate GeLU (`gelu_pytorch_tanh`). If you want to use the legacy `%s`, edit the config JSON to set `hidden_activation=%s` instead of `hidden_act`. See https://github.com/huggingface/transformers/pull/29402 for more details.tanh) approximategelu_pytorch_tanhgelunonezActivation function z# is not supported for Gemma models.)loggerwarningr ValueError)r"r#r-T/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py_get_gemma_act_fn<s    r/csXeZdZ    ddedededBdedBdedBded dffd d Zd d ZZS)GemmaMLPN hidden_sizeintermediate_sizer"r# quant_configprefixr$csTtt||gdd||dd|_t||d||dd|_t|||_dS)NFz .gate_up_projbiasr4r5z .down_proj)super__init__r gate_up_projr down_projr/act_fn)selfr2r3r"r#r4r5 __class__r-r.r:[s zGemmaMLP.__init__cCs*||\}}||}||\}}|SN)r;r=r<)r>xgate_up_r-r-r.forwardus zGemmaMLP.forward)NNNr1) __name__ __module__ __qualname__intstrrr:rE __classcell__r-r-r?r.r0Zs*r0cs|eZdZ    ddededededeeefd ed edBd edBd ed dffdd Z de j de j d e j fddZ Z S)GemmaAttention Nr1r2 num_heads num_kv_headshead_dimrope_parametersmax_position_embeddings cache_configr4r5r$c s4t||_t} ||_|j| dksJ|j| |_||_|j| kr/|j| dks.Jn | |jdks8Jtd|j| |_||_ |j|j |_ |j|j |_ |j d|_ t ||j |j|jd|| dd|_t|j|j |d|| dd|_t|j ||dd |_t|j|j |j |j||| d d |_dS) NrrgFz .qkv_projr7z.o_projT) max_positionrQ is_neox_stylez.attn)rOrSr4r5)r9r:r2r total_num_headsrNtotal_num_kv_headsmaxrOrPq_sizekv_sizescalingrqkv_projro_projr rotary_embrattn) r>r2rNrOrPrQrRrSr4r5tp_sizer?r-r.r:}s\      zGemmaAttention.__init__ positions hidden_statesc Cs`||\}}|j|j|j|jgdd\}}}||||\}}||||}||\} }| S)N)dim)r\splitrYrZr^r_r]) r>rarbqkvrDqkv attn_outputoutputr-r-r.rEs  zGemmaAttention.forward)rMNNr1)rFrGrHrIdictrJrr rr:torchTensorrErKr-r-r?r.rL|sB    BrLc sreZdZ   ddededBdedBdeddf fdd Zd ej d ej d ej dBde ej ej ffd dZ Z S)GemmaDecoderLayerNr1configrSr4r5r$c st|j|_t|j|j|j|j|j|j|||dd |_ t |j|j |j t |dd||dd|_t|j|jd|_t|j|jd|_dS)Nz .self_attn) r2rNrOrPrRrQrSr4r5r#z.mlp)r2r3r"r#r4r5eps)r9r:r2rLnum_attention_headsnum_key_value_headsrPrRrQ self_attnr0r3r"getattrmlpr rms_norm_epsinput_layernormpost_attention_layernorm)r>rprSr4r5r?r-r.r:s2    zGemmaDecoderLayer.__init__rarbresidualcCsX|dur |}||}n|||\}}|j||d}|||\}}||}||fS)N)rarb)ryrurzrw)r>rarbr{r-r-r.rEs  zGemmaDecoderLayer.forward)NNr1) rFrGrHrr rrJr:rmrntuplerErKr-r-r?r.ros0!roc seZdZdddedeffddZdejdejfd d Z ddejd Bd ejd e d Bdejd Bdeje Bf ddZ de e eejfde efddZZS) GemmaModelr1r5 vllm_configr5cst|jj|j|j|_tjj |_ t j fdd|dd\|_ |_|_tj jd|_|jj d}|jdt|dd td d gj |_dS) Ncst|dS)Nr~)ror~rSrpr4r-r.sz%GemmaModel.__init__..z.layersr~rqg? normalizerF) persistentrbr{)r9r: model_config hf_configrSr4rpr vocab_sizer2 embed_tokensr num_hidden_layers start_layer end_layerlayersrrxnormregister_bufferrmtensorrmake_empty_intermediate_tensors)r>rr5rr?rr.r:s(    zGemmaModel.__init__ input_idsr$cCs ||SrA)rr>rr-r-r.embed_input_ids's zGemmaModel.embed_input_idsNraintermediate_tensors inputs_embedsc Cstjr|dur |}n||}||j9}d}n|d}|d}t|j|j|jD] }||||\}}q)tjs?t ||dS| ||\}}|S)Nrbr{)rbr{) r is_first_rankrrrrrr is_last_rankrr) r>rrarrrbr{layerrDr-r-r.rE*s(   zGemmaModel.forwardweightsc Csgd}t|}t}|D]Y\}}|D].\}}} ||vrq|||}|dr/||vr/qt||r5q||} | j} | | || n|drN||vrNqt||rTq||} t| dt} | | || |q|S)N))r\q_projrg)r\k_projrh)r\v_projri)r; gate_projr)r;up_projrz.bias weight_loader) rlnamed_parameterssetreplaceendswithrrrvradd) r>rstacked_params_mapping params_dict loaded_paramsname loaded_weight param_name shard_nameshard_idparamrr-r-r. load_weightsHs2         zGemmaModel.load_weightsrA)rFrGrHr rJr:rmrnrrrErr|rrrKr-r-r?r.r}s   ,r}c seZdZgdddgdZdddedeffd d Zd ejd ejfd dZ  dd ejdBdejde dBdejdBd eje Bf ddZ dejd ejdBfddZ de eeejfd eefddZZS)GemmaForCausalLM)rrrrr)r\r;r1r~rr5cs\t|jj}|j}||_|jsJ||_t|t|dd|_ t |j |_ |j j |_ dS)Nmodel)rr5)r9r:rrr4rptie_word_embeddingsr}r!rrrlogits_processorr)r>rr5rpr4r?r-r.r:|s    zGemmaForCausalLM.__init__rr$cCs |j|SrA)rrrr-r-r.rs z GemmaForCausalLM.embed_input_idsNrarrcCs|||||}|SrA)r)r>rrarrrbr-r-r.rEszGemmaForCausalLM.forwardrbcCs||jj|}|SrA)rrr)r>rblogitsr-r-r.compute_logitsszGemmaForCausalLM.compute_logitsrcCs$t||jjr dgndd}||S)Nzlm_head.) skip_prefixes)rrprr)r>rloaderr-r-r.rs  zGemmaForCausalLM.load_weights)NN)rFrGrHpacked_modules_mappingr rJr:rmrnrrrErrr|rrrKr-r-r?r.ros6    ,r)@__doc__collections.abcr functoolsr itertoolsrtypingrrmr transformersrvllm.compilation.decoratorsr vllm.configr r vllm.distributedr r vllm.loggerr %vllm.model_executor.layers.activationr$vllm.model_executor.layers.attentionr$vllm.model_executor.layers.layernormr!vllm.model_executor.layers.linearrrr+vllm.model_executor.layers.logits_processorr'vllm.model_executor.layers.quantizationr+vllm.model_executor.layers.rotary_embeddingr3vllm.model_executor.layers.vocab_parallel_embeddingr-vllm.model_executor.model_loader.weight_utilsr vllm.sequencer interfacesrrutilsrrrr r!rFr*rJModuler/r0rLror}rr-r-r-r.sN                  "P9i