o پi.<@s^dZddlmZmZmZddlZddlZddlm m Z ddlm Z ddl m Z ddlmZmZmZddlmZddlmZmZmZdd lmZdd lmZdd lmZdd lmZdd l m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(Gddde j)Z*Gddde j)Z+Gddde j)Z,Gddde j)Z-Gddde j)Z.Gddde j)Z/e/Z0dS)zInference-only Mixtral model.)IterableOptionalTupleN)nn) MixtralConfig)get_tensor_model_parallel_rank$get_tensor_model_parallel_world_size tensor_model_parallel_all_reduce)RMSNorm)QKVParallelLinearReplicatedLinearRowParallelLinear)LogitsProcessor)QuantizationConfig)RadixAttention)get_rope)ParallelLMHeadVocabParallelEmbedding) ForwardBatch)default_weight_loader) add_prefixc sVeZdZ  ddedededeededdf fd d Zd ej dej fd d Z Z S) MixtralMLPN num_experts hidden_sizeintermediate_size quant_configprefixreturncst||_||_||_t|j|jd|td|d|_t|j|jd|td|d|_t|j|jd|td|d|_ t |_ dS)NFw1biasrrw2w3) super__init__rffn_dim hidden_dimr rrr"r#rSiLUact_fn)selfrrrrr __class__S/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/models/mixtral_quant.pyr%4s4  zMixtralMLP.__init__ hidden_statescCs@||\}}||}||\}}||}||\}}|SN)rr)r#r")r*r/w1_out_w3_outcurrent_hidden_statesr-r-r.forwardZs  zMixtralMLP.forwardNr) __name__ __module__ __qualname__intrrstrr%torchTensorr5 __classcell__r-r-r+r.r3s"&rcsJeZdZ  d dedeedeffdd Zdej d ej fd d Z Z S) MixtralMoENrconfigrrcst_t_t_j_j _ jjkr+t djdjdt tjjj_jsGt djdtfddtjD_tjjddtd d _dS) NzTensor parallel size z' is greater than the number of experts .zRank z has no experts assigned to it.c s>g|]}|jvrtjjjtd|dndqS)zexperts.rrN)expert_indiciesrnum_total_expertsrrr).0idxr@rrr*r-r. }s z'MixtralMoE.__init__..Fgater )r$r%r@rrankrtp_sizenum_local_expertsrDnum_experts_per_toktop_k ValueErrornp array_splitrangetolistrCr ModuleListexpertsr rrrIr*r@rrr+rGr.r%dsB      zMixtralMoE.__init__r/rc Cs||\}}tj|dtjd}tj||jdd\}}||jddd}d}|jD]'}|j |}||k} || jddd} || | } |durK| }q)| | q)t |S)N)dimdtyperXT)rXkeepdim) rIFsoftmaxr<floattopkrNsumrCrUmul_add_r ) r*r/ router_logitsr2routing_weightsselected_expertsfinal_hidden_states expert_idx expert_layer expert_maskexpert_weightsr4r-r-r.r5s     zMixtralMoE.forwardr6) r7r8r9rrrr;r%r<r=r5r>r-r-r+r.r?cs0r?csreZdZ     ddededed ed ed ed eed eddffdd Zde j de j de de j fddZ Z S)MixtralAttentionr'Nrr num_heads num_kv_headslayer_id max_position rope_thetarrrc sJt||_t} ||_|j| dksJ|j| |_||_|j| kr/|j| dks.Jn | |jdks8Jtd|j| |_||j|_ |j|j |_ |j|j |_ |j d|_ ||_ t||j |j|jd|td|d|_t|j|j |d|td|d|_t|j |j |t|j dd |_t|j|j |j |j||td |d |_dS) NrrWgFqkv_projr o_projT) rotary_dimrrbase is_neox_styleattn)rprqrr)r$r%rrtotal_num_headsrototal_num_kv_headsmaxrphead_dimq_sizekv_sizescalingrsr rrtr rurr: rotary_embrry) r*rrorprqrrrsrrrKr+r-r.r%s`       zMixtralAttention.__init__ positionsr/ forward_batchc Csb||\}}|j|j|j|jgdd\}}}||||\}}|||||} || \} }| S)NrZr[)rtsplitr~rrryru) r*rr/rqkvr2qkv attn_outputoutputr-r-r.r5s  zMixtralAttention.forward)rrmrnNr)r7r8r9r:r_rrr;r%r<r=rr5r>r-r-r+r.rlsD  Brlc sheZdZ   ddededeededdf fd d Zd e j d e j d e dee j de j f ddZ Z S)MixtralDecoderLayerrNrr@rqrrrc st|j|_t|dd}t|j|j|j|j|||td|d|_ t ||td|d|_ t |j|j d|_t |j|j d|_dS)Nrsrn self_attn)rrorrrprqrsrrblock_sparse_moe)r@rreps)r$r%rgetattrrlnum_attention_headsmax_position_embeddingsnum_key_value_headsrrr?rr rms_norm_epsinput_layernormpost_attention_layernorm)r*r@rqrrrsr+r-r.r%s,    zMixtralDecoderLayer.__init__rr/rresidualcCsZ|dur |}||}n|||\}}|j|||d}|||\}}||}||fS)N)rr/r)rrrr)r*rr/rrr-r-r.r5s  zMixtralDecoderLayer.forward)rNr)r7r8r9rr:rrr;r%r<r=rr5r>r-r-r+r.rs4rc sbeZdZ  ddedeededdffdd Z dd ej d ej d e d ej dej f d dZ Z S) MixtralModelNrr@rrrcsntj|_j|_tjjtdd|_t fddt j D|_ tjjd|_dS)N embed_tokensrc s(g|]}t|td|dqS)zlayers.rB)rr)rEir@rrr-r.rHGsz)MixtralModel.__init__..r)r$r% pad_token_id padding_idx vocab_sizerrrrrrTrRnum_hidden_layerslayersr rnormrVr+rr.r%7s  zMixtralModel.__init__ input_idsrr input_embedsc Cs`|dur ||}n|}d}tt|jD]}|j|}|||||\}}q|||\}} |Sr0)rrRlenrr) r*rrrrr/rrlayerr2r-r-r.r5Ss   zMixtralModel.forwardr6r0) r7r8r9rrrr;r%r<r=rr5r>r-r-r+r.r6s0!rc seZdZ  ddedeededdffdd Ze dd ej d ej d e d ej dej f d dZ de eeej ffddZZS)QuantMixtralForCausalLMNrr@rrrcsTt||_||_t||td|d|_t|j|j td|d|_ t ||_ dS)NmodelrBlm_headr) r$r%r@rrrrrrrrrlogits_processorrVr+r-r.r%is  z QuantMixtralForCausalLM.__init__rrrrcCs"|||||}||||j|Sr0)rrr)r*rrrrr/r-r-r.r5zs zQuantMixtralForCausalLM.forwardweightsc Csgd}t|}|D]`\}}d|vrq |D]-\}}}||vr!q|||}|dr1||vr1q||vr6q||} | j} | | ||n'|drO||vrOq d|vrX||vrXq ||vr]q ||} t| dt} | | |q dS)N))rtq_projr)rtk_projr)rtv_projrzrotary_emb.inv_freqz.biaszblock_sparse_moe.experts. weight_loader)dictnamed_parametersreplaceendswithrrr) r*rstacked_params_mapping params_dictname loaded_weight param_name weight_nameshard_idparamrr-r-r. load_weightss8      z$QuantMixtralForCausalLM.load_weightsr6r0)r7r8r9rrrr;r%r<no_gradr=rr5rrrr>r-r-r+r.rhs4 $ r)1__doc__typingrrrnumpyrPr<torch.nn.functionalr functionalr] transformersrsglang.srt.distributedrrr sglang.srt.layers.layernormr sglang.srt.layers.linearr r r "sglang.srt.layers.logits_processorr*sglang.srt.layers.quantization.base_configr!sglang.srt.layers.radix_attentionr"sglang.srt.layers.rotary_embeddingr*sglang.srt.layers.vocab_parallel_embeddingrr,sglang.srt.model_executor.forward_batch_infor$sglang.srt.model_loader.weight_utilsrsglang.srt.utilsrModulerr?rlrrr EntryClassr-r-r-r.s2          0IQ92F