o پi@sUddlmZddlmZddlmZmZddlZddlm Z m Z m Z m Z m Z ddlmZer7ddlmZmZdaded <eGd d d e ZeGd d d e ZeGddde Ze dddddZdS)) annotations) dataclass) TYPE_CHECKINGOptionalN) MoeQuantInfoMoeRunnerConfig RunnerInput RunnerOutputregister_fused_func)MoeRunnerBackend)StandardCombineInputStandardDispatchOutputOptional[torch.Tensor]MARLIN_MOE_WORKSPACEc@s@eZdZUdZded<ded<ded<ded<ed d d Zd S) MarlinRunnerInputz.Input bundle passed to the Marlin runner core. torch.Tensor hidden_states topk_weightstopk_ids router_logitsreturnr cCtjSNr MARLINselfr[/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/layers/moe/moe_runner/marlin.pyrunner_backend#z MarlinRunnerInput.runner_backendNrr __name__ __module__ __qualname____doc____annotations__propertyrrrrrrs rc@s(eZdZUdZded<ed ddZdS) MarlinRunnerOutputz3Output bundle returned from the Marlin runner core.rrrr cCrrrrrrrr.r z!MarlinRunnerOutput.runner_backendNr!r"rrrrr)(s r)c@seZdZUdZded<ded<ded<ded<ded<ded <d ed <d Zded <d Zded<dZded<d Zded<d Z ded<d Z ded<d S)MarlinMoeQuantInfoz4Quantization payload consumed by the Marlin backend.r w13_qweight w2_qweight w13_scales w2_scalesrw13_g_idx_sort_indicesw2_g_idx_sort_indicesint weight_bitsN w13_g_idxw2_g_idxTbool is_k_full w13_qzeros w2_qzeros expert_map) r#r$r%r&r'r3r4r6r7r8r9rrrrr*3s      r*nonemarlindispatch_outputr quant_info runner_configrrr c Csddlm}ddlm}ddlm}|j}|j}|jdks!Jdt dus+t j |j kr2||j dda |did |d |j d |j d |j d |jd|jd|jd|jd|jd|jd|jd|jd|jd|jd|jdt d|jd|jd|jd|j|j}||dS)Nr)fused_marlin_moe)r )marlin_make_workspacesiluz"Only SiLU activation is supported.)max_blocks_per_smrw1w2w1_scalew2_scale gating_outputrrr9g_idx1g_idx2 sort_indices1 sort_indices2w1_zerosw2_zeros workspacenum_bitsr6inplacerouted_scaling_factor)rr)7sglang.srt.layers.moe.fused_moe_triton.fused_marlin_moer?/sglang.srt.layers.moe.token_dispatcher.standardr +sglang.srt.layers.quantization.marlin_utilsr@r topk_output activationrdevicer+r,r-r.rrrr9r3r4r/r0r7r8r2r6rQrRtodtype) r<r=r>r?r r@rrVoutputrrrfused_experts_none_to_marlinLsr         r\)r<r r=r*r>rrr ) __future__r dataclassesrtypingrrtorch%sglang.srt.layers.moe.moe_runner.baserrrr r sglang.srt.layers.moe.utilsr &sglang.srt.layers.moe.token_dispatcherr r rr'rr)r*r\rrrrs"