o ,i!@sz UddlZddlmZddlZddlmZddlmZddlm Z ddl m Z m Z ddl mZmZe Zdefdd ZeZdefd d Zd edefd dZer[ddlmZejZ        ddejdejdejdejdejdejdBdedededejdBdejdBdejdBdejdBdejdBdejdBdejf d d!Z        ddejdejdejdejdejdejdBdedededejdBdejdBdejdBdejdBdejdBdejdBdejf d"d#Z        ddejdejdejd$ejdejd%ejdBd&ejdBd'ejdBd(ejdBd)ed*ejdBdejdBdedejfd+d,Z        ddejdejdejd$ejdejd%ejdBd&ejdBd'ejdBd(ejdBd)ed*ejdBdejdBdedejfd-d.Z!d$ejd/ejd0ejd1ejd2eddf d3d4Z"d$ejd/ejd0ejd1ejd2eddf d5d6Z#d$ejd/ejd1ejddfd7d8Z$d$ejd/ejd1ejddfd9d:Z% ;dd1ejded?ed@e&ddfdAdBZ' ;dd1ejded?ed@e&ddfdCdDZ( E ;dd1ejd$ejdejd=ed>ed?edFe)d@e&ddfdGdHZ* E ;dd1ejd$ejdejd=ed>ed?edFe)d@e&ddfdIdJZ+da,edBe-dK<defdLdMZ.    ; N  ddOejdPejdQejdRejdSedTejdBdUejdBdVejdBdWe&dXe&dYejdBdZejdBddfd[d\Z/    ; N  ddOejdPejdQejdRejdSedTejdBdUejdBdVejdBdWe&dXe&dYejdBdZejdBddfd]d^Z0dej1fd_ejd`ejdaejdbejdcejdBdejdejfdddeZ2dej1fd_ejd`ejdaejdbejdcejdBdejdejfdfdgZ3ej1fd_ejd`ejdaejdbejdejdejf dhdiZ4ej1fd_ejd`ejdaejdbejdejdejf djdkZ5ej1fd_ejd`ejdaejdbejdejdejf dldmZ6ej1fd_ejd`ejdaejdbejdejdejf dndoZ7dpejdqejdre&dejfdsdtZ8dpejdqejdre&dejfdudvZ9dpejdwejdqejdre&de:ejejff dxdyZ;dpejdwejdqejdre&de:ejejff dzd{Zdpejdqejd|e&d}ejde:ejejff ddZ?dpejdqejd|e&d}ejde:ejejff ddZ@ ddpejd}ejdejdBde:ejejffddZA ddpejd}ejdejdBde:ejejffddZB ddpejd}ejdejdBde:ejejffddZC ddpejd}ejdejdBde:ejejffddZDdpejdwejdqejdre&dede:ejejejff ddZEdpejdwejdqejdre&dede:ejejejff ddZFdpejdqejdre&dede:ejejff ddZGdpejdqejdre&dede:ejejff ddZHdpejdede:ejejffddZIdpejdede:ejejffddZJdpejdede:ejejffddZKdpejdede:ejejffddZLdaMGdddZNeNOdS)N)Callable) OpOverload)current_platform)direct_register_custom_opis_torch_equal_or_newer)rocm_aiter_sparse_attn_indexer#rocm_aiter_sparse_attn_indexer_fakereturncCsddlm}|dduS)Nr find_specaiter)importlib.utilr r rL/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/_aiter_ops.pyis_aiter_founds  rcCs"trtrddlm}|SdS)Nron_gfx9F)ris_rocmIS_AITER_FOUNDvllm.platforms.rocmrrrrris_aiter_found_and_supported!s  rfunccstfdd}|S)zdDecorator that only executes the function if ROCm AITER package is supported on gfx9 archs. cstr |i|SdSN)r)argskwargsrrrwrapper.sz#if_aiter_supported..wrapper) functoolswraps)rrrrrif_aiter_supported)sr)dtypesF hidden_statesw1w2 topk_weighttopk_ids expert_maskactivation_method quant_methoddoweight_stage1w1_scalew2_scalea1_scalea2_scalenum_local_tokens output_dtypecCsRddlm}m}ddlm}||}||}||||||||||| | | | | |dS)Nr)ActivationType QuantType) fused_moe)r.dtype)r r0r1aiter.fused_moer2)r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2 activation quant_typerrr_rocm_aiter_fused_moe_implDs* r7cCs |dur tj||dSt|S)Nr3torch empty_liker!r"r#r$r%r&r'r(r)r*r+r,r-r.r/rrr_rocm_aiter_fused_moe_fakens r= topk_weights fc1_scale fc2_scalefc1_smooth_scalefc2_smooth_scalea16per_tensor_quant_scalec CsBddlm} ddlm}| | }||||||||||| | | |d S)Nr)r0) asm_moe_tkw1)r?r@rArBrCrDr&r5)r r0aiter.fused_moe_bf16_asmrE)r!r"r#r>r%r?r@rArBrCrDr&r'r0rEr5rrr_rocm_aiter_asm_moe_tkw1_impls$  rGc C t|Srr9 r!r"r#r>r%r?r@rArBrCrDr&r'rrr_rocm_aiter_asm_moe_tkw1_fakes rJ topk_indicestoken_expert_indices gating_output renormalizecCs ddlm}||||||dS)Nr) topk_softmax)r rO)r>rKrLrMrNrOrrr_rocm_aiter_topk_softmax_impls  rPcCdSrrr>rKrLrMrNrrr_rocm_aiter_topk_softmax_fakesrScCsddlm}||||dS)Nr) topk_sigmoid)r rT)r>rKrMrTrrr_rocm_aiter_topk_sigmoid_impls rUcCrQrr)r>rKrMrrr_rocm_aiter_topk_sigmoid_fakesrV?correction_biasnum_expert_group topk_group need_renormrouted_scaling_factorc Cs&ddlm}|||||||||dS)Nr)biased_grouped_topk)r r]) rMrXr>r%rYrZr[r\r]rrr$_rocm_aiter_biased_grouped_topk_impls r^cCrQrrrMrXr>r%rYrZr[r\rrr$_rocm_aiter_biased_grouped_topk_fake r`softmax scoring_funcc Cs.|dk}ddlm} | ||||||||dS)Nrbr) grouped_topk)r rd) rMr>r%rYrZr[rcr\ is_softmaxrdrrr_rocm_aiter_grouped_topk_impl s rfcCrQrrrMr>r%rYrZr[rcr\rrr_rocm_aiter_grouped_topk_fake#rarh_AITER_MLA_SUPPORTS_FP8c Cshtdur2zddl}ddlm}||}d|jvod|jvaWtStttt t fy1daYtSwtS)zKCheck if aiter.mla.mla_decode_fwd supports q_scale and kv_scale parameters.Nrmla_decode_fwdq_scalekv_scaleF) riinspect aiter.mlark signature parameters ImportErrorModuleNotFoundErrorAttributeError ValueError TypeError)rnrksigrrr_check_aiter_mla_fp8_support4s&     rxq kv_buffero qo_indptr max_seqlen_qo kv_indptr kv_indiceskv_last_page_lenssm_scale logit_caprlrmc  Cs`ddlm} || d} tr| | d<| | d<| ||ddd|jd||||||fi| dS)Nrrj)rrrlrm)rorkrxviewshape)rzr{r|r}r~rrrrrrlrmrkrrrr_rocm_aiter_mla_decode_fwd_implPs&  rc CrQrr) rzr{r|r}r~rrrrrrlrmrrr_rocm_aiter_mla_decode_fwd_fakewsrABAsBsbiascCsddlm}|||||||S)Nr) gemm_a8w8_CK)r r)rrrrrr/rrrr_rocm_aiter_gemm_a8w8_impls rc Cs,|jd}|jd}tj||||jd}|SNrr3devicerr:emptyr) rrrrrr/mnYrrr_rocm_aiter_gemm_a8w8_fakes  rcCddlm}||||||dSNr)gemm_a8w8_blockscaler8)%aiter.ops.triton.gemm_a8w8_blockscalerrrrrr/rrrr,_rocm_aiter_triton_gemm_a8w8_blockscale_impl rcC,|jd}|jd}tj||||jd}|Srrrrrrr/rrrrrr,_rocm_aiter_triton_gemm_a8w8_blockscale_fake  rcCrr)r rrrrr%_rocm_aiter_gemm_a8w8_blockscale_implrrcCrrrrrrr%_rocm_aiter_gemm_a8w8_blockscale_fakerrxweightvariance_epsiloncCsPddlm}|dkr"|j}|d|d}||||}||S||||S)Nr)rms_normr)r rdimrreshape)rrrrx_original_shaperrr_rocm_aiter_rms_norm_impls     rcCrHrr9rrrrrr_rocm_aiter_rms_norm_fakes rresidualcCs:ddlm}t|}t|}|||||||||fS)Nr)rmsnorm2d_fwd_with_add)r rr:r;)rrrrr residual_outoutrrr'_rocm_aiter_rmsnorm2d_fwd_with_add_impls   rcCst|}t|}||fSrr9)rrrrrrrrr'_rocm_aiter_rmsnorm2d_fwd_with_add_fakes  repsilon quant_dtypec Csxddl}|tjtfvs Jtj|jddtj|jd}tj|j||jd}t|}|j |||||||dd|||fSNrrr)use_model_sensitive_rmsnorm) r r:int8 _FP8_DTYPErrfloat32rr;#rmsnorm2d_fwd_with_add_dynamicquant) rrrrr rocm_aitery_scalerrrrr0_rocm_aiter_rmsnorm_fused_add_dynamic_quant_impl s   rcCsDtj|jddtj|jd}tj|j||jd}t|}|||fSNrrr)r:rrrrr;)rrrrrrrrrrr0_rocm_aiter_rmsnorm_fused_add_dynamic_quant_fake*s  rcCshddl}|tjtfvs Jtj|jddtj|jd}tj|j||jd}|j|||||dd||fSr) r r:rrrrrrrmsnorm2d_fwd_with_dynamicquant)rrrrrrrrrr,_rocm_aiter_rmsnorm_fused_dynamic_quant_impl8s rcCs8tj|jddtj|jd}tj|j||jd}||fSr)r:rrrr)rrrrrrrrr,_rocm_aiter_rmsnorm_fused_dynamic_quant_fakeLsrscalecCsddlm}||||S)Nr)per_tensor_quant_hip)aiter.ops.quantr)rrrrrrr!_rocm_aiter_per_tensor_quant_implXs  rcCs"tj||dtjdtj|jdfS)Nr8rr)r:r;rrrrrrrrr!_rocm_aiter_per_tensor_quant_fakebs rc Csddlm}|tjtfvsJ|j}tj|jt|jd}|dur3tjg|dddRtj|jd}||||ddddd||fS)Nr)dynamic_per_token_scaled_quantrrrF)scale_ub shuffle_scalenum_rowsnum_rows_factor) rrr:rrrrrr)rrrr out_shaperrrr _rocm_aiter_per_token_quant_implls & rcCs@|j}tj|jt|jdtjg|dddRtj|jdfS)Nrrr)rr:rrrr)rrrrrrr _rocm_aiter_per_token_quant_fakes$r group_sizec Cs<ddlm}||||ddd|t|d \\}}}}} || |fSNr)fused_rms_fp8_group_quant)r dtype_quantres1 aiter.ops.triton.fused_fp8_quantrAITER_FP8_DTYPE) rrrrrrx_quantx_quant_scales_resrrr1_rocm_aiter_rmsnorm_with_add_fp8_group_quant_impls  rcCsR|j\}}|||d|f}tj|t|jdtj||jdtj|tj|jdfS)Nrr)rrr:r;rrrr)rrrrrMN scale_shaperrr1_rocm_aiter_rmsnorm_with_add_fp8_group_quant_fakes rc Cs:ddlm}||||ddd|tdd \\}}}}}||fSrr) rrrrrrrrrrrr(_rocm_aiter_rmsnorm_fp8_group_quant_impls  rcCsD|j\}}|||d|f}tj|t|jdtj|tj|jdfS)Nrrr)rrrrrrrrrr(_rocm_aiter_rmsnorm_fp8_group_quant_fakes rcCsD|jd|dks Jdddlm}m}||j}||tdS)Nrrz+Input shape must be divisible by group size)r1 get_hip_quant)r)rr r1r per_1x128 contiguousr)rrr1raiter_per1x128_quantrrr _rocm_aiter_group_fp8_quant_impls rcCsL|j\}}tj||ft|jd}tj|||d|ftj|jd}||fS)Nrrrr:rrrr)rrrrx_fp8out_bsrrr _rocm_aiter_group_fp8_quant_fakes rcCsddlm}||d|tdS)Nr)act_mul_and_fp8_group_quantsilu)r5rr)aiter.ops.triton.activationrr)rrrrrr,_rocm_aiter_act_mul_and_fp8_group_quant_impls rcCsd|j\}}|ddks J|d}tj||ft|jd}tj|||d|ftj|jd}||fS)Nrrrrr)rrrrN_halfrrrrr,_rocm_aiter_act_mul_and_fp8_group_quant_fakes rc"@sjeZdZdZejZejZej Z ej Z ej ZejZejZejZejZejZejZejZejZejZ e!ddZ"e!e#de$fddZ%e!e#de$fddZ&e!e#de$fd d Z'e!e#de$fd d Z(e!e#de$fd dZ)e!e#de$fddZ*e!e#de$fddZ+e!e#de$fddZ,e!e#de$fddZ-e!e#de$fddZ.e!e#de$fddZ/e!e#de$fddZ0e!e#de$fddZ1e!e#de$fdd Z2e!e#de$fd!d"Z3e4e#dd$d%Z5e4de6fd&d'Z7e4de6fd(d)Z8e4de6fd*d+Z9e4de6fd,d-Z:e4de6fd.d/Z;e4de6fd0d1Ze4de6fd6d7Z?e4d8e@jAd9e@jAd:eBde@jAfd;d<ZCe4d8e@jAd=e@jAd9e@jAd:eBdeDe@jAe@jAff d>d?ZEe4d#e@jFfd@e@jAdAe@jAdBe@jAdCe@jAdDe@jAd#BdEe@jGde@jAfdFdGZHe4e@jFfd@e@jAdAe@jAdBe@jAdCe@jAdHeIeJdEe@jGde@jAfdIdJZKe4e@jFfd@e@jAdAe@jAdBe@jAdCe@jAdHeIeJdEe@jGde@jAfdKdLZLe4 # M M N # # # # # #ddOe@jAdPe@jAdQe@jAdRe@jAdSe@jAdTe@jAd#BdUeJdVeJdWe$dXe@jAd#BdYe@jAd#BdZe@jAd#Bd[e@jAd#Bd\e@jAd#BdEe@jGd#Bde@jAf d]d^ZMe4 # # # # N # # MddOe@jAdPe@jAdQe@jAd_e@jAdSe@jAd`e@jAd#Bdae@jAd#Bdbe@jAd#Bdce@jAd#Bdde$dee@jAd#BdTe@jAd#BdUeJde@jAfdfdgZNe4d_e@jAdhe@jAdie@jAdje@jAdke$deDe@jAdlff dmdnZOe4d_e@jAdhe@jAdie@jAdje@jAdke$deDe@jAdlff dodpZPe4 qddje@jAdre@jAd_e@jAdSe@jAdseJdteJdue$dveBdd#fdwdxZQe4 y qddje@jAd_e@jAdSe@jAdseJdteJdue$dzeRdveBdd#fd{d|ZSe4 # # # } # #dd~e@jAde@jAde@jAdeBde@jAdeJde@jAd#Bde@jAd#Bde@jAd#BdeBde@jAd#Bde@jAd#BfddZTe4 #dd8e@jAde@jGde@jAd#BdeDe@jAe@jAffddZUe4 #dd8e@jAde@jGde@jAd#BdeDe@jAe@jAffddZVe4e@jWd#fd8e@jAd9e@jAde@jAde@jGd#Bde@jAd#Bde@jAf ddZXe4de@jAde@jAde@jAde@jAdeJdeJde$fddZYe4 N N #dde@jAde@jAde@jAde@jAde$d#Bde$d#Bde@jAd#Bde@jAfddZZe4dd#e@jWd#d#dNd#fde@jAde@jAde@jAdeJdDe@jAd#Bde@jGd#BdeJd#Bde@jAd#Bde$d#Bde[d#Bde@jAfddZ\e4 dde@jAdeJdeDe@jAdlffddZ]e4deJdeJde$fddZ^e4deJdeJde$fddZ_e4 dde@jAdeDeJeJfde@jAfddZ`e4ddde@jAdeDeJeJfdeDe@jAdlffddÄZad#S)rocm_aiter_opsa ROCm AITER operations wrapper for AMD GPU acceleration in vLLM. This class centralizes the import and registration of AITER ops, and provides a unified interface for checking if AITER is enabled. Operations are only available on supported gfx9 architectures when aiter is installed. The class uses environment variables to control which features are enabled, allowing fine-grained control over which AITER optimizations are used. Environment Variables: VLLM_ROCM_USE_AITER: Main toggle for all AITER operations. VLLM_ROCM_USE_AITER_LINEAR: Controls GEMM and quantization ops. VLLM_ROCM_USE_AITER_RMSNORM: Controls RMSNorm operations. VLLM_ROCM_USE_AITER_MOE: Controls MoE (Mixture of Experts) ops. VLLM_ROCM_USE_AITER_MLA: Controls MLA (Multi-head Latent Attention) ops. VLLM_ROCM_USE_AITER_MHA: Controls MHA ops including flash_attn_varlen. VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION: Controls Triton unified attention. VLLM_ROCM_USE_AITER_FP8BMM: Controls FP8 batched matrix multiply. VLLM_ROCM_USE_AITER_FP4_ASM_GEMM: Controls FP4 assembly GEMM. VLLM_ROCM_USE_AITER_TRITON_ROPE: Controls Triton rotary embeddings. VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: Controls shared expert fusion. VLLM_ROCM_USE_AITER_TRITON_GEMM: Controls Triton unquantized GEMM. Note: The environment variables are assigned when the module is imported, so you can't change the environment variables after the module is imported. This is done out of performance consideration. Accessing environment variables is expensive as described in issue https://github.com/vllm-project/vllm/issues/17067 so we don't want to do it repeatedly, especially in the hot path (the forward pass). You can call the refresh_env_variables() function to reload the env variables after monkey patching the env variables in the unit test. Check Functions: All check functions (is_*_enabled) are decorated with @if_aiter_supported, which verifies: (1) platform is ROCm, (2) device arch is gfx9, and (3) aiter library is installed. The check function then also verifies the corresponding environment variable is enabled. i.e. ___ is_enabled() == current_platform.is_rocm() and | checked by current_platform.is_on_gfx9() and | @if_aiter_supported IS_AITER_FOUND and _______________| cls._AITER_ENABLED -----> Check by the logic in `is_enabled()` Example: from vllm._aiter_ops import rocm_aiter_ops # Check if aiter is enabled before using operations if rocm_aiter_ops.is_enabled(): result = rocm_aiter_ops.rms_norm(x, weight, epsilon) Operations: - RMS normalization: rms_norm, rms_norm2d_with_add - GEMM operations: gemm_a8w8, gemm_a8w8_blockscale - Fused MoE: fused_moe, asm_moe_tkw1 - Routing: topk_softmax, biased_grouped_topk, grouped_topk - MLA decode: mla_decode_fwd - Quantization: per_tensor_quant, per_token_quant, group_fp8_quant - Triton ops: triton_rotary_embed, triton_fp8_bmm, triton_gemm_a8w8_blockscale cCsttj|_tj|_tj|_tj|_tj |_ tj |_ tj |_tj|_tj|_tj|_tj|_tj|_tj|_tj|_dS)aT Since the environment variables are assigned when the module is imported, This is a helper function to reload all the env variables from the environment variables. for example, after monkey patching the env variables in the unit test, you can call this function to reload the env variables. N)envsVLLM_ROCM_USE_AITER_AITER_ENABLEDVLLM_ROCM_USE_AITER_LINEAR_LINEAR_ENABLEDVLLM_ROCM_USE_AITER_RMSNORM_RMSNORM_ENABLEDVLLM_ROCM_USE_AITER_MOE _FMOE_ENABLEDVLLM_ROCM_USE_AITER_MLA _MLA_ENABLEDVLLM_ROCM_USE_AITER_MHA _MHA_ENABLED!VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT_SHUFFLE_KV_CACHE_ENABLED%VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION_TRITON_UNIFIED_ATTN_ENABLEDVLLM_ROCM_USE_AITER_FP8BMM_FP8BMM_ENABLEDVLLM_ROCM_USE_AITER_FP4BMM_FP4BMM_ENABLED VLLM_ROCM_USE_AITER_FP4_ASM_GEMM_FP4_GEMM_DYNAMIC_QUANT_ASMVLLM_ROCM_USE_AITER_TRITON_ROPE_TRITON_ROTARY_EMBED)VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS_MOE_SHARED_EXPERTS_ENABLEDVLLM_ROCM_USE_AITER_TRITON_GEMM_TRITON_UNQUANT_GEMMclsrrrrefresh_env_variablesos  z$rocm_aiter_ops.refresh_env_variablesr cCs|jSr)rrrrr is_enabledszrocm_aiter_ops.is_enabledcC |jo|jSr)rrrrrris_linear_enabled z rocm_aiter_ops.is_linear_enabledcCs|Sr)rrrrris_linear_fp8_enabledsz$rocm_aiter_ops.is_linear_fp8_enabledcCrr)rrrrrris_rmsnorm_enabledrz!rocm_aiter_ops.is_rmsnorm_enabledcCrr)rrrrrris_fused_moe_enabledrz#rocm_aiter_ops.is_fused_moe_enabledcCs|o|jSr)r rrrrr$is_fusion_moe_shared_experts_enabledsz3rocm_aiter_ops.is_fusion_moe_shared_experts_enabledcCrr)rrrrrris_mla_enabledrzrocm_aiter_ops.is_mla_enabledcCrr)rrrrrris_mha_enabledrzrocm_aiter_ops.is_mha_enabledcCrr)rrrrrris_shuffle_kv_cache_enabledrz*rocm_aiter_ops.is_shuffle_kv_cache_enabledcCrr)rr rrrris_triton_unified_attn_enabledrz-rocm_aiter_ops.is_triton_unified_attn_enabledcCrr)rr rrrris_fp8bmm_enabledrz rocm_aiter_ops.is_fp8bmm_enabledcCrr)rrrrrris_fp4bmm_enabledrz rocm_aiter_ops.is_fp4bmm_enabledcCrr)rrrrrr%is_asm_fp4_gemm_dynamic_quant_enabledrz4rocm_aiter_ops.is_asm_fp4_gemm_dynamic_quant_enabledcCrr)rrrrrris_triton_rotary_embed_enabledrz-rocm_aiter_ops.is_triton_rotary_embed_enabledcCrr)rrrrrris_triton_gemm_enabledrz%rocm_aiter_ops.is_triton_gemm_enabledNcCststdr tntjjf}tdtgtt j dtdt gt t j dtdt gdtt j dtdtdd gtt j dtd tdd gtt j dtd tdd gtt j dtd tdgt|dtdtgtt j dtdttdtdttdtdttdtdtt t j dtdt!t"t j dtdt#t$t j dtdt%t&dtdt't(dtdt)t*dtdt+t,dtdt-gt.t j dtdt/t0t j dtdt1d gt2t j dd!adSdS)"Nz2.7.0rocm_aiter_asm_moe_tkw1)op_nameop_func mutates_args fake_impl dispatch_keyrocm_aiter_fused_moerocm_aiter_topk_softmax)r>rKrLrocm_aiter_topk_sigmoidr>rKrocm_aiter_biased_grouped_topkr%rocm_aiter_grouped_topkrocm_aiter_mla_decode_fwdr|)r,r-r.r/tagsrocm_aiter_gemm_a8w8&rocm_aiter_triton_gemm_a8w8_blockscale)r,r-r/rocm_aiter_gemm_a8w8_blockscalerocm_aiter_rms_norm!rocm_aiter_rmsnorm2d_fwd_with_add)r,r-r/r0&rocm_aiter_rmsnorm_fused_dynamic_quant*rocm_aiter_rmsnorm_fused_add_dynamic_quant"rocm_aiter_rmsnorm_fp8_group_quant+rocm_aiter_rmsnorm_with_add_fp8_group_quant&rocm_aiter_act_mul_and_fp8_group_quantrocm_aiter_group_fp8_quantrocm_aiter_per_tensor_quantrocm_aiter_per_token_quantrtopk_indices_bufferT)3_OPS_REGISTEREDrtupler:Tagneeds_fixed_stride_orderrrGrJrr0r7r=rPrSrUrVr^r`rfrhrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr)r7rrrregister_ops_oncesz rocm_aiter_ops.register_ops_oncecC tjjjjSr)r:opsvllmr<defaultrrrrget_rmsnorm_fused_add_opv z'rocm_aiter_ops.get_rmsnorm_fused_add_opcCrKr)r:rLrMr;rNrrrrget_rmsnorm_opzrPzrocm_aiter_ops.get_rmsnorm_opcCrKr)r:rLrMr>rNrrrr&get_rmsnorm_fused_add_dynamic_quant_op~rPz5rocm_aiter_ops.get_rmsnorm_fused_add_dynamic_quant_opcCrKr)r:rLrMr=rNrrrr"get_rmsnorm_fused_dynamic_quant_oprPz1rocm_aiter_ops.get_rmsnorm_fused_dynamic_quant_opcCrKr)r:rLrMr?rNrrrr get_rmsnorm_group_fused_quant_oprPz/rocm_aiter_ops.get_rmsnorm_group_fused_quant_opcCrKr)r:rLrMr@rNrrrr$get_rmsnorm_group_add_fused_quant_oprPz3rocm_aiter_ops.get_rmsnorm_group_add_fused_quant_opcCrKr)r:rLrMrDrNrrrrget_per_token_quant_oprPz%rocm_aiter_ops.get_per_token_quant_opcCrKr)r:rLrMrBrNrrrrget_group_quant_oprPz!rocm_aiter_ops.get_group_quant_opcCrKr)r:rLrMrArNrrrr$get_act_mul_fused_fp8_group_quant_oprPz3rocm_aiter_ops.get_act_mul_fused_fp8_group_quant_oprrrcCtjj|||Sr)r:rLrMr;rrrrrszrocm_aiter_ops.rms_normrcCstjj||||Sr)r:rLrMr<)rrrrrrrrms_norm2d_with_addsz"rocm_aiter_ops.rms_norm2d_with_addrrrrrr/cCstjj||||||Sr)r:rLrMr8)rrrrrr/rrr gemm_a8w8s zrocm_aiter_ops.gemm_a8w8 block_sizecCtjj|||||Sr)r:rLrMr9rrrrr\r/rrrtriton_gemm_a8w8_blockscale z*rocm_aiter_ops.triton_gemm_a8w8_blockscalecCr]r)r:rLrMr:r^rrrrr`z#rocm_aiter_ops.gemm_a8w8_blockscalerFr!r"r#r$r%r&r'r(r)r*r+r,r-r.cCs*tjj|||||||||| | | | | |Sr)r:rLrMr1r<rrrr2s"zrocm_aiter_ops.fused_moer>r?r@rArBrCrDc Cs&tjj|||||||||| | | | Sr)r:rLrMr+rIrrrrEszrocm_aiter_ops.asm_moe_tkw1rKrLrMrN.cCstjj|||||||fSr)r:rLrMr2rRrrrrOs zrocm_aiter_ops.topk_softmaxcCstjj|||||fSr)r:rLrMr3rRrrrrT!szrocm_aiter_ops.topk_sigmoidrWrXrYrZr[r\c C tjj||||||||dSr)r:rLrMr4r_rrrr]. z"rocm_aiter_ops.biased_grouped_topkrbrcc Crar)r:rLrMr5rgrrrrdDrbzrocm_aiter_ops.grouped_topkryrzr{r|rr}r~rrrrrlrmc Cs<tjjj||ddd|jd|||||||| | | d dS)Nrr)rrrlrm)r:rLrMr6rr) rzr{r|rr}r~rrrrrlrmrrrrkZs zrocm_aiter_ops.mla_decode_fwdrrcCrYr)r:rLrMrCrrrrper_tensor_quantxzrocm_aiter_ops.per_tensor_quantcCrYr)r:rLrMrDrrrrper_token_quantrdzrocm_aiter_ops.per_token_quant weight_scale out_dtypex_scalesc Csnddlm}ddlm}|dur||\}}n|}|}tj|jd|jd|j|d} |||||j|| | S)Nr) gemm_afp4wfp4)dynamic_mxfp4_quant)rr3) aiter.ops.triton.gemm_afp4wfp4riaiter.ops.triton.quantrjr:rrrT) rrrfrgrhrirjx_qx_syrrrtriton_fp4_gemm_dynamic_qaunts  z,rocm_aiter_ops.triton_fp4_gemm_dynamic_qaunt positionsquerykey cos_sin_cache head_size rotary_dim is_neox_stylec Csddlm}|}|jddd\} } |j} |j} |rdnd} ||d|}||d|}|dd|f}|dd|f}|j|jdd}|||| | || dd d || }|| }dS) Nr)(rope_cached_thd_positions_2c_fwd_inplacerr)rr.TF)reuse_freqs_front_part nope_first)aiter.ops.triton.roperynumelchunkrr)rrrsrtrurvrwrxry num_tokenscossin query_shape key_shape rotate_stylequery_key_rrrtriton_rotary_embeds.   z"rocm_aiter_ops.triton_rotary_embedXWw_scaler transpose_bmprequantrc Cs"ddlm}||||||||dS)Nr)batched_gemm_a16wfp4)rprrr)%aiter.ops.triton.batched_gemm_a16wfp4r)rrrrrrrrrrrrs z#rocm_aiter_ops.batched_gemm_a16wfp4WQrr3splitKYQconfigc Cs(ddlm} | |||||||||| d S)Nr)Gbatched_gemm_a8w8_a_per_token_group_prequant_w_per_batched_tensor_quant)rrr3rrrr)Xaiter.ops.triton.batched_gemm_a8w8_a_per_token_group_prequant_w_per_batched_tensor_quantr) rrrrrr3rrrraiter_triton_fp8_bmmrrrtriton_fp8_bmms zrocm_aiter_ops.triton_fp8_bmminput_2dcCs |dksJdtjj||S)NrzGroup size must be 128)r:rLrMrB)rrrrrgroup_fp8_quantszrocm_aiter_ops.group_fp8_quantrkcC ||fdvS)N) ) )i@)i i)r)r)ir)ir)r)rrr)rrrrrrrris_triton_gemm_w8a8_tunedrPz(rocm_aiter_ops.is_triton_gemm_w8a8_tunedcCr)N))rr)ir)@i)ir)ir)rr)i r)i(r)rr)rp)rr)iHrr)rr)ir)rr)rr)8r)rr)rirrrrr&is_triton_gemm_afp4wfp4_presh_ws_tunedrPz5rocm_aiter_ops.is_triton_gemm_afp4wfp4_presh_ws_tunedrtensorlayoutcCsddlm}|||dS)Nrshuffle_weightr)aiter.ops.shuffler)selfrrrrrrr+s  zrocm_aiter_ops.shuffle_weightrtensorscs$ddlmtfdd|DS)a Applies shuffle_weight function from AITER to each input tensor and returns them. Rearranges (shuffles) the input tensor/s into a specified block layout for optimized computation. Args: *tensors: Variable number of torch.Tensor objects. layout: A pair of integers specifying the block sizes used to divide the tensors during shuffling. Default is (16, 16). Returns: A Tuple of shuffled tensors. rrc3s|] }|dVqdS)rNr).0rrrrr Hsz1rocm_aiter_ops.shuffle_weights..)rrrG)rrrrrshuffle_weights3s zrocm_aiter_ops.shuffle_weights)r N NrrFNNNNNNNNNNFNNrrWrbrW)NNNryNNr)FFN)r)r)b__name__ __module__ __qualname____doc__rrrrrrrrrrrrrrrr r r r r rrrrrrrrr classmethodrrboolrrrrr r!r"r#r$r%r&r'r(r)r* staticmethodrJrrOrQrRrSrTrUrVrWrXr:TensorfloatrrGrZfloat16r3r[listintr_rr2rErOrTr]strrdrkrcrebfloat16rqrrdictrrrrrrrrrrrs> #               #                       #          rrrrr)NNNrWryNNr)Prcollections.abcrr: torch._opsr vllm.envsrvllm.platformsrvllm.utils.torch_utilsrr+vllm.v1.attention.ops.rocm_aiter_mla_sparserr fp8_dtyperrrrrrr r fp8rrrr3r7r=rGrJrPrSrUrVrr^r`rrfrhri__annotations__rxrrrrrrrrrrrrGrrrrrrrrrrrrrrrrrrrFrrJrrrrsF              0            +                  "     -                                      4