o پim<@sp dZ ddlZddlmZmZddlZddlZddlmZddl m Z ddl m Z ddl mmZz ddlmZdZWneyKd Zd d ZYnwd Zd ZdZd Zd Zd ZdZdZdZdZdZd Z d Z!dZ"d Z#dZ$d Z%ee%Z&dZ'd Z(d Z)ddde*de*de*fddZ+ddde*de*de*fddZ,ej-dej.dej/dej0dej1e*d ej1e*d!ej/d"ej/d#ej/d$ej/d%ej/d&ej/d'ej/d(ej/d)ej/d*ej/d+ej1e2d,ej1e2d-ej1e2d.ej1e*d/ej1e*d0ej1e*d1ej1e*d2ej1e*d3ej1e*d4ej1e3d5ej1e3d6ej1e3f6d7d8Z4ej-dej.dej/dej0dej1e*d ej1e*d!ej/d"ej/d#ej/d$ej/d%ej/d&ej/d'ej/d(ej/d)ej/d*ej/d+ej1e2d,ej1e2d-ej1e2d.ej1e*d/ej1e*d0ej1e*d1ej1e*d2ej1e*d3ej1e*d4ej1e3d5ej1e3d6ej1e3f6d9d:Z5ej6dej/d!ej/d"ej/d#ej/d$ej/d%ej/d&ej/d'ej/d(ej/d)ej/d*ej/d+ej1e2d,ej1e2d-ej1e2d.ej1e*d/ej1e*d0ej1e*d1ej1e*d2ej1e*d3ej1e*d4ej1e3d5ej1e3d6ej1e3d;ej7f0dd?Z9ej:d/e*d0e*d1e*d.e*d2e*d3e*d@ej;d-e2d5e3fdAdBZej-dej.dej/dej0d ej1e*d$ej/d%ej/d&ej/d"ej/d'ej/d!ej/d#ej/d(ej/d)ej/d+ej1e2d,ej1e2d-ej1e2d1ej1e*d.ej1e*d5ej1e3f&dIdJZ?ej-dej.dej/dej0d ej1e*d$ej/d%ej/d&ej/d"ej/d'ej/d!ej/d#ej/d(ej/d)ej/d+ej1e2d,ej1e2d-ej1e2d1ej1e*d.ej1e*d5ej1e3f&dKdLZ@ej6d*ej/d$ej/d%ej/d&ej/d"ej/d'ej/d!ej/d#ej/dej/d)ej/d(ej/d+ej1e2d,ej1e2d-ej1e2d/ej1e*d0ej1e*d1ej1e*d.ej1e*d2ej1e*d3ej1e*d4ej1e3d5ej1e3d;ej7f.dMdNZAej6d*ej/d$ej/d%ej/d&ej/d"ej/d'ej/d!ej/d#ej/dej/d)ej/d(ej/d+ej1e2d,ej1e2d-ej1e2d/ej1e*d0ej1e*d1ej1e*d.ej1e*d2ej1e*d3ej1e*d4ej1e3d5ej1e3d;ej7f.dOdPZBe   ded$ej/d%ej/d&ej/dEej/d!ej/d"ej/d#ej/d'ej/d-ee2dFeej/d5e3deej/ej/ffdQdRZCej-dej/dSej/dej1e*d ej1e*dTej1e*d!ej/d"ej/d#ej/d$ej/d%ej/d&ej/d'ej/d(ej/d)ej/d*ej/d+ej1e2d,ej1e2d-ej1e2d.ej1e*d/ej1e*d0ej1e*d1ej1e*d2ej1e*d3ej1e*d4ej1e3d5ej1e3d6ej1e3dUej1e3dVej1e3f:dWdXZDej6dej/dSej/d!ej/d"ej/d#ej/d$ej/d%ej/d&ej/d'ej/d(ej/d)ej/d*ej/d+ej1e2d,ej1e2d-ej1e2d.ej1e*d/ej1e*d0ej1e*d1ej1e*d2ej1e*d3ej1e*dTej1e*dej1e*d4ej1e3d5ej1e3d6ej1e3dUej1e3dVej1e3d;ej7f:dYdZZEej:d/e*d0e*d1e*d.e*d2e*d3e*d[e*d\e*dUe3dVe3d-e2d5e3dTe*de*fd]d^ZFe     dfd$ej/d%ej/d&ej/d_ej/d`ej/d!ej/d"ej/d#ej/d'ej/d-ee2dFeej/daeej/dUe3d5e3deej/ej/ffdbdcZGdS)ga3 Copyright (c) 2025 by FlashInfer team. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. N)OptionalTuple)cpasync) from_dlpack)flashinfer_apiTFcCs|S)N)funcrrI/home/ubuntu/.local/lib/python3.10/site-packages/flashinfer/gdn_decode.pyr;sr $ batch_sizeseq_lenreturncCsdS)zSelect vec_size for MTP kernel. Always use vec_size=4 (32 threads per group = full warp, 4 groups per block). Full warp shuffle is more efficient and achieves >= 1.0x speedup vs Triton. rrrrrrr get_vec_size_mtpisrcCs4|dkrdS|dkr dS|dkrdS|dkrdSdS)zSelect optimal TILE_V for MTP kernel based on batch size and sequence length. With vec_size=4, num_groups=4, rows_per_group = tile_v / 4. Tuned via grid search for optimal performance. r rr rr@rrrrr get_tile_v_mtprsrtiled_copy_load h0_sourcesmem_layout_stagedvec_size num_v_tilesA_logadt_biasqkvbo h0_indices cu_seqlens softplus_betasoftplus_thresholdscaleHVBTHKVuse_initial_state use_qk_l2norm is_varlenca Cstj\}}}|d}tj}tj|}tj\}}}|t} |t}!|t}"| |}#| |}$|$||}%d}&t||$}'t||#|&|$f}(t||$})t| |#|&|$f}*tj }+|+ tj|d},|+ tj t |fd}-|+ tjt |fd}.ttj |fddtj}/ttj |fddtj}0ttj |fddtj}1ttj |fddtj }2ttj |fddtj }3ttj |fddtj }4||}5tj|| ddf}6t|dttf| ddf}7t|6ttfd }8||}9|!|"}:ttd|"};t|:|:|;D]-}<|<|:t}=|8dd||,dd|=f}?|9|>}@|9|?}At||@|Atjqt|ddd|f|#|&|%|f}Bt| ddd|f|#|&|%|f}Ct|B|2t|C|3t|D]}Dt|2|D|0|D<t|3|D|/|D<qkt| ddd|f|#|&|$|f}Et|E|4t|D]}Dt|4|D|.|5|D<qtjd }Fd }G|dkr |(|)}H||H}Id }J|I|krtj|Id d }Ktd |K}Lttj|Ld d }Mttd ||M}Jn|H}Jtj|'d d  |J}Nd d tj|* d d }Gtj|Nd d }Ftj|Fd}Ftj|Gd}G|rd }Od }Pt|D]}D|O|0|D|0|D7}O|P|/|D|/|D7}Pq%dD]}Q|Otjj |O|Qddd7}O|Ptjj |P|Qddd7}Pq?tj!|Odd d }Rtj!|Pdd d }St|D]}D|0|D|R|0|D<|/|D|S|/|D<qrt|D] }D|0|D||0|D<q|:|"}Tt|:|TD] }<|<|:t}=tj"dtj|<|;}U|U|Tkr|U|:t}V|8dd|Uf}W|,dd|Vf}X|9|W}@|9|X}At||@|AtjtdtdD]}Y|d}Zd }[t|,d|df|Y|Z||=f}\t|\|1t|D]}D|1|D|F|1|D<|[|1|D|/|D7}[qdD]}Q|[tjj |[|Qddd7}[q(|.|r)#cutearch thread_idxwarp_idxmake_warp_uniform block_idxNUM_BLOCKS_PER_STATEcutlassFloat32utils SmemAllocatorallocate_tensorBFloat16 make_layoutmake_rmem_tensorbarrier local_tileTILE_VTILE_K get_slicemin NUM_STAGESrange partition_S partition_Dcopycp_async_commit_group autovec_copyrange_constexprexplog shuffle_syncshuffle_sync_bflyrsqrtcp_async_wait_group)arrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5tidx_lane_idrIrK batch_idx batch_innernum_v_tiles_per_blocki_ni_hvi_hi_tr_A_logr_a r_dt_biasr_bsmemsDatasOutputsVr_kr_qr_hr_q_bf16r_k_bf16r_v_bf16k_start gSrc_batchgDstgSrc thr_copy_load start_v_tilesprefetch_countv_tilesstage gSrc_tile sData_stagethr_gSrc thr_sDataq_tilek_tileiv_tiler_gr_betaxbeta_x softplus_x exp_beta_x log_input log_result r_g_valuesum_qsum_krB inv_norm_q inv_norm_k end_v_tiles next_v_tiles next_stage gSrc_next sData_nextrow row_offsetsum_hk sData_tilev_newsum_hq gDst_tileo_idxrrr *gdn_decode_kernel_small_batch_pretransposes<                               .rc\ CsPtj\}}}|d}tj}tj|}tj\}}}||} ||}!|!||}"d}#t||!}$t|| |#|!f}%t||!}&t| | |#|!f}'tj }(|( tj|d})|( tj t |fd}*|( tjt |fd}+t tj |fddtj},t tj |fddtj}-t tj |fddtj}.t tj |fddtj }/t tj |fddtj }0t tj |fddtj }1||}2tj||ddf}3t|dttf|ddf}4t|3ttfd }5||}6ttd|}7t|7D]*}8|8t}9|5dd|8f}:|)dd|9f};|6|:}<|6|;}=t||<|=tjqt|ddd|f| |#|"|f}>t| ddd|f| |#|"|f}?t|>|/t|?|0t|D]}@t|/|@|-|@<t|0|@|,|@<qUt| ddd|f| |#|!|f}At|A|1t|D]}@t|1|@|+|2|@<qtjd }Bd }C|dkr|%|&}D||D}Ed }F|E|krtj|Ed d }Gtd |G}Httj|Hd d }Ittd ||I}Fn|D}Ftj|$d d  |F}Jd d tj|' d d }Ctj|Jd d }Btj|Bd}Btj|Cd}C|rpd }Kd }Lt|D]}@|K|-|@|-|@7}K|L|,|@|,|@7}LqdD]}M|Ktjj|K|Mddd7}K|Ltjj|L|Mddd7}Lq)tj |Kdd d }Ntj |Ldd d }Ot|D]}@|-|@|N|-|@<|,|@|O|,|@<q\t|D] }@|-|@||-|@<qut|D] }8|8t}9tj!dtj|8|7}P|P|kr|Pt}Q|5dd|Pf}R|)dd|Qf}S|6|R}<|6|S}=t||<|=tjtdtdD]}T|d}Ud }Vt|)d|df|T|U||9f}Wt|W|.t|D]}@|.|@|B|.|@<|V|.|@|,|@7}VqdD]}M|Vtjj|V|Mddd7}Vq |+|8t|T|U|V}X|X|C}Xd }Yt|D]}@|.|@|,|@|X7<|Y|.|@|-|@7}Yq0t|4dd|dfd|T|U||8f}Zt|.|ZdD]}M|Ytjj|Y|Mddd7}Yqc|8t|T|U}[|dkr|[|krt |Y|*|[<qϐqtj||kr|*|| | |#|!|f<dSdSr6)"rFrGrHrIrJrKrMrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrh)\rrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5rirjrkrIrlrorprqrrrsrtrurvrwrxryrzr{r|r}r~rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrBrrrrrrrrrrrrrrrrr (gdn_decode_kernel_big_batch_pretransposes2                              . rstreamc#Cs8|jjd|jjd|jjd}}}tjtjtjjdtj dd}tj ddd }t d }t |||}t |t }|||d d d td } tj t ttftdt tfd }!d t ttd |d|d }"t|||!| |||||||||| | | | | |||||||||j|tddftddg|"|ddS)z>Launch original pipelined kernel for small batch pretranspose.rrr  cache_moder num_bits_per_copyrrrrr8rrrrgridblockrwrN)layoutshaperFmake_copy_atomr CopyG2SOp LoadCacheModeGLOBALrMrNrSmake_tiled_copy_tvceil_divrWrXr[rlaunchrL NUM_THREADS#rr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5rrv_dimk_dim copy_atom thread_layout val_layoutrrrr smem_bytesrrr .run_gdn_decode_kernel_small_batch_pretransposest      $  rc#Cs4|jjd|jjd|jjd}}}tjtjtjjdtj dd}tj ddd }t d }t |||}t |t }|||d d d td } tj t ttftdt tfd }!d t ttd |d|d }"t|||!| |||||||||| | | | | |||||||||j|ddftddg|"|ddS)Nrrr rr rrrr8rrrrr)rrrFrrrrrrMrNrSrrrWrXr[rrrrrrr ,run_gdn_decode_kernel_big_batch_pretranspose st       $ rdtypec CiS)zECache compiled kernel for given configuration (pretranspose version).r r.r/r0r-r1r2rr,r4rrr _get_compiled_decode_kernel~rc Cr)zECache compiled kernel for given configuration (nontranspose version).rrrrr (_get_compiled_decode_kernel_nontransposerrstateoutputc 'Csf|j\} } } }| dksJd| |j\}}}}|j| |||fks6Jd| d|d|d|d|j |dksAJd ||dksLJd ||td ks\Jd td ||jtjtjfvsmJd|j|jtjks{Jd|j|jtjksJd|j|dur|d}| du}|r| jn|j}| durtj| | ||ftj|jd} | | |||}| | | ||||j|| f }t |}d|vs|dj|jkrtj| tj |jd|d<tj| dtj |jd|d<|d}|d}d|vrvt tj j}t|dd}t|dd}t|dd}t|dd}t|dd}t|dd}t|dd} t|dd}!t| dd}"t|dd}#t|dd}$| dkrNt}%nt}%tj|%||||||| |!|"|#|$f dd||| | | ||d| d|dd}&|&|d<n|d}&t tj j}|d||||||||| ||| |s|| | |||| j|kr| |} | |fS) a`Gated Delta Rule Decode kernel for single-token generation. This implements the decode phase of gated delta rule linear attention, processing one token at a time and updating the recurrent state. Args: q (torch.Tensor): Current query of shape ``[B, 1, H, K]``. Must be float16/bfloat16. k (torch.Tensor): Current key of shape ``[B, 1, H, K]``. Must be float16/bfloat16. v (torch.Tensor): Current value of shape ``[B, 1, HV, V]``. Must be float16/bfloat16. state (torch.Tensor): Current state of shape ``[B, HV, V, K]`` (v-major layout). Must be float32. Will be updated in-place. A_log (torch.Tensor): Log decay parameter of shape ``[HV]``. Must be float32. a (torch.Tensor): Input-dependent decay of shape ``[B, 1, HV]``. Must be float16/bfloat16. dt_bias (torch.Tensor): Decay bias of shape ``[HV]``. Must be bfloat16 or float32. b (torch.Tensor): Update gate (beta) input of shape ``[B, 1, HV]``. Must be float16/bfloat16. scale (Optional[float]): Scale factor for queries. If None, defaults to ``1 / sqrt(K)``. output (Optional[torch.Tensor]): Pre-allocated output tensor of shape ``[B, 1, HV, V]``. If None, will be allocated automatically. use_qk_l2norm (bool): Whether to apply L2 normalization to q and k. Default: ``True``. Returns: Tuple[torch.Tensor, torch.Tensor]: - output: Output tensor of shape ``[B, 1, HV, V]`` - state: Updated state tensor of shape ``[B, HV, V, K]`` Note: - Requires SM90 (Hopper) architecture - State is updated in-place - K and V must be multiples of 4 for vectorized loads - State layout is v-major: [B, HV, V, K] r Decode only supports T=1, got T=Expected state shape [B=, HV=, V=, K=], got r K must be at least 128, got K=V must be at least 128, got V=rV must be divisible by ( to prevent out-of-bounds access, got V= q must be float16/bfloat16, got state must be float32, got A_log must be float32, got N࿩rdevicer(r)compiledr assumed_alignrr=4@TF--enable-tvm-ffi)r*r+r,r-r.r/r0r1r2r3r4r5roptions)rrWrtorchfloat16bfloat16float32zerosrreshaperint32cudaCUstreamcurrent_stream cuda_streamrrrrFcompile is_contiguouscopy_to)'r#r$r%rr r!r"r&r,rr4r.r/r0r1rjr-r2output_provided target_dtyper cache_keycacher(r)rh0_source_tensor A_log_tensora_tensordt_bias_tensorq_tensork_tensorv_tensorb_tensoro_tensorh0_indices_tensorcu_seqlens_tensorrun_funcrrrr $gated_delta_rule_decode_pretransposes9"                   r ci Cstj\}}}|d}tj}tj|}tj\}}}d}t|}d|}t|}|t}|t}|t}||}||} ||}!|!||}"| | }#|#dkr||}$||}%||}&|&|%}'t j }(|( t j |d})tjtfdd}*|( t j |*d}+tjtfdd},tjtfdd}-|( t j |,d}.|( t j |-d}/|tkrt || d|"|f|.|<t || d|"|f|/|<|#||!}0||0ddf}1t|1ttfd}2||}3t td |}4t|4D].}5||5}6|5t}7|2dd|6f}8|)dd|7f}9|3|8}:|3|9};t||:|;tjqt | |!}t || d|!f}?d }@d }A|dkr|>|=}B| |B}Cd }D|C|krttj|Cd d }Et d |E}Ft tj|Fd d }Gt t d | |G}Dn|B}Dtj|r?r@rA)r rrErunrollrr r)!rFrGrHrIrJrKTILE_V_SMALL_NT TILE_K_NTNUM_BLOCKS_PER_STATE_SMALL_NTrMrOrPrQrNrSrVrYrZ NUM_STAGES_NTr\r]r^r_r`rcrdrerUrfrgrhrRrb)irrrrr#r$r%r!r&r r"r'r(r*r+r,r0r-r4rirj in_warp_tidrIrKNUM_WARPS_SMALLV_PER_WARP_SMALLROWS_PER_ITER_SMALLNUM_K_ITERS_SMALLrlrmrn start_v_tilerorprqpool_idxk_localv_localv_basev_idxrwrx smem_o_layoutsmem_o smem_k_layout smem_q_layoutsKsQflat_idxrrrr v_tile_offsetrrrrrrrsrurtrvrrrrrrrrr sum_q_partial sum_k_partialq_valk_valrBrr local_sum_q local_sum_knext_v_tile_offset next_v_tilerrrv_globalr_vrk_iterk_basek_idxh_valr_k_valrrh_oldr_q_valh_new v_global_outflat_tidk_writev_writev_global_writerrr *gdn_decode_kernel_small_batch_nontransposeYsr                                          wr=c_ CsJtj\}}}|d}tj}tj|}tj\}}}||}||}|||}| |}|dkr|t}|t}|t}||}tj } | tj |d}!tj t fdd}"| tj |"d}#tj tfdd}$tj tfdd}%| tj |$d}&| tj |%d}'|tkrt ||d||f|&|<t ||d||f|'|<|||}(||(ddf})t|)tt fd}*||}+ttd|},t|,D]*}-|-t}.|*dd|-f}/|!dd|.f}0|+|/}1|+|0}2t||1|2tjqt | |}3t | |}4t ||d|f}5t ||d|f}6d }7d }8|dkrq|5|4}9| |9}:d };|:|krRtj|:d d }t t d | |>};n|9};tj|3d d  |;}?d d tj|6 d d }8tj|?d d }7tj|7d}7tj|8d}8tj|rQd }@d }A|tkr|'|}B|&|}C|B|B}@|C|C}Ad D]}D|@tjj|@|Dddd7}@|Atjj|A|Dddd7}Aq|dkr|@|#|<|A|#|d<tjd }Ed }F|dkr'd }Gd }H|tkr|#|}G|#|d}HdD]}D|Gtjj|G|Dddd7}G|Htjj|H|Dddd7}Hq|dkr'tj|Gdd d |#d<tj|Hdd d |#d<tj|#d}E|#d}F|tkrK|&||F|&|<|'|||E|'|<tjn|tkr^|'|||'|<tjt|D]:}-|-t}.tjdtj|-|,}I|I|kr|It}J|*dd|If}K|!dd|Jf}L|+|K}1|+|L}2t||1|2tj|-t |}Mt ||d||Mf}Nd }Ott ddD]}P|Pt!}Q|Q|}R|!|R||.f|7}S|&|R}T|O|S|T7}OqdD]}D|Otjj|O|Dtddd7}Oq|N|O|8}Utj|U|}Ud }Vtt ddD]/}P|Pt!}Q|Q|}R|!|R||.f|7}W|&|R}T|'|R}X|W|T|U}Y|Y|!|R||.f<|V|Y|X7}Vq dD]}D|Vtjj|V|Dtddd7}Vq=|dkre|-t |}Zt"|V| |d||Zf<tjt#t D]*}P||Pd}[|[t }\|[t }]|\tkr|!|\|]|.f}S|-t |]}^|S||(|\|^f<qotjqgdSdS)zDLarge batch kernel for (N, 1, ...) format with K-major state layout.rrr r7r8Nr rr:Tr;r=r>r?r@rAr rrEr r)$rFrGrHrIrJrK V_PER_WARP_NTrMrOrPrQrNrS TILE_V_NTrrVrYrZrr\r]r^r_r`rcrdrerUrfNUM_WARPS_LARGE_NTrgrhNUM_K_ITERS_NTROWS_PER_ITER_NTrRrb)_rrrrr#r$r%r!r&r r"r'r(r*r+r,r0r-r4rirjrrIrlrorprqrrrrrrwrxrrr r!r"r#r$rrrrrrrrrrrsrurtrvrrrrrrrrrr&r'r(r)rBrrr*r+r-rrrr.r/rr0r1r2r3r4rrr5r6r7r8r9r:r;r<rrr (gdn_decode_kernel_big_batch_nontransposeKsP                                            vrCc"Cs|jj\}}}| jjd|}tjtjtjjdtj dd}t |t }tj t t tftdt tfd}tj ddd}t d }t|||} d t ttd t d t d d }!t| ||||||||||| | | | | |||j|tddftddg|!|d dS)Nrrr rrr8)rr)rrrrr rr)rrrFrrrrrrMrNrrrSrrTILE_V_SMALL_PADDED_NTrr=rrNUM_THREADS_NT)"r)r#r$r%r!r&r r"rr(r'r*r+r,r.r/r0r-r1r2r3r4r batch_hv_dimrrrrnum_v_tiles_smallsmem_layout_smallthread_layout_smallval_layout_smalltiled_copy_load_smallsmem_bytes_smallrrr .run_gdn_decode_kernel_small_batch_nontranspose)sj       rMc"Cs|jj\}}}| jjd|}tjtjtjjdtj dd}t |t }tj t t tftdt tfd}tj ddd}t d }t|||} d t ttd t d t d d }!t| ||||||||||| | | | | |||j|ddftddg|!|d dS)Nrrr rrr8)rr )r rrrr rr)rrrFrrrrrrMrNrr?rSrrTILE_V_PADDED_NTrrCrNUM_THREADS_LARGE_NT)"r)r#r$r%r!r&r r"rr(r'r*r+r,r.r/r0r-r1r2r3r4rrFrrrrrbase_smem_layoutrrrrrrr ,run_gdn_decode_kernel_big_batch_nontransposezsf      rQc )Csh|j\} } } }| dksJd| |j\}}}}|j| |||fks6Jd| d|d|d|d|j |dksAJd ||dksLJd ||td ks\Jd td ||jtjtjfvsmJd|j|jtjks{Jd|j|jtjksJd|j|dur|d}| du}|r| jn|j}| durtj| | ||ftj|jd} | }| | |||}| | | ||||j|| f }t |}d|vs|dj|jkrtj | tj |jd|d<tj| dtj |jd|d<|d}|d}d|vr{ttjj}| tk}|rt}nt}t|dd}t|dd}t|dd}t|dd} t|dd}!t|dd}"t|dd}#t|dd}$t| dd}%t|dd}&t|dd}'tj||'|!|"|#||$|| ||&|%f dd|| | | |||d| |dd }(|(|d<n|d}(ttjj}|(||||||||||| | ||kr||| j|kr| |} | |fS)aGated Delta Rule Decode kernel (K-major layout, no transpose needed). This implements the decode phase of gated delta rule linear attention, processing one token at a time and updating the recurrent state. This version uses K-major state layout [B, HV, K, V] which is more natural and doesn't require transposition. Args: q (torch.Tensor): Current query of shape ``[B, 1, H, K]``. Must be float16/bfloat16. k (torch.Tensor): Current key of shape ``[B, 1, H, K]``. Must be float16/bfloat16. v (torch.Tensor): Current value of shape ``[B, 1, HV, V]``. Must be float16/bfloat16. state (torch.Tensor): Current state of shape ``[B, HV, K, V]`` (k-major layout). Must be float32. Will be updated in-place. A_log (torch.Tensor): Log decay parameter of shape ``[HV]``. Must be float32. a (torch.Tensor): Input-dependent decay of shape ``[B, 1, HV]``. Must be float16/bfloat16. dt_bias (torch.Tensor): Decay bias of shape ``[HV]``. Must be bfloat16 or float32. b (torch.Tensor): Update gate (beta) input of shape ``[B, 1, HV]``. Must be float16/bfloat16. scale (Optional[float]): Scale factor for queries. If None, defaults to ``1 / sqrt(K)``. output (Optional[torch.Tensor]): Pre-allocated output tensor of shape ``[B, 1, HV, V]``. If None, will be allocated automatically. use_qk_l2norm (bool): Whether to apply L2 normalization to q and k. Default: ``True``. Returns: Tuple[torch.Tensor, torch.Tensor]: - output: Output tensor of shape ``[B, 1, HV, V]`` - state: Updated state tensor of shape ``[B, HV, K, V]`` Note: - Requires SM90 (Hopper) architecture - State is updated in-place - K and V must be multiples of 4 for vectorized loads - State layout is k-major: [B, HV, K, V] (no transpose needed) rrrrrrrr rrrrrrrrNrrr(r)rrrr=rTr) r*r+r,r.r/r0r-r1r2r3r4rr)rr?rrrrrrr contiguousviewrarangerrrrrSMALL_BATCH_THRESHOLD_NTrMrQrrFrdata_ptrrr))r#r$r%rr r!r"r&r,rr4r.r/r0r1rjr-r2rrstate_contiguousrrrr(r)ruse_small_batchr rrrrrrrrrrrrrrr gated_delta_rule_decodes;"                  rYintermediate_statestile_vdisable_state_updatecache_intermediate_statesc] CsTtj\}}}|d}tj} tj| } ||}!d|!}"d|"}#||!}$||!}%| |"|%}&tj\}'}}|'|}(|'|})|)|}*|)|}+|*||},| |+}-t||*}.t||*}/tj }0|0 tjtj ||f|ddfdd}1|0 tjtj ||f|ddfdd}2|0 tjt |fd}3|0 tjt |fd}4t tj |fddtj}5t tj |fddtj}6t tj |fddtj}7t tj |fddtj }8t tj |fddtj }9|-dkr|$|}:t|D]o};t|ddd|f|+|;|,|$f}t|8|>|5|><t|9|>|6|><q%t|rd }?d }@t|D]}>|?|5|>|5|>7}?|@|6|>|6|>7}@qJd D]}A|?tjj|?|Ad d d 7}?|@tjj|@|Ad d d 7}@qdtj|?ddd|}Btj|@ddd}Ct|D]}>|5|>|B|5|><|6|>|C|6|><qnt|D] }>|5|>||5|><q||!krt|D]}>|5|>|1|;|:|>f<|6|>|2|;|:|>f<qt||+|;|*f}Dt| |+|;|*f}E|D|/}F||F}Gtj|Gdd}Htd|tjtd|Hdd}I|G|kr!tdntd }J|J|Itd|J|F}Ktj|.dd |K}Ltdtdtj|E dd}Mtj|Ldd}N|dkrc|N|3|;<|M|4|;<qtj||#}Ot|OD]4}P|(||&|O|P}Q|Q|kr|-||*}Rt|dd|f|R|Q|$f}St|S|7t|D]};t|1d|f|;|$f}Tt|2d|f|;|$f}Ut|T|5t|U|6|3|;}N|4|;}Mt|D] }>|7|>|N|7|><qd }Vt|D] }>|V|7|>|6|>7}Vqd D]}A|Vtjj|V|Ad d d 7}Vqt| |+|;|*|Qf}W|W|V|M}Xt|D]}>|7|>|6|>|X7<qt|rQ|+|||;||*}Yt|dd|f|Y|Q|$f}Zt|7|Zd }[t|D] }>|[|7|>|5|>7}[qXd D]}A|[tjj|[|Ad d d 7}[qh|$dkrt |[| |+|;|*|Qf<qt| rt|dd|f|R|Q|$f}\t|7|\qrdSdS)av Parallel MTP kernel - each block handles one [TILE_V, TILE_K] tile. Grid: (B * HV * num_v_tiles, 1, 1) Each block: - Loads its v_tile of state into registers - Processes all T time steps with state in registers - Writes output and optionally updates state This matches Triton's parallelization strategy for better small-batch performance. rrr rr8rr7rr:r>r?r@rArETr;r=N)rFrGrHrIrJrKrMrNrOrPrQrSrTrRrbrVra const_exprrfrgrcrdrU)]rrZrrr[r r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r\r]rirjrkrIthreads_per_groupgroups_per_warp num_groups lane_in_group group_in_warp group_idxrli_vtmprprorq cache_idxrsrurwr#r"sGsBetar|r{r}r~rrrrrrrrrrBinv_norm_q_scaledrrtrvrrr softplus_val use_softplusrrrrrows_per_group row_in_grouprflat_state_idxh_tilesQ_tilesK_tilerr/rr$ inter_tiler h_tile_outrrr gdn_verify_kernel_mtpsH+                          \ruc#Cs|jjd|jjd|jjd}}}t||} ||| }!d||dd||dd|d|d}"t|||| ||||||||| | | | | ||||||||||||j|!ddftddg|"|ddS)Nrrr rr r r)rrrFrrurNUM_THREADS_MTP)#rrZr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r[rr3r4r5r\r]rrjrrr grid_sizerrrr run_gdn_verify_kernel_mtpsj !     rx pool_size cache_stepscCr)z2Cache compiled MTP kernel for given configuration.r)r.r/r0r-r1r2ryrzr\r]r,r4r[rrrr _get_compiled_mtp_kernel sr{ initial_stateinitial_state_indicesintermediate_states_bufferc0Cs,|j\}}}}|j\}}}}|jd}t||}t||}|j||||fks:Jd|d|d|d|d|j |dksEJd||dksPJd |||dks`Jd |d ||jtjtjfvsqJd |j|jtjksJd |j|jtjksJd|j| dur|d} | du}|r| jn|j}| durtj||||ftj|j d} | tj ||||}| du}|r| jd}| jd}||ksJd|d|d| tj ||||| }n|}tjdddtj|j d}||||||||| || | ||f}t |} d| vs!| dj |j kr/tj|dtj|j d| d<| d}!d| vrttjj}"t|dd}#t|dd}$t|dd}%t|dd}&t|dd}'t|dd}(t|dd})t|dd}*t|dd}+t| dd},t|dd}-t|!dd}.tjt|#|$|%|&|'|(|)|*|+|,|-|.f iddddd| d|d |d!|d"|d#|d$|d%|d&|d'd(d)| d*d+d,| d-|d.|"d/d0}/|/| d<n| d}/ttjj}"|/|||||||||| ||!|" | s|s|| ||||| j|kr| |} | |fS)1am Gated Delta Rule MTP Kernel (Multiple Token Processing). This function processes multiple tokens (T > 1) in sequence, typically used for speculative decoding verification. It supports intermediate state caching for potential rollback scenarios. Args: q (torch.Tensor): Query tensor of shape ``[B, T, H, K]``. k (torch.Tensor): Key tensor of shape ``[B, T, H, K]``. v (torch.Tensor): Value tensor of shape ``[B, T, HV, V]``. initial_state (torch.Tensor): Initial state tensor of shape ``[pool_size, HV, V, K]`` (K-last layout). initial_state_indices (torch.Tensor): Indices mapping each batch to its initial state, shape ``[B]``. A_log (torch.Tensor): Log decay parameter of shape ``[HV]``. a (torch.Tensor): Input-dependent decay of shape ``[B, T, HV]``. dt_bias (torch.Tensor): Decay bias of shape ``[HV]``. b (torch.Tensor): Update gate input of shape ``[B, T, HV]``. scale (Optional[float]): Scaling factor for queries. If None, uses ``1/sqrt(K)``. output (Optional[torch.Tensor]): Pre-allocated output tensor of shape ``[B, T, HV, V]``. intermediate_states_buffer (Optional[torch.Tensor]): Buffer for caching intermediate states, shape ``[pool_size, T, HV, V, K]``. If None, intermediate states are not cached. disable_state_update (bool): If True, the initial state is not updated. Default: ``True``. use_qk_l2norm (bool): Whether to apply L2 normalization to q and k. Default: ``True``. Returns: Tuple[torch.Tensor, torch.Tensor]: - output: Output tensor of shape ``[B, T, HV, V]`` - initial_state: Updated state tensor (unchanged if disable_state_update=True) Note: - Requires SM90 (Hopper) architecture - Supports T > 1 (multiple token processing) - State layout is K-last: [pool_size, HV, V, K] - Optimized for speculative decoding verification scenarios rz(Expected initial_state shape [pool_size=rrrrr rrrrrz#initial_state must be float32, got rNrrrz9intermediate_states_buffer second dimension (cache_steps=z) must be at least T=z" to prevent out-of-bounds indexingr)rrrr*r=r+rr,r-r.r/r0r1r2r[rr3Tr4r5Fr\r]rrr)rrrrrrrrrrrrrRr{rrrrrrrFrrxrr)0r#r$r%r|r}r r!r"r&r,rr~r\r4r.r/r0r1rjr-r2ryr[rrrrr] buffer_sizerzrZrrr)rrintermediate_states_tensorrrrrrrrrrrrrrr gated_delta_rule_mtp s&C   "                    !  rr7)NNT)NNNTT)H__doc__ functoolstypingrrrrM cutlass.cuterFcutlass.cute.nvgpurcutlass.cute.runtimercuda.bindings.driverbindingsdriverr api_loggingr_FLASHINFER_AVAILABLE ImportErrorrWrXr[rrLrr?rNrrDrrErrOr@r>rBrArU TILE_K_MTPrvintrrkernel TiledCopyTensorLayout Constexprfloatboolrrjitrrrrrrrr r=rCrMrQrYrurxr{rrrrr s                           g      q       9      r      ^      P      S     L            Y