o i"&@sFdZddlZddlmZddlZddlmZddlmZm Z m Z ddl m Z m Z mZejdd Zejd d Zejd d ZdejfddZe          d/dejdejdejdejdejdejdejdeejdeejdedeejdedeejd ed!eejd"eejd#ed$ejf$d%d&Ze d'd(d)dejdejdejdejdejdejdejdeejdeejdedeejded*ejd ed!eejd"eejd#ed$df$d+d,Zed'dejdejdejdejdejdejdejdeejdeejdedeejded*ejd ed!eejd"eejd#ed$df$d-d.ZdS)0a3 Copyright (c) 2025 by FlashInfer team. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. N)Optional)flashinfer_api)!gen_selective_state_update_module&gen_selective_state_update_sm90_module'gen_selective_state_update_sm100_module)get_compute_capabilityregister_custom_opregister_fake_opcC tS)zEGet cached JIT-compiled selective_state_update module (base version).)rbuild_and_loadr r ]/home/ubuntu/vllm_env/lib/python3.10/site-packages/flashinfer/mamba/selective_state_update.py&get_selective_state_update_module_base rcCr )zLGet cached JIT-compiled selective_state_update module (SM90/Hopper version).)rr r r r r&get_selective_state_update_module_sm90%rrcCr )zQGet cached JIT-compiled selective_state_update module (SM100+/Blackwell version).)rr r r r r'get_selective_state_update_module_sm100+rrdevicecCs.t|\}}|dkr tS|dkrtStS)N )rrrr)rmajor_r r r!get_selective_state_update_module1s rFstatexdtABCDzdt_bias dt_softplusstate_batch_indices pad_slot_idoutdisable_state_updateintermediate_states_bufferintermediate_state_indices cache_stepsreturncCs|dk}|dkr|d}|dkr|d}|dkr%|d}|dur4|dkr4|d}|dkr?|d}|rL|dkrL|d}|dkrW|d}|rd|dkrd|d}|dkro|d}|r||dkr||d}|dkr|d}|r|dkr|d}|dur|dkr|d}|r|dkr|d}| durt|}n| }t|||||||||| | | || ||||S)a[ Selective state update operation for Mamba layers (the generation phase). Parameters ---------- state : torch.Tensor State tensor with shape (state_cache_size, dim, dstate) or (state_cache_size, nheads, dim, dstate) x : torch.Tensor Input tensor with shape (batch, dim) or (batch, nheads, dim) for single-token or (batch, T, nheads, dim) for multi-token dt : torch.Tensor Delta time tensor with shape (batch, dim) or (batch, nheads, dim) for single-token or (batch, T, nheads, dim) for multi-token A : torch.Tensor A matrix with shape (dim, dstate) or (nheads, dim, dstate) B : torch.Tensor B matrix with shape (batch, dstate) or (batch, ngroups, dstate) for single-token or (batch, T, ngroups, dstate) for multi-token C : torch.Tensor C matrix with shape (batch, dstate) or (batch, ngroups, dstate) for single-token or (batch, T, ngroups, dstate) for multi-token D : torch.Tensor D vector with shape (dim,) or (nheads, dim) z : Optional[torch.Tensor] Optional z tensor with shape (batch, dim) or (batch, nheads, dim) for single-token or (batch, T, nheads, dim) for multi-token dt_bias : Optional[torch.Tensor] Optional dt bias with shape (dim,) or (nheads, dim) dt_softplus : bool Whether to apply softplus to dt state_batch_indices : Optional[torch.Tensor] Optional batch indices for cache processing with shape (batch,) pad_slot_id : int If state_batch_indices is passed, lets the kernel identify padded entries that will not be processed. For example: state_batch_indices = [pad_slot_id, 1, 20, pad_slot_id] in this case, the kernel will not process entries at indices 0 and 3 out : Optional[torch.Tensor] Optional output tensor (same shape as x) disable_state_update : bool If True, skip updating the state tensor (useful for speculative decoding verification) intermediate_states_buffer : Optional[torch.Tensor] Optional buffer for caching intermediate states during speculative decoding with shape (batch, cache_steps, nheads, dim, dstate) intermediate_state_indices : Optional[torch.Tensor] Optional indices mapping batch elements to intermediate state buffer positions with shape (batch,) cache_steps : int Number of steps/tokens to cache for speculative decoding Returns ------- output : torch.Tensor Output tensor with shape (batch, dim) or (batch, nheads, dim) for single-token or (batch, T, nheads, dim) for multi-token rrN)dim unsqueezetorch empty_like_selective_state_update)rrrrrrr r!r"r#r$r%r&r'r(r)r*is_mtpoutputr r rselective_state_update>sjK                       r5z"flashinfer::selective_state_update)rr4r() mutates_argsr4cCs4t|j|||||||||| | | | | |||dS)zLInternal function registered with torch.library for torch.compile() support.N)rrr5rrrrrrr r!r"r#r$r%r4r'r(r)r*r r rr2s& r2cCsdS)z@Fake implementation for torch.compile() meta tensor propagation.Nr r7r r r_selective_state_update_fakesr8) NNFNrNFNNr)__doc__ functoolstypingrr0 api_loggingr jit.mambarrrutilsrr r cacherrrrrTensorboolintr5r2r8r r r rs                    )