o ٷi- @sddlZddlmmZddlmZdZdZe r:zddl m Z e r-ddl mZmZWne efy9Yn}werRzddlmZWnoe efyQYnewz ddlmZmZWn e efygYnwedurz ddlmZmZWn e efyYnwedurz ddlmZmZWn e efyYnwedurz ddlmZmZWn e efyYnwedupeduZddZddd Zd d Zd ejd eejejeffddZdejdejdejd ejdef ddZddZdS)N)current_omni_platform)is_aiter_found_and_supported)flash_attn_funcflash_attn_varlen_func)rcCs$|jdg|jddR}||S)a A local implementation of the PyTorch indexing operation `tensor[indices]` on the first axis, after flattening the first two dimensions of the tensor. This is functionally equivalent to FA2's `index_first_axis` and replaces the need to import it. N)reshapeshape)tensorindicesreshaped_tensorr c/home/ubuntu/.local/lib/python3.10/site-packages/vllm_omni/diffusion/attention/backends/utils/fa.py_index_first_axisJsrc Cs|dur||n|}|jdtjd}|jdtjd}tj|dd}|}ttj |dtjdd}t ||||||fS)a4 unpad_input function for flash attention variants that do not have them within their pkg themselves, e.g. fa3. Arguments: hidden_states: (batch, seqlen, ...) attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid. unused_mask: (batch, seqlen), bool / int, 1 means the element is allocated but unused. Return: hidden_states: (total_nnz, ...), where total_nnz = number of tokens selected in attention_mask + unused_mask. indices: (total_nnz), the indices of masked tokens from the flattened input sequence. cu_seqlens: (batch + 1), the cumulative sequence lengths, used to index into hidden_states. max_seqlen_in_batch: int seqused: (batch), returns the number of tokens selected in attention_mask + unused_mask. NrdimdtypeFas_tuplerr) sumtorchint32nonzeroflattenmaxitemFpadcumsumr) hidden_statesattention_mask unused_mask all_masksseqlens_in_batchused_seqlens_in_batchr max_seqlen_in_batch cu_seqlensr r r _unpad_inputVs r)cCsL|jdd}tj||g|R|j|jd}|||<|j||g|RS)a pad_input function for flash attention variants that do not have them within their pkg themselves, e.g. fa3. Arguments: hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask. indices: (total_nnz), the indices that represent the non-masked tokens of the original padded input sequence. batch: int, batch size for the padded sequence. seqlen: int, maximum sequence length for the padded sequence. Return: hidden_states: (batch, seqlen, ...) rN)devicer)r rzerosr*rview)r!r batchseqlenroutputr r r _pad_inputvs "r0r"returncCsV|jdtjd}tj|dd}|}ttj |dtjdd}|||fS)a~ Retrieves indexing data required to repad unpadded (ragged) tensors. Arguments: attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: indices (`torch.Tensor`): The indices of non-masked tokens from the flattened input sequence. cu_seqlens (`torch.Tensor`): The cumulative sequence lengths, used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). max_seqlen_in_batch (`int`): Maximum sequence length in batch. rrFrrr) rrrrrrrrrr )r"r%r r'r(r r r_get_unpad_datas r2 query_layer key_layer value_layer query_lengthcCs<tjr dtjj_t|\}}}|jd|jd} kr>|ddd| ddddf|ddd| ddddf}}|j\} } } } t||}t||}|| kr_t||}|}|}|}n3|dkr}d}tj | dtj |j d}|dd}| d}n|dd| df}|||^}}}}}||||||f||ffS)a Unpads query, key, and values tensors, using a single dimension for all tokens even though they belong to different batches. This function is used instead of `flash_attn.bert_padding.unpad_input` in order to avoid the recomputation of the same intermediary tensors for query, key, value tensors. Arguments: query_layer (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. query_length (`int`): Target length. unpad_input_func: The function to use for unpadding the input tensors. Return: query_layer (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). TrrN)rr*) rcompiler is_compiling_dynamoconfigcapture_scalar_outputsr2r rarangerr*squeeze)r3r4r5r"r6unpad_input_func indices_k cu_seqlens_kmax_seqlen_in_batch_kseq_len batch_size kv_seq_lennum_key_value_headshead_dim cu_seqlens_qmax_seqlen_in_batch_q indices_q_r r r _upad_inputs: + B     rKcCsF|durdStj|jd|jd|}|dko"||S)a" Check the position ids whether packed sequences are indicated or not 1. Position ids exist 2. Flattened sequences only are supported 3. Compile-friendly `not (torch.diff(position_ids, dim=-1) >= 0).all()`, i.e. we have multiple increasing sequences NFr)r*)rr<r r*minabsrbool) position_idsrCincreasing_position_sequencesr r r_is_packed_sequencesrQ)N)rtorch.nn.functionalnn functionalrvllm_omni.platformsrrris_rocmvllm._aiter_opsraiter ImportErrorModuleNotFoundErroris_xpu#vllm.v1.attention.backends.fa_utilsfa3_fwd_interfaceflash_attn_interface flash_attnflash_attn.flash_attn_interfaceHAS_FLASH_ATTNrr)r0Tensortupleintr2rKrQr r r rsx    " U