o i$o@sddlZddlmZddlmZddlmZmZmZm Z m Z ddl Z ddl m Z ddl mZmZmZmZmZmZddlmZmZmZdd lmZdd lmZmZeGd d d Z dLd e jde jdee jde ee ee jffddZ!dee e dfdee fddZ"dee dee jde jfddZ#dee e dfdee dee j$dee jde jf ddZ%dee e dfdee dee jde jfddZ&dee e dfdee jde e jee ffd d!Z'dee d"e dee jde jfd#d$Z(dee e dfdee jde jfd%d&Z)dee e dfdee jde e je jffd'd(Z*dee e dfdee jde jfd)d*Z+d+e d,e d-e de,fd.d/Z-d+e d,e d-e d0e,d1ed2e j$dee efd3d4Z.   5 dMd6e jd7e jd8e jd9e jd:e jd;e jdee ed0e,d?ee e je jfde jfd@dAZ/GdBdCdCej0Z1edDe dEe dee e dffdFdGZ2dEee dee e dffdHdIZ3dDe dEe de fdJdKZ4dS)NN) dataclass) lru_cache)ListOptionalSequenceTupleType)fmha)flashflash30memory_efficient_attention_forward_requires_grad"memory_efficient_attention_partialmerge_attentions triton_splitk) AttentionBiasPagedBlockDiagonalGappyKeysMask PagedBlockDiagonalPaddedKeysMask)AttentionFwOpBase) _get_use_fa3 fa3_availablec @s:eZdZUdZeeedfed<ej ed<ej ed<ej ed<e eed<ej ed<ej ed <ej ed <ej ed <ej ed <ej ed <ej ed<e ej ed<e ej ed<e eed<e e  ddeeedfde ejde ejddfddZe  ddeeedfde ejde ejddfddZdS)TreeAttnMetadataa tree_choices: definition of the tree, tuples sorted by length, each corresponding to a node. See the docstring of TreeAttnMetadata.from_tree_choices. attention_bias: Medusa-style tree attention bias as an explicit tensor of shape (tree_size, tree_size), where tree_size is the total number of nodes in the tree. It can be used as a spec_attn_bias ("right" or "suffix" attention part) in tree_attention. See tree_attention_with_sync for a usage example. tree_indices: 1D tensor of size tree_size which maps tree nodes to draft tokens. Tree nodes are assumed to be in the same order as in tree_choices (see TreeAttnMetadata.from_tree_choices). retrieval_indices: a tensor of shape (number of leaves, depth + 1), where one row corresponds to one path, and contains indices of the tree nodes on that path. Paths are padded with -1 from the right. The paths (row dim) are unsorted. path_lengths: real lengths for each of the paths. tree_seq_position_ids: 1D tensor of size tree_size which indicates which head a node belongs to. Equivalently, it shows the sequence position of the node within the corresponding path. parent_node_indices: 1D tensor of size tree_size which for each node contains position of its parent + 1. For root node(s) it contains 0. child_node_indices: a tensor of shape (tree_size, max_num_children_per_node), in which each row contains indices of children of the corresponding node. Rows corresponding to nodes which have less than max_num_children_per_node children are padded by repeating the last child index. For leaf nodes the values are meaningless and filled with 0. num_children_per_node: 1D tensor of size tree_size which contains the number of children for each node. candidate_idx: 1D tensor of size tree_size, contains index of each node among its "siblings". Takes values from 0 to the number of children of the parent node minus 1. num_nodes_per_level: 1D tensor of the number of nodes at each level (including root). num_children_per_node_at_level: List of 1D tensors, each containing the number of children at the tree level. subtree_size: List of integers, each containing the number of nodes in the subtree at the tree level. Example: Tree choices `[(0,), (0, 0), (0, 1), (0, 2), (1,), (1, 0), (1, 1), (1, 2), (2,), (2, 0), (2, 1), (2, 2)]` represents a tree that looks like this: 0 |-- 1 | |-- 4 | |-- 5 | |-- 6 | |-- 2 | |-- 7 | |-- 8 | |-- 9 | |-- 3 |-- 10 |-- 11 |-- 12 with TreeAttnMetadata tree_indices=tensor([0, 1, 2, 3, 4, 5, 6, 4, 5, 6, 4, 5, 6]) retrieval_indices=tensor([[ 0, 1, 5], [ 0, 2, 9], [ 0, 3, 11], [ 0, 1, 4], [ 0, 2, 8], [ 0, 3, 10], [ 0, 1, 6], [ 0, 2, 7], [ 0, 3, 12]]) path_lengths=[3, 3, 3, 3, 3, 3, 3, 3, 3] tree_seq_position_ids=tensor([0, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2]) child_node_indices=tensor([[ 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8], [ 9, 10, 11], [ 0, 0, 0], ... [ 0, 0, 0]]) num_children_per_node=tensor([3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0]) candidate_idx=tensor([0, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2]) num_nodes_per_level=tensor([1, 3, 3]) num_children_per_node_at_level=[tensor([3]), tensor([3, 3, 3]), tensor([0, 0, 0, 0, 0, 0, 0, 0, 0])] subtree_sizes=[1, 4, 13] . tree_choicesattention_bias tree_indicesretrieval_indices path_lengthstree_seq_position_idsparent_node_indiceschild_node_indicesnum_children_per_node candidate_idxnum_nodes_per_levelnum_nodes_per_level_cpunum_children_per_node_at_level"num_children_per_node_at_level_cpu subtree_sizesNdtypedevicereturncCs||||SN)from_tree_choices)clsrr'r(r-O/home/ubuntu/.local/lib/python3.10/site-packages/xformers/ops/tree_attention.pyfrom_tree_choices_cachedsz)TreeAttnMetadata.from_tree_choices_cachedcCst|ddd}t|d}t|}t|||}t||\}} t|||} t||||} t||} t||\} }t ||}t ||}| }t |||\}}dd|D}t || ||| | | | |||||||S)a\ Args: tree_choices: tree description in the style of https://github.com/FasterDecoding/Medusa/blob/5e9805386/medusa/model/medusa_choices.py A typical tree description would look like: [(node0, node1, ...), (node0, node2), (node0, node3), (node1, node3), ..., (node0, node2, ..., nodeN)] Every tuple is corresponds to one node in the tree, encoded as a path from one of the root nodes to the node in question. For example, a node encoded as (1, 0, 3, ..., 2) is understood as: list all the root nodes and take node number 1 list all children of that node and take node number 0 list all children of that node and take node number 3 ... list all children of that node and take node number 2 - that's the node encoded by this tuple. dtype: data type of the output mask tensor. device: device of the output tensors. Returns: TreeAttnMetadata object with all the fields. cSs t||fSr*lenxr-r-r.s z4TreeAttnMetadata.from_tree_choices..)keyr cSsg|]}|qSr-)cpu).0rowr-r-r. sz6TreeAttnMetadata.from_tree_choices..)sortedr1_get_depth_counts_prepare_tree_indices_prepare_retrieval_indices_prepare_tree_position_ids_prepare_tree_attn_bias_prepare_parent_node_indices_prepare_child_node_indices_prepare_candidate_idx_get_num_nodes_per_levelr64_get_subtree_size_and_num_children_per_node_at_levelr)r,rr'r(sorted_tree_choicestree_len depth_countsrrrrtree_attn_maskrrr r!r"r#r&r$r%r-r-r.r+s\     z"TreeAttnMetadata.from_tree_choices)NN)__name__ __module__ __qualname____doc__rrint__annotations__torchTensorr classmethodrrr'r(r/r+r-r-r-r.r!sT P            rr"r r(r)c Cstt|}dg}|ddg}td|D]!}|tt|d|d||||d||q||fS)Nr r)r1 unsqueezerangeappendrMrOsum)r"r r(depthr&r$ir-r-r.rDs  rDrE.cCsHg}d}|D]}t|}||kr|d||dd7<|}q|S)Nrr )r1rT)rErG prev_depthpathrVr-r-r.r;s r;rGcCs tjdg||d}||dkS)Nr )r(r)rOtensor)rGr(depth_counts_tensorr-r-r.rCs rCr'c Cst|d}tj||ftj ||d}d}t|D]}||||f<q||dddf<d}tt|D]E}t||D]6} ||| } t| dkrIq:g} tt| dD]} | || d| ddqS||| |d| f<q:|||7}q2|S)a Construct a Medusa-style tree attention bias as an explicit tensor. It can be used as a spec_attn_bias ("right" or "suffix" attention part) in tree_attention. See run_tree_attention_inner in test for a usage example. Args: sorted_tree_choices: tree description in the style of https://github.com/FasterDecoding/Medusa/blob/5e9805386/medusa/model/medusa_choices.py A typical tree description would look like: [(node0, node1, ...), (node0, node2), (node0, node3), (node1, node3), ..., (node0, node2, ..., nodeN)] Every tuple is corresponds to one node in the tree, encoded as a path from one of the root nodes to the node in question. Passed in sorted order. For example, a node encoded as (1, 0, 3, ..., 2) is understood as: list all the root nodes and take node number 1 list all children of that node and take node number 0 list all children of that node and take node number 3 ... list all children of that node and take node number 2 - that's the node encoded by this tuple depth_counts: a list of integers, where the i-th element is the number of choices with depth i. dtype: data type of the output tensor. device: device of the output tensor. Returns: attention bias of shape (tree_size, tree_size), where tree_size is the total number of nodes in the tree. r r(r'rN)r1rOfullinfrSrTindex) rErGr'r(rFrHmask_valrWstartjcur_tree_choice ancestor_idxcr-r-r.r?s,    r?c Cst|d}tj||tjd}d|d<d\}}tt|D].}|}t||D]} ||| } | d|d} | ||| d<t| |}q'|||7}q|S)a Construct an index tensor for choices in the tree and their corresponding index in the draft tokens. Args: sorted_tree_choices: sorted from tree_choices input of prepare_tree_attn_metadata function depth_counts: a list of integers, where the i-th element is the number of choices with depth i. device: device of the output tensor. Returns: tree indices of shape (tree_len,). See docstring of TreeAttnMetadata for details. r r\r)rr)r1rOzeroslongrSmax) rErGr(rFrramax_idx_prev_levelrW cur_offsetrbrctree_idxr-r-r.r<=s   r<rc sr tddDdnd}t}dddD]}|dd|vr-||ddqgg}}|D]+dgfddtdtdD}|t|||dg|t|q5tj|tj|d }||fS) a Convert tree definition from the format used by Medusa and EAGLE (tree_choices, see docstring of TreeAttnMetadata.from_tree_choices) to a list of paths: [ (node_index0_path0, node_index1_path0, ...), (node_index0_path1, node_index1_path1, ...), ... ] where each value is an index of a node inside the corresponding level of a tree. Returns: retrieval indices of shape (number of leaves, depth + 1) length of each path. cs|]}t|VqdSr*r0)r7noder-r-r. lz-_prepare_retrieval_indices..r Nrfrcs"g|] }d|dqS)Nr )r_)r7levelleafrr-r.r9vsz._prepare_retrieval_indices..r'r() risetremoverSr1rTrOrZrh) rr( tree_depthleavesrnpathsrrY paths_tensorr-rrr.r=\s r=rFcCsXtj|tj|d}d}tt|D]}|d||d|||d<|||7}q|S)a Construct sequence position of each node within its path, can be used for positional embedding. Args: depth_counts: number of nodes at each of the levels of the tree. tree_len: total number of nodes in the tree including the root. device: device of the output tensor. Returns: tree position ids of shape (tree_len,). See docstring of TreeAttnMetadata for details. rtrr )rOrgint32rSr1)rGrFr(tree_position_idsrarWr-r-r.r>s   r>c Cs\g}|D] }z|||dddWqty$|dYqwtj|tj|dS)Nrfr rrt)rTr_ ValueErrorrOrZrh)rEr(rdcur_medusa_choicer-r-r.r@s  r@csgg}dg|D]"fddt|D}|t||r%|q dgq tddDfddDtfddDsKJtjtj|d tj|tj|d fS) Nr-cs8g|]\}}tdt|kr|ddkr|qS)r Nrfr0)r7rWyr2r-r.r9s $z/_prepare_child_node_indices..rcsrmr*r0r7r3r-r-r.rorpz._prepare_child_node_indices..cs(g|]}||ddt|qS)rfNr0r)max_num_childrenr-r.r9s(c3s$|] }t|tdkVqdS)rNr0r)resr-r.ros"rt) enumeraterTr1riallrOrZrh)rr(r curr_childrenr-)rrr3r.rAs   rAcs(fddtD}tj|tj|dS)Ncs.g|]\}tfddd|DqS)c3s(|]}dd|ddkVqdS)Nrfr-)r7 another_node curr_noder-r.ros  z4_prepare_candidate_idx...N)rU)r7 curr_node_idxrrr.r9s  z*_prepare_candidate_idx..rt)rrOrZrh)rr(r!r-rr.rBs rBBG tree_sizecCs4||dkr |dkp||dko|dkp||dkS)n Heuristic to decide whether to use Triton Split-k or default (Flash Attention) for prefix attention. @dr-)rrrr-r-r.use_triton_splitk_for_prefixs  rautotune attn_biaskv_cache_dtypec Cs|rtntj}tjjr |St|tjj }t|tjj }t|t jj } |s*|s*| s*|Stjj s0Jt o5t } |tjkr=|S|dkrK| rK|rHtjStjSt|||rS|SdS)rrN)SplitKAutotunerFwOprOversionhip isinstancer FwOp_KVSplitSUPPORTED_ATTN_BIAS_TYPESr cudarruint8r) rrrrrrtriton_splitk_opfa3_splitkv_supported fa3_supportedflash2_supporteduse_fa3r-r-r.select_prefix_ops*       rFqspec_kspec_vcache_kcache_vspec_attn_biasprefix_attn_bias prefix_op suffix_opquantized_kv_scalesc " Cs|jdk} | r"|d}|d|d}}|d|d}}|j\} } }}}|j\}}}}}|j\}}t|ttfrE|dksDJn|| ksKJ||krW||krW||ksYJ|j|jksaJ|| krk|ksynJd|d| d||jdd|jddkr|jddksnJd|jd |jd |j|| ||| |}| rtntj }|durt | ||| ||j }|j t j kr(| dusJ|tj usJtj|d| | ||||d|||||t j|d|||||t j|| d d||||| dd||||d d }tj||d\}}||j}}n&t|d| | ||||d||||||d|||||||d\}}|| | |||}|||| | dd dd}t|||||pn|d\}}t||g||g|j d\} }!| r| d} | S)a Compute Medusa/EAGLE/Hydra-style tree attention. Notice that this function takes as arguments biases for the left (prefix) and right (speculative suffix) parts of the attention. This way we avoid creating these biases on the fly, and allow this function to be used in performance-critical decoding jobs, including in CUDA graph mode. In the latter case one should construct the biases once, and update prefix_attn_bias with current seqlens before every graph replay; spec_attn_bias stays static, as it's determined by the tree structure. Args: q: query from speculative tokens, of shape (B, tree_size_q, (G), H, D) spec_k, spec_v: keys/values from speculative tokens, each of shape (B, tree_size_kv, (G), H, D). If tree_size_q < tree_size_kv, we assume the end of the query sequence aligns with end the k/v sequence, like in "from-bottom-right" attention masks. Such rectangular attention masks can be used when we are adding new nodes to the tree, and want to avoid recomputing attention for the existing nodes. For example, this can be used during draft token generation in EAGLE. cache_k/cache_v: queries/keys/values from the existing context, each of shape (B, Mk, (G), H, D) spec_attn_bias: attention bias of the "right" part of the attention (tree_size_q x spec tokens). This would typically be a an explicit tensor mask, precomputed once and not changing during decoding prefix_attn_bias: attention bias of the "left" part of the attention (tree_size_q x existing context). This bias would typically be block-diagonal padded non-causal (BlockDiagonalPaddedKeysMask), and it changes at every decoding step as K/V sequence lengths grow during decoding. prefix_op: attention backend which will be passed to memory_efficient_attention to compute prefix attention. If None, will use Triton Split-K or Flash Attention depending on the heuristics. suffix_op: same as prefix_op, but for the suffix. autotune: If True, Triton Split-K will use autotuning when chosen as a default backend for prefix/suffix attention. Returns: attention output of shape (B, tree_size_q, (G), H, D) :Usage example: See also tree_attention_with_sync in tests/test_tree_attention.py .. code-block:: python # Create an attention bias for the prefix part of the attention prefix_attn_bias = BlockDiagonalPaddedKeysMask.from_seqlens( q_seqlen=[tree_size_q for _ in range(B)], kv_seqlen=kv_lens, kv_padding=Mk ) # Create an explit attention bias for the speculative part of the attention spec_attn_bias = TreeAttnMetadata.from_tree_choices(tree_choices, q.dtype, q.device).attention_bias attn_output = tree_attention( q, spec_k, spec_v, cache_k, cache_v, spec_attn_bias, prefix_attn_bias ) rrr z tree_size_q1=z tree_size_q=z tree_size_kv=Nzq.shape=z spec_k.shape=z spec_v.shape=rT)queryr5valuerk_fp8_scale_shiftv_fp8_scale_shift is_partial)op)rr) output_dtype)ndimrRshaperrrexpandrrrrr'rOr InputsFp8viewr{r 1_memory_efficient_attention_forward_requires_gradlserpermuter rsqueeze)"rrrrrrrrrrris_bmhkr tree_size_qrHDBkvMkG1H1D1 tree_size_q1 tree_size_kvrfp8_inpoutctx attn_prefix lse_prefix attn_suffix lse_suffix attn_output_r-r-r.tree_attentions =   6        rc@seZdZdZdS)rTN)rIrJrKAUTOTUNEr-r-r-r.rsrrw branchingcCst|g|dS)a Construct a full tree of a given depth where each node (except for leaves) has a given number of children. The format is compatible with that used by Medusa and EAGLE: https://github.com/FasterDecoding/Medusa/blob/5e98053/medusa/model/medusa_choices.py For detailed description, see docstring of xformers.ops.tree_attention.TreeAttnMetadata.from_tree_choices . r)construct_tree_choicesrwrr-r-r.construct_full_tree_choicess rcs@g}ttD]}|tjfddt|dDq|S)zS Construct a tree based on given branching factor for each non-root level. csg|]}t|qSr-)rS)r7krr-r.r9sz*construct_tree_choices..r )rSr1extend itertoolsproduct)rchoicesrWr-rr.rs(rcstfddt|DS)zi Number of nodes in a full tree of a given depth (including the root node) and branching factor. c3s|]}|VqdSr*r-)r7rWrr-r.rorpz%get_full_tree_size..)rUrSrr-rr.get_full_tree_sizesrr*)NNFN)5r dataclassesr functoolsrtypingrrrrrrO xformers.opsr r r r rrrfmha.attn_biasrrr fmha.commonr fmha.dispatchrrrrPr(rMrDr;rCr'r?r<r=r>r@rArBboolrrrrrrrrr-r-r-r.s*     = "   :  #         4