o il@sdZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z m Z mZddlZdejdejfddZd e ed ejdejfd d ZGd ddZdKde ejdejfddZejdfdddde edfd ejde eejfde ededejf ddZeGdddeZGdddeZeGdd d eZeGd!d"d"ZeGd#d$d$eZ eGd%d&d&eZ!eGd'd(d(eZ"eGd)d*d*e"Z#eGd+d,d,e"Z$eGd-d.d.eZ%eGd/d0d0e%Z&eGd1d2d2e%Z'eGd3d4d4eZ(eGd5d6d6e(Z)eGd7d8d8eZ*eGd9d:d:e*Z+eGd;d<dd>e,Z-eGd?d@d@e#Z.eGdAdBdBe$Z/GdCdDdDejeZ0GdEdFdFej1j2Z3GdGdHdHe0Z4GdIdJdJe4Z5ej67e4ej67e5e"e*e%e(e,fZ8dS)LaT This file contains biases that can be used as the `attn_bias` argument in :attr:`xformers.ops.memory_efficient_attention`. Essentially, a bias is a Tensor which will be added to the ``Q @ K.t`` before computing the ``softmax``. The goal of having custom made classes (instead of dense tensors) is that we want to avoid having to load the biases from memory in the kernel, for performance reasons. We also want to be able to know before-hand which parts of the attention matrix we will need to compute (eg causal masks). Some very common biases are LowerTriangularMask and BlockDiagonalMask. N) dataclass) AnyClassVarIterableListOptionalSequenceTupleTypeUnioncasttdevicecCs4|j|kr|S|tdkr||S|j|ddS)NcpuT non_blocking)rtorchto)r rrO/home/ubuntu/.local/lib/python3.10/site-packages/xformers/ops/fmha/attn_bias.py _to_device's  rseqdtypecCs6|tdkrtj||dStj||ddj|ddS)Nr)rT)r pin_memoryr)rrtensorr)rrrrrr_to_device_tensor0src @sLeZdZdZdZejdfdeedfdej de e ej fdej fd d Zd S) AttentionBiasaLBase class for a custom bias that can be applied as the attn_bias argument in :attr:`xformers.ops.memory_efficient_attention`. That function has the ability to add a tensor, the attention bias, to the QK^T matrix before it is used in the softmax part of the attention calculation. The attention bias tensor with shape (B or 1, n_queries, number of keys) can be given as the attn_bias input. The most common use case is for an attention bias is to contain only zeros and negative infinities, which forms a mask so that some queries only attend to some keys. Children of this class define alternative things which can be used as the attn_bias input to define an attention bias which forms such a mask, for some common cases. When using an :attr:`xformers.ops.AttentionBias` instead of a :attr:`torch.Tensor`, the mask matrix does not need to be materialized, and can be hardcoded into some kernels for better performance. See: - :attr:`xformers.ops.fmha.attn_bias.LowerTriangularMask` - :attr:`xformers.ops.fmha.attn_bias.LowerTriangularFromBottomRightMask` - :attr:`xformers.ops.fmha.attn_bias.LowerTriangularMaskWithTensorBias` - :attr:`xformers.ops.fmha.attn_bias.BlockDiagonalMask` - :attr:`xformers.ops.fmha.attn_bias.BlockDiagonalCausalMask` Frshape.rrreturncCtz Materializes the bias as a `torch.Tensor`. This is very slow and we don't attempt to make it fast. Only use for debugging/testing. Shape should be like `[*, q_seqlen, k_seqlen]` NotImplementedErrorselfrrrrrr materialize[ zAttentionBias.materializeN)__name__ __module__ __qualname____doc__HOLDS_DENSE_TENSORrfloat32r intrr strrTensorr%rrrrr7s!  rrcCs*|durtjrtdStdS|S)Ncudar)rr0 is_availablerrrrr_get_default_bias_devicejs    r3rF) window_sizefrom_bottomrightr.r4r5c Cs|tjur|ntj}tj||d|d}|dd\}}d} |r#||} tj|| d|} |dur 0` but got window_left=ztInvalid window value passed to `LocalAttentionFromBottomRightMask`: expected`window_right > 0` but got window_right=)rH ValueErrorrIr$rrr __post_init__s  z/LocalAttentionFromBottomRightMask.__post_init__rr.rrc Csz|tjur|ntj}tj||d|d}|dd\}}||}tj|||jd}tj|||jd}t|}| |S)Nr6r7r9r:) rr<r,r=r?rHr>rIr@r) r$rrrrArErBrCrDrrrr%s  z-LocalAttentionFromBottomRightMask.materialize)rrGrN)r'r(r)r*r-__annotations__rrNrr,r rr r.rr/r%rrrrrGs" )    rGc @sneZdZdZdejddfddZejdfdee dfd ej de e ejfdej fd d Zd e dd fddZdS)"LowerTriangularFromBottomRightMaska A causal masking. This mask is exactly the same as :attr:`LowerTriangularMask` when there is the same number of queries and keys. When the number of queries is different from the number of keys, it is a triangular mask shifted so that the last query can attend to the last key. In other words, a query Q cannot attend to a key which is nearer the final key than Q is to the final query. .. figure:: /_static/causal_bottom_right.png The difference between :attr:`LowerTriangularMask` (left) and :attr:`LowerTriangularFromBottomRightMask` (right). They become equivalent if the number of queries equals the number of keys. rrcCt|tus Jd|SNPlease implement in subclass)typerQrKrrrrsz%LowerTriangularFromBottomRightMask.torr.rcCst|||ddS)NT)rrr5rFr#rrrr%sz.LowerTriangularFromBottomRightMask.materializer40LowerTriangularFromBottomRightLocalAttentionMaskcCst|S)z Create a new bias which combines local + causal attention. See :attr:`LowerTriangularFromBottomRightLocalAttentionMask` )rWr$r4rrrmake_local_attentionsz7LowerTriangularFromBottomRightMask.make_local_attentionN)r'r(r)r*rrrr,r r-rr r.r/r%rYrrrrrQs&     rQc @speZdZUdZeed<dejddfddZddd Z ej d fd e ed fd ej de eejfdejfddZdS)rWaT A mask that combines both :attr:`LowerTriangularFromBottomRightMask` and local attention. A query whose distance from the final query is X cannot attend to a key whose distance to the final key is either of: * less than X (i.e. "causal attention", same as :attr:`LowerTriangularFromBottomRightMask`) * greater than X + window_size (i.e. "local attention") .. figure:: /_static/causal_bottom_right_local.png The mask from :attr:`LowerTriangularFromBottomRightLocalAttentionMask`. The green area is calculated, and the grey area is masked out. _window_sizerrcCrRrS)rUrWrKrrrr*sz3LowerTriangularFromBottomRightLocalAttentionMask.toNcCs|jdkr td|jdSNr,Expected `window_size > 0`, but window_size=)rZrLrMrrrrN2s  z>LowerTriangularFromBottomRightLocalAttentionMask.__post_init__rr.rcCt||||jddSNT)rrr4r5rFrZr#rrrr%8zsz%_SeqLenInfo.split..dim)rerrLrprorgsplit)r$rvrw split_chunksit batch_sizerrrrs("  z_SeqLenInfo.splitrJ)r'r(r)r*rr/rPr-rrrrr rh classmethodrtrrurrrrrrraGsB       rac seZdZUdZejed<eeed<eed<dddZ d ej ddfd d Z de e eefffd d Zeddde ed eej ddfddZedddeeded eej ddfddZ ddejdeeedeejfddZZS)_PaddedSeqLenInfouW (Internal) Represents the division of a dimension into blocks which are padded out to the same total length. For example, to represent a dimension of length 12 with space for three blocks of length 4, but where the occupied lengths are 2, 3 and 2, use `from_seqlens_padded([2, 3, 2], 4)`. The layout along the dimension is 0 ─► block 0 block 0 4 ─► block 1 block 1 block 1 8 ─► block 2 block 2 12 ─► The members will be: max_seqlen: 3 min_seqlen: 2 seqstart_py: [0, 4, 8, 12] seqstart: torch.IntTensor([0, 4, 8, 12]) seqlen_py: [2, 3, 2] seqlen: torch.IntTensor([2, 3, 2]) padding: 4 rs seqlen_pypaddingrNcCs t|jt|jdksJdSrf)rprerrMrrrrNs z_PaddedSeqLenInfo.__post_init__rc CsTt|tus Jd|jj|kr|Stt|j||j|j|jt|j||j |j dS)NrT)rbrcrdrersrr) rUrrsrrrbrcrdrerrrKrrrrs   z_PaddedSeqLenInfo.toc#4tt|jD] \\}}}|||fVq dSrJrgsuperrhrr$start_length __class__rrrhz_PaddedSeqLenInfo.intervalsr2riracCtd)NzPUse either `_SeqLenInfo.from_seqlens` or `_PaddedSeqLenInfo.from_seqlens_padded`) RuntimeErrorrrrirrrrrusz_PaddedSeqLenInfo.from_seqlensc st|tjrJtfdd|DsJd|dt|}ttdt|d}t|tj |d}||t|tr@|nt|t |t |t|tj |d|dS) zz Input tensors are assumed to be in shape [B, M, *] seqstart = padding * torch.arange(batch_size) c3s|]}|kVqdSrJr)r{rsrrr s z8_PaddedSeqLenInfo.from_seqlens_padded..zSeqlens z Padding rr6rk)rsrrcrdrbrer) rlrr/allr3listrangerprrqrnrm)rrrirrrersrrrfrom_seqlens_paddeds$  z%_PaddedSeqLenInfo.from_seqlens_paddedrvrwcCr)N_PaddedSeqLenInfo.splitr!r$rvrwrrrrrrOrJ)r'r(r)r*rr/rPrr-rNrrrr rhrrrurrr __classcell__rrrrrsH  "     rc seZdZUdZejed<eeed<dej ddfddZ de e eefffdd Z ed d d e edeej dd fddZedeed eededej ddf ddZ ddejdeeedeejfddZZS) _GappySeqInfou (Internal) Flexible equivalent of _PaddedSeqLenInfo. There are two distinct semantics. (1) For non-paged masks: Represents the division of a dimension into blocks which are anywhere. Each just has a start and a length. The final start is the total length of the dimension. For example, to represent a dimension of length 14 like follows with three occupied lengths of 6, 3 and 1, use `from_seqlens_padded([0, 7, 12, 14], [6, 3, 1])`. The layout along the dimension is 0 ─► block 0 block 0 block 0 block 0 4 ─► block 0 block 0 block 1 8 ─► block 1 block 1 12 ─► block 2 The members will be: max_seqlen: 6 min_seqlen: 1 seqstart_py: [0, 7, 12, 14] seqstart: torch.IntTensor([0, 7, 12, 14]) seqlen_py: [6, 3, 1] seqlen: torch.IntTensor([6, 3, 1]) (2) For paged masks: The notional space is divided into batch-size-many blocks. seqstart and seqstart_py is an offset in the block, not in the whole space, and doesn't have an extra last element. Otherwise as above. rsrrrcCsPt|tus Jd|jj|kr|Stt|j||j|j|jt|j||j dS)NrT)rbrcrdrersr) rUrrsrrrbrcrdrerrKrrrrJs   z_GappySeqInfo.toc#rrJrrrrrrhYrz_GappySeqInfo.intervalsNr2riracCrrJr!rrrrru]sz_GappySeqInfo.from_seqlens seqstartspagedc Cst|tjrJt|}t|dkrtdt|t||r!dndkr7|r(dnd}td|d|d|t|tj|d }|||t|t |t|tj|d |d S) Nrz No elementsr6z1 + zlen(seqstarts)=z should be z len(seqlens)=rk)rsrrcrdrbre) rlrr/rrprLrrqrnrm)rrrrirrreextrarsrrrfrom_seqlens_gappycs$   z _GappySeqInfo.from_seqlens_gappyrvrwcCr)N_GappySeqInfo.splitr!rrrrrrrrJ)r'r(r)r*rr/rPrr-rrrr rhrrruboolrrrrrrrrrsF  -    rc@seZdZUdZeed<eed<dZeee ed<d3ddZ e j d fd e e d fd e jd eee jfde jfddZe j d fd e e d fd e jd eee jfde jfddZe d4dddee deee d ee jddfddZedee jde de jffddZe d4dee jdee jdeee jde de je jee jffddZde jdee jfd d!Zde jdee jfd"d#Zde jdee jfd$d%Zd5d'd(Zd6d*d+Zd,e dd-fd.d/Zd,e dd0fd1d2ZdS)7BlockDiagonalMaska A block-diagonal mask that can be passed as ``attn_bias`` argument to :attr:`xformers.ops.memory_efficient_attention`. Queries and Keys are each divided into the same number of blocks. Queries in block i only attend to keys in block i. .. figure:: /_static/block_diag_bias.png This bias can be used to handle a batch of sequences of different lengths, via :attr:`BlockDiagonalMask.from_tensor_list` :Example: .. code-block:: python import torch from xformers.ops import fmha K = 16 dtype = torch.float16 device = "cuda" list_x = [ torch.randn([1, 3, 1, K], dtype=dtype, device=device), torch.randn([1, 6, 1, K], dtype=dtype, device=device), torch.randn([1, 2, 1, K], dtype=dtype, device=device), ] attn_bias, x = fmha.BlockDiagonalMask.from_tensor_list(list_x) linear = torch.nn.Linear(K, K * 3).to(device=device, dtype=dtype) q, k, v = linear(x).reshape([1, -1, 1, 3, K]).unbind(-2) out = fmha.memory_efficient_attention(q, k, v, attn_bias=attn_bias) list_out = attn_bias.split(out) print(list_out[0].shape) # [1, 3, 1, K] assert tuple(list_out[0].shape) == (1, 3, 1, K) q_seqinfo k_seqinfoN _batch_sizesrcC4t|tus Jdt|j||j||jdSNrTrrr)rUrrrrrrKrrrr   zBlockDiagonalMask.torr.rrcCstj|||dSNrk)rzerosr#rrr_create_block_masks z$BlockDiagonalMask._create_block_maskc Cs|d|jjdksJ|d|jjdf|d|jjdks,J|d|jjdftj|dd||d}|tj tt |j |j D] \}\\}}\}} |j ||| |f||d||||| f<qLt t |dD]} |d}qu||S)8Materialize the attention bias - for debugging & testingrjr9Nrkrxr)rrerremptyfill_mathinf enumeratergrhrrrp unsqueezeexpand r$rrrrEiq_startq_endk_startk_endrrrrr%s0    zBlockDiagonalMask.materializer2q_seqlen kv_seqlencCsbt|}|dust|t|ksJtj||d}|dus!||kr$|}ntj||d}|||dS)aCreates a :attr:`BlockDiagonalMask` from a list of tensors lengths for query and key/value. Args: q_seqlen (Union[Sequence[int], torch.Tensor]): List or tensor of sequence lengths for query tensors kv_seqlen (Union[Sequence[int], torch.Tensor], optional): List or tensor of sequence lengths for key/value. (Defaults to ``q_seqlen``.) Returns: BlockDiagonalMask Nr2rr)r3rpraru)rrrrrrrrrrrus zBlockDiagonalMask.from_seqlenstensorsc Csxdd|D}g}|D]}t|jdD] }||jdqq ||}||_tdd|D}tj|dd}||fS)aRCreates a :attr:`BlockDiagonalMask` from a list of tensors, and returns the tensors concatenated on the sequence length dimension .. figure:: /_static/block_diag_cat_split.png See also :attr:`BlockDiagonalMask.split` to split the returned :attr:`torch.Tensor` back to a list of tensors of varying sequence length Args: tensors (Sequence[torch.Tensor]): A list of tensors of shape ``[B, M_i, *]``. All tensors should have the same dimension and the same batch size ``B``, but they can have different sequence length ``M``. Returns: Tuple[BlockDiagonalMask, torch.Tensor]: The corresponding bias for the attention along with `tensors` concatenated on the sequence length dimension, with shape ``[1, sum_i{M_i}, *]`` cSg|]}|jdqSrrr{rrrrr}z6BlockDiagonalMask.from_tensor_list..rr6css,|]}|ddg|jddVqdSr6rjrxNryr{rvrrrrs*z5BlockDiagonalMask.from_tensor_list..r~)rrrorurtuplercat) rrrrwrirvr block_diag tensors_bs1concat_tensorsrrrfrom_tensor_lists z"BlockDiagonalMask.from_tensor_list tensors_q tensors_k tensors_vc CsDt|t|ks J|dust|t|ksJdd|D}gg}}tt||D]B\}\}} |jd| jdks=J||jdg|jd7}|| jdg| jd7}|dusm||jdd| jddksmJq+|||} || _| tjdd|Dddtjdd|Ddd|durtjd d|DddfSdfS) NcSrrrrrrrr})rz;BlockDiagonalMask.from_tensor_lists_qkv..rr6rxcS(g|]}|ddg|jddqSrryrrrrr}4(r~cSrrryrrrrr}5rcSrrryrrrrr}6r)rprrgrrurrr) rrrrrrw q_seqlens kv_seqlensrqkrrrrfrom_tensor_lists_qkv s( , z'BlockDiagonalMask.from_tensor_lists_qkvrcC|j||jSrJ)rrrr$rrrr split_queries;zBlockDiagonalMask.split_queriescCrrJ)rrrrrrrsplit_kv>rzBlockDiagonalMask.split_kvcCs |j|jusJ|j||jS)a%The inverse operation of :attr:`BlockDiagonalCausalMask.from_tensor_list` Args: tensor (torch.Tensor): Tensor of tokens of shape ``[1, sum_i{M_i}, *]`` Returns: Sequence[torch.Tensor]: A list of tokens with possibly different sequence lengths )rrrrrrrrrAs zBlockDiagonalMask.splitBlockDiagonalCausalMaskcCt|j|j|jdS)zMakes each block causalr)rrrrrMrrr make_causalM zBlockDiagonalMask.make_causal&BlockDiagonalCausalFromBottomRightMaskcCr)z9Makes each block causal with a possible non-causal prefixr)rrrrrMrrrmake_causal_from_bottomrightUrz.BlockDiagonalMask.make_causal_from_bottomrightr4%BlockDiagonalCausalLocalAttentionMaskcCt|j|j|j|dS)z:Experimental: Makes each block causal with local attentionrrrrZ)rrrrrXrrrrY] z&BlockDiagonalMask.make_local_attention4BlockDiagonalCausalLocalAttentionFromBottomRightMaskcCr)zSExperimental: Makes each block causal with local attention, start from bottom rightr)rrrrrXrrr%make_local_attention_from_bottomrighthrz7BlockDiagonalMask.make_local_attention_from_bottomright)rrrJrrrr) r'r(r)r*rarPrrrr-rrr,r rr r.rr/rr%rrurrrrrrrrYrrrrrrs &                   rc @sReZdZdZd ddZejdfdeedfdej d e e ej fdej fd d Zd S)rah Same as :attr:`xformers.ops.fmha.attn_bias.BlockDiagonalMask`, except that each block is causal. Queries and Keys are each divided into the same number of blocks. A query Q in block i cannot attend to a key which is not in block i, nor one which is farther from the initial key in block i than Q is from the initial query in block i. rcCrr)rUrrrrrrKrrrrrzBlockDiagonalCausalMask.torr.rrcCtj|||dSr)LowerTriangularMaskr%r#rrrrs z*BlockDiagonalCausalMask._create_block_maskNr)r'r(r)r*rrr,r r-rr r.rr/rrrrrrts    rc @s\eZdZdZdddZdddZejdfd ee d fd ej d e e ej fdejfd dZdS)ra Same as :attr:`xformers.ops.fmha.attn_bias.BlockDiagonalMask`, except that each block is causal. This mask allows for a non-causal prefix NOTE: Each block should have `num_keys >= num_queries` otherwise the forward pass is not defined (softmax of vector of `-inf` in the attention) Queries and keys are each divided into the same number of blocks. A query Q in block i cannot attend to a key which is not in block i, nor one which nearer the final key in block i than Q is to the final query in block i. rcCrr)rUrrrrrrKrrrr  z)BlockDiagonalCausalFromBottomRightMask.toNc Cshtt|j|jD]$\}\\}}\}}||}||}||kr1td|d|d|dq dS)NzBlock #z has num_keys=z and num_queries=z$. Expected `num_keys >= num_queries`)rrgrrhrrL)r$rrrrrrBrCrrrrNsz4BlockDiagonalCausalFromBottomRightMask.__post_init__rr.rrcCrNrrrrQr%r#rrrrz9BlockDiagonalCausalFromBottomRightMask._create_block_maskrrO)r'r(r)r*rrNrr,r r-rr r.rr/rrrrrrs   rc@seZdZUdZeed<eed<dddZej dfde e d fd ej d e eejfdejfd d Zej dfde e d fd ej d e eejfdejfddZe ddddee de dee ded eejddf ddZdejde dedddfddZdS) BlockDiagonalPaddedKeysMaskax Same as :attr:`xformers.ops.fmha.attn_bias.BlockDiagonalMask`, except we support padding for k/v The keys and values are divided into blocks which are padded out to the same total length. For example, if there is space for 12 keys, for three blocks of max length 4, but we only want to use the first 2, 3 and 2 of each block, use `kv_padding=4` and `kv_seqlens=[2, 3, 2]`. The queries are divided into blocks, without padding, of lengths given by q_seqlen. A query Q in block i cannot attend to a key which is not in block i, nor one which is not in use (i.e. in the padded area). rrrcC0t|tus Jdt|j||j|dSNrTr)rUrrrrrKrrrr   zBlockDiagonalPaddedKeysMask.torr.rrcCstjd||dS)Ng)rr)rrr#rrrrsz.BlockDiagonalPaddedKeysMask._create_block_maskc Cs|d|jjdkrtd|d|jjdkrtdtj|dd||d}|tj t t |j |j D] \}\\}}\}} |j ||| |f||d||||| f<qBlockDiagonalCausalWithOffsetPaddedKeysMask._create_block_maskr2rrrcCsV|dust|t|ksJ||ft|}tj||d}tj|||d}|||dS)a#Creates a :attr:`BlockDiagonalCausalWithOffsetPaddedKeysMask` from a list of tensor lengths for query and key/value. Args: q_seqlen (Sequence[int]): List or tensor of sequence lengths for query tensors kv_padding (int): Padding for k/v - also an upperbound on each individual key length kv_seqlen (Sequence[int]): List or tensor of sequence lengths for key/value. causal_diagonal: unused, for BC only Returns: BlockDiagonalCausalWithOffsetPaddedKeysMask Nr2r)rpr3rarurrrrrrru_s z8BlockDiagonalCausalWithOffsetPaddedKeysMask.from_seqlens)rrrJ)r'r(r)r*rrrPrrr,r r-rr r.rr/rrrrrurrrrr6s@        rc @seZdZUdZeed<dddZejdfde edfd ej d e e ej fdejfd d Zed eededeededdf ddZdS)/BlockDiagonalCausalLocalAttentionPaddedKeysMaska Like :attr:`xformers.ops.fmha.attn_bias.BlockDiagonalCausalWithOffsetPaddedKeysMask`, except with a window size. A query Q in block i cannot attend to a key which is not in block i, nor one which is not in use (i.e. in the padded area), nor one which is nearer to the final key in block i than Q is to the final query in block i, nor one that is more than window_size further from the final key in block i than Q is to the final query in block i. rZrcCr)NrTrrrZ)rUrrrrrZrKrrrrrz2BlockDiagonalCausalLocalAttentionPaddedKeysMask.torr.rrcCr])NT)rrrr4r5r_r#rrrrr`zBBlockDiagonalCausalLocalAttentionPaddedKeysMask._create_block_maskrrrr4cCsH|dust|t|ksJ||ft|}t||}||||dS)Nr)rprarurr)rrrrrr4rrrrrfrom_seqlens_locals  zBBlockDiagonalCausalLocalAttentionPaddedKeysMask.from_seqlens_localN)rr)r'r(r)r*r-rPrrr,r rr r.rr/rrrr rrrrrs6      rc@seZdZUdZeed<eed<ejed<e ed<e Z e e e ed<dejddfd d Zejd fd ee d fdejdeeejfdejfddZedddee dee dejde deejddf ddZdS)ra Same as BlockDiagonalPaddedKeysMask, but for paged attention. block_tables has shape [batch_size, max_num_pages] and K/V have shape [1, max_num_pages * page_size, num_heads, head_dim] or [1, max_num_pages * page_size, num_groups, num_heads, head_dim] rrrr _UNPAGED_TYPErrcC>t|tus Jdt|j||j|t|j||jdSNrTr)rUrrrrrrrrKrrrr   z#PagedBlockDiagonalPaddedKeysMask.torr.rcCs2|jjd|j}|j|jt|jj|d}| |||}t t |j d}||j}tj|jdd|f||d} | tj t|jD]I\} \} } t|jjdD]:} t t |j| |  }| || |j}||j}||j}||j}|d| | ||f| d| | ||f<q[qM| S)rr6rNrjrk.)rrrr rrrrrr%r r-rnitemrrrrrrrhr)r$rrr max_row_len bias_nonpaged mask_nonpaged n_used_blocksmax_physical_len mask_pagedbrrlogical_page_idxphysical_page_idxk_logical_start k_logical_endk_physical_startk_physical_endrrrr%s:      z,PagedBlockDiagonalPaddedKeysMask.materializeNr2rrcCs\t|t|ksJ||ft|}tj||d}tj||jd||d}|||||dS)a Creates a :attr:`PagedBlockDiagonalPaddedKeysMask` from a list of tensor lengths for query and key/value. Args: q_seqlen (Sequence[int]): List or tensor of sequence lengths for query tensors kv_padding (int): Padding for k/v - also an upperbound on each individual key length kv_seqlen (Sequence[int]): List or tensor of sequence lengths for key/value. causal_diagonal: unused, for BC only Returns: PagedBlockDiagonalPaddedKeysMask r2r6)rrr)rpr3rarurrr)rrrrrrrrrrrrrusz-PagedBlockDiagonalPaddedKeysMask.from_seqlens)r'r(r)r*rarPrrr/r-rr rr rrr,r rr r.r%rrrrurrrrrsJ     'rc@(eZdZdZeZdejddfddZdS)0PagedBlockDiagonalCausalWithOffsetPaddedKeysMaska  Same as BlockDiagonalCausalWithOffsetPaddedKeysMask, but for paged attention. block_tables has shape [batch_size, max_num_pages] and K/V have shape [1, max_num_pages * page_size, num_heads, head_dim] or [1, max_num_pages * page_size, num_groups, num_heads, head_dim] rrcCr r )rUrrrrrrrrKrrrr0   z3PagedBlockDiagonalCausalWithOffsetPaddedKeysMask.toN) r'r(r)r*rr rrrrrrrr#src @seZdZUdZeed<eed<dejddfddZ ej dfd e e d fd ej deeejfdejfd d Zedddee dee dee deejddf ddZdejde de deddef ddZdS)BlockDiagonalGappyKeysMaskz Same as :attr:`xformers.ops.fmha.attn_bias.BlockDiagonalMask`, except k/v is gappy. A query Q in block i only attends to a key which is in block i. rrrrcCrr)rUrrrrrKrrrrJrzBlockDiagonalGappyKeysMask.torr.rc Cs|d|jjdkrtd||jf|d|jjdkr$td||jftj|dd||d}|tj t |j |j D]\\}}\}}d|||||f<qBt t |dD]} | d}q]||S) rrjrr9rNrkrrx)rrerLrrrrrrrgrhrrprr) r$rrrrErrrrrrrrr%Qs  z&BlockDiagonalGappyKeysMask.materializeNr2r kv_seqstartsrcCsPt|t|ksJ||ft|}tj||d}tj||d|d}|||dS)ztCreates a :attr:`BlockDiagonalGappyKeysMask` from a list of tensor lengths for query and key/value. r2Fr)rpr3rarurr)rrrr rrrrrrrrugs  z'BlockDiagonalGappyKeysMask.from_seqlensrrnotional_paddingrPagedBlockDiagonalGappyKeysMaskcs|jd|fddt|jjddD}tfdd|Ds%Jtj||jjd|jd }|jj ks9J||j |||d }|S) z Assuming our keys actually live in separate blocks of length notional_padding, convert to a Paged version. r6csg|] \}}||qSrrr{rr)r!rrr} z9BlockDiagonalGappyKeysMask.make_paged..Nrjc3s(|]}d|ko knVqdS)rNr)r{rrrrrs&z8BlockDiagonalGappyKeysMask.make_paged..Tr2r) rrrrerrrrrrcr)r$rrr!r new_seqstartsrrr)rr!rr~s  z%BlockDiagonalGappyKeysMask.make_paged)r'r(r)r*rarPrrrrr,r r-rr r.r/r%rrrrur rrrrrrr>sP      rc @s\eZdZdZdejddfddZejdfdee dfd ej de e ejfdej fd d Zd S) *BlockDiagonalCausalWithOffsetGappyKeysMaska Same as :attr:`xformers.ops.fmha.attn_bias.BlockDiagonalCausalMask`, except k/v is gappy. A query Q in block i cannot attend to a key which is not in block i, nor one which is nearer to the final key in block i than Q is to the final query in block i. rrcCrr)rUr'rrrrKrrrrrz-BlockDiagonalCausalWithOffsetGappyKeysMask.torr.rc Cs|d|jjdkrtd|d|jjdkrtdtj|dd||d}|tj t t |j |j D]!\}\\}}\}} t j ||| |f||d||||| f<q.rjFr2rNrk.)rrrrrrer rrrrrrr%r r-rnrrrrrrhr)r$rrrr&rrrrrrrrrrrrrrrr%rr%sJ        z+PagedBlockDiagonalGappyKeysMask.materializeNr2rr rc Csrt|t|krt|ksnJ|||f|dur|jn|}tj||d}tj||d|d}|||||dS)aCreates a :attr:`PagedBlockDiagonalGappyKeysMask` from a list of tensor lengths for query and key/value. Note that unlike :attr:`BlockDiagonalGappyKeysMask`, kv_seqstarts is addressing in a different space for each batch element. For example if you were doing a BlockDiagonalPaddedKeysMask with two batch elements and padding=100, but wanted to change it so that the first key is ignored, then you would use BlockDiagonalGappyKeysMask with kv_seqstarts [1, 101, 200]. But if you were using PagedBlockDiagonalPaddedKeysMask but wanted to ignore the first key, you would provide this function with kv_seqstarts = [1, 1]. Nr2Tr)rprrarurr) rrrr rrrrrrrrrrus $z,PagedBlockDiagonalGappyKeysMask.from_seqlens)r'r(r)r*rarPrrr/r-rr rr rrr,r rr r.r%rrrrurrrrr"sN     .  r"c@r)/PagedBlockDiagonalCausalWithOffsetGappyKeysMaska Same as BlockDiagonalCausalWithOffsetGappyKeysMask, but for paged attention. block_tables has shape [batch_size, max_num_pages] and K/V have shape [1, max_num_pages * page_size, num_heads, head_dim] or [1, max_num_pages * page_size, num_groups, num_heads, head_dim] rrcCr r )rUr(rrrrrrrKrrrrMrz2PagedBlockDiagonalCausalWithOffsetGappyKeysMask.toN) r'r(r)r*r'r rrrrrrrr(Bsr(c @sheZdZUdZdZeed<dddZddZe j d fd e ed fd e j d e ee jfde jfddZdS)r (Experimental feature) Same as :attr:`xformers.ops.fmha.attn_bias.BlockDiagonalCausalMask`. This makes the mask "local" and the attention pattern banded. Query i only attends to keys in its block and cannot attend keys further than "window_size" from it. rrZrcC8t|tus Jdt|j||j||j|jdSNrTr)rUrrrrrrZrKrrrrh  z(BlockDiagonalCausalLocalAttentionMask.tocCs|jdkr td|jddt|jjdd|jjddD}ddt|jjdd|jjddD}t||D]\}}||j|krWtd|d |d |jq>dS) Nrr\cSg|]\}}||qSrrr{rvyrrrr}xzGBlockDiagonalCausalLocalAttentionMask.__post_init__..rjr6cSr-rrr.rrrr}~r0z!No keys are attended in q_seqlen z k_seqlen z with sliding window )rZrLrgrrerr)r$rrrrrrrrNss*  z3BlockDiagonalCausalLocalAttentionMask.__post_init__rr.rrcCst||||jdS)N)rrr4r_r#rrrrs z8BlockDiagonalCausalLocalAttentionMask._create_block_maskN)rr)r'r(r)r*rZr-rPrrNrr,r rr r.rr/rrrrrr[s     rc speZdZUdZdZeed<dddZfddZe j d fd e ed fd e j d e ee jfde jfddZZS)rr)rrZrcCr*r+)rUrrrrrrZrKrrrrr,z7BlockDiagonalCausalLocalAttentionFromBottomRightMask.tocs(t|jdkrtd|jdSr[)rrNrZrLrMrrrrNs   zBBlockDiagonalCausalLocalAttentionFromBottomRightMask.__post_init__rr.rrcCr]r^r_r#rrrrr`zGBlockDiagonalCausalLocalAttentionFromBottomRightMask._create_block_mask)rr)r'r(r)r*rZr-rPrrNrr,r rr r.rr/rrrrrrrs      rc seZdZUdZejed<edddddZdfdd Z d d Z e dd dZ ddZ e ddZejdfdeedfdejdeeejfdejfddZZS)AttentionBiasSubTensorF _subtensorNr2rcKrrJr!)rrr2rkwargsrrr__new__szAttentionBiasSubTensor.__new__rcstdSrJ)r__init__)r$argsr4rrrr6szAttentionBiasSubTensor.__init__cCs |jjSrJ)rr'rMrrr__repr__ zAttentionBiasSubTensor.__repr__rcCs`|pi}|jtjjjtjjjtjjjtjjjfvr.|||djg|ddRi|dSt S)Nrr6r2) _overloadpacketropsatenclonedetach_to_copyrr2NotImplemented)rrfunctypesr7r4rrr__torch_dispatch__s*z)AttentionBiasSubTensor.__torch_dispatch__cCs dgdfS)Nr2rrMrrr__tensor_flatten__r9z)AttentionBiasSubTensor.__tensor_flatten__cCs|dusJ||ddS)Nr2r:r)rr inner_tensorsmeta outer_size outer_striderrr__tensor_unflatten__s z+AttentionBiasSubTensor.__tensor_unflatten__rr.rrcCrr r!r#rrrr%r&z"AttentionBiasSubTensor.materializerOrN)r'r(r)r+rr/rP staticmethodr5r6r8rrDrErJr,r r-rr r.rr%rrrrrr1s.       r1c@s$eZdZeddZeddZdS) _AddDenseBiascCst|tusJt|SrJ)rUr!LowerTriangularMaskWithTensorBias)ctx causal_biasrrrrforwardsz_AddDenseBias.forwardcCsd|fSrJr)rOgrad_outrrrbackwardsz_AddDenseBias.backwardN)r'r(r)rLrQrSrrrrrMs  rMc @steZdZdZdZedddddZejdfde e d fd ej d e e ejfd ejfd dZdejd dfddZdS)ra! A lower-triangular (aka causal) mask A query Q cannot attend to a key which is farther from the initial key than Q is from the initial query. See also :attr:`LowerTriangularFromBottomRightMask` if the number of queries is not equal to the number of keys/values. FNrr3cKs:|dur tjd|d}tjj|g|j|jdd}||_|S)zg Note: create on CPU by default to avoid initializing CUDA context by mistake. Nrr2Frr requires_grad)rrr/_make_wrapper_subclassrrr2)rrr2rr4rrrrr5szLowerTriangularMask.__new__r.rrrcCst|||dSrrVr#rrrr%&szLowerTriangularMask.materializebiasrNcCs t||S)zS Creates a new causal mask with an arbitrary ``torch.Tensor`` bias )rMapply)r$rWrrradd_bias.s zLowerTriangularMask.add_bias)r'r(r)r*r+rLr5rr,r r-rr r.rr/r%rYrrrrrs"    rc sneZdZdZdZeddZejdfde e dfdej d e e ejfd ejffd d ZedddZZS)rNz:A lower-triangular (aka causal) mask with an additive biasTcCs(tjj||j|j|j|jd}||_|S)NrT)rr/rVrrrrUr2)rrrWrrrrr5:sz)LowerTriangularMaskWithTensorBias.__new__rr.rrrcstj|||d|jSr)rr%r2r#rrrr%Fsz-LowerTriangularMaskWithTensorBias.materializerNc sz|pi}|jtjjjtjjjtjjjtjjjtjjjtjjj tjjj tjjj fvr;|fdd|Di|}|St S)Ncs g|] }t|r |jn|qSr)rlr2)r{arrrrr}\s zHLowerTriangularMaskWithTensorBias.__torch_dispatch__..) r;rr<r=rselectslicer>r?r@rviewrA)rrrBrCr7r4outputrr[rrDNs$ z4LowerTriangularMaskWithTensorBias.__torch_dispatch__rK)r'r(r)r*r+rLr5rr,r r-rr r.rr/r%rrDrrrrrrN5s$   rNrJ)9r*r dataclassesrtypingrrrrrrr r r r rr/rrr-rrrr3r,r.rrFrGrQrWrarrrrrrrrrrrr'r"r(rrr1autogradFunctionrMrrN_dynamoallow_in_graph VARLEN_BIASESrrrrs 0  3    U/ 3`loo1oI8i ]1s> /2 . .