o i&@sddlZddlmmZddlmZmZGdddejj Z e j Z Gdddejj Z e j ZGdddejj Zej Zdd d Zd d Zd dZdS)N) rearrangerepeatc@$eZdZeddZeddZdS)IndexFirstAxisc Csh|||jdks J|jd|jdd|_}|}tt|ddt|d|dj dg|RS)Nrb ... -> b (...)z -> z dd) save_for_backwardndimshapefirst_axis_dimnumeltorchgatherrrreshape)ctxinputindices other_shape second_dimrU/home/ubuntu/.local/lib/python3.10/site-packages/xformers/_flash_attn/bert_padding.pyforward s zIndexFirstAxis.forwardcCs|j\}|jdks J|jdd}t|d}tj|j|jdg|j|jd}| dt |d|jdd||j |jg|RdfS)Nrrrdevicedtyperr r ) saved_tensorsrrrrzerosrrrscatter_rr)r grad_outputrr grad_inputrrrbackwards zIndexFirstAxis.backwardN__name__ __module__ __qualname__ staticmethodrr%rrrrrs   rc@r)IndexPutFirstAxiscCsZ|||jdks J|jdksJtj|g|jddR|j|jd}|||<|S)Nrrr)r rrr!rrr)rvaluesrroutputrrrr*s  zIndexPutFirstAxis.forwardcCs|j\}||}|ddfSN)r )rr#r grad_valuesrrrr%7s zIndexPutFirstAxis.backwardNr&rrrrr+)   r+c@r)IndexFirstAxisResidualcCsP|||jdks J|jd|jdd|_}|}||}||fS)Nrrr)r rrrrdetach)rrrrrr-rrrrDs  zIndexFirstAxisResidual.forwardcCs|j\}|jdks J|jdd}|jdd|ksJ|}|j|jdgd|jdR}||}|d|||j|jg|RdfS)Nrrr)r)r rrr expand_as scatter_add_r)rr# grad_residualrrr$rrrr%Qs" zIndexFirstAxisResidual.backwardNr&rrrrr1Cr0r1c Cs|dur||n|}|jdtjd}|jdtjd}tj|dd}|}ttj |dtjdd}t t |d|||||fS) a Arguments: hidden_states: (batch, seqlen, ...) attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid. unused_mask: (batch, seqlen), bool / int, 1 means the element is allocated but unused. Return: hidden_states: (total_nnz, ...), where total_nnz = number of tokens selected in attention_mask + unused_mask. indices: (total_nnz), the indices of masked tokens from the flattened input sequence. cu_seqlens: (batch + 1), the cumulative sequence lengths, used to index into hidden_states. max_seqlen_in_batch: int seqused: (batch), returns the number of tokens selected in attention_mask + unused_mask. Nr dimrFas_tuplerrrb s ... -> (b s) ...) sumrint32nonzeroflattenmaxitemFpadcumsumindex_first_axisr) hidden_statesattention_mask unused_mask all_masksseqlens_in_batchused_seqlens_in_batchrmax_seqlen_in_batch cu_seqlensrrr unpad_inputbs  rNc Cs|jdd}|d}tj||j|jdt|||dk}tj | dd }| |}tj | dd }| }t tj|dtjdd } tt|d ||| |fS) a Supports concatenating short samples in one sequence. The attention_mask_in_length is utilized to mask other short samples. It helps efficient training of variant lengths-based samples (e.g., the supervised fine-tuning task in large language model). The motivation for this function is explained [here](https://github.com/Dao-AILab/flash-attention/issues/432#issuecomment-1668822286). For example, if batch = 3 and seqlen = 6, the attention_mask_in_length is: ``` [ [2, 3, 0, 0, 0, 0], [3, 2, 0, 0, 0, 0], [6, 0, 0, 0, 0, 0] ] ``` , which refers to the 3D-attention mask: ``` [ [ [1, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0], [0, 0, 1, 0, 0, 0], [0, 0, 1, 1, 0, 0], [0, 0, 1, 1, 1, 0], [0, 0, 0, 0, 0, 1] ], [ [1, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0], [0, 0, 0, 1, 0, 0], [0, 0, 0, 1, 1, 0], [0, 0, 0, 0, 0, 1] ], [ [1, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0], [1, 1, 1, 1, 0, 0], [1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 1] ] ] ```. Arguments: hidden_states: (batch, seqlen, ...) attention_mask_in_length: (batch, seqlen), int, a nonzero number (e.g., 1, 2, 3, etc.) means length of concatenated sequence in b-th batch, and 0 means none. Return: hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask. indices: (total_nnz), the indices of non-masked tokens from the flattened input sequence. cu_seqlens: (batch + 1), the cumulative sequence lengths, used to index into hidden_states. max_seqlen_in_batch: int r )r7rrFr8rr6r:r;)r<sizerarangerrexpandlen unsqueezer>r?r@rArBrCrDr=rEr) rFattention_mask_in_lengthlengthseqlenattention_mask_2dreal_indices_idxrJrrLrMrrr&unpad_input_for_concatenated_sequencess 4 *  rYcCs(|jd}t||||}t|d|dS)a Arguments: hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask. indices: (total_nnz), the indices that represent the non-masked tokens of the original padded input sequence. batch: int, batch size for the padded sequence. seqlen: int, maximum sequence length for the padded sequence. Return: hidden_states: (batch, seqlen, ...) r z(b s) ... -> b s ...)b)rindex_put_first_axisr)rFrbatchrVr7r-rrr pad_inputs r]r.)rtorch.nn.functionalnn functionalrBeinopsrrautogradFunctionrapplyrEr+r[r1index_first_axis_residualrNrYr]rrrrs ! I