o wiQd(@sNddlZddlZddlZddlmZmZddlZddlmm Z ddl m Z m Z mZmZmZmZeeZdZddZdLddZd d ZdZe rmdd lmZdd lmZdd lmZ ddlm!Z"ddl#m$Z$dZ%dZn2erddl&m'Z$ddl&m(Zddl&m)Zdd l&mZ ddl&m!Z"dZ%dZn dZdZdZ dZ"dZ$dZ%e rdd l*mZ+dd l*mZ,eZ-eZ.dZ/dZn dZ+dZ,dZ-dZ.dZ/ere0dedZe0dedZe0deZ!e0deZdZ1erde2e3ej4vZ1ddZ5dd Z6d!ej7d"e8ej7ej7e9ffd#d$Z:d%ej7d&ej7d'ej7d!ej7d(e9f d)d*Z;d+d,Zfd3d4Z?ed5Z@daA 6            dMd7ej7d8ej7d9ej7d!eej7d(e9d:eBd;eCdee9d?eBd@eeCdAeeBdBeejDdCeejDdDee9dEee9d2eej>dFeeEf&dGdHZFGdIdJdJeddKZGdS)NN)Optional TypedDict)is_flash_attn_2_availableis_flash_attn_3_availableis_flash_attn_greater_or_equal#is_flash_attn_greater_or_equal_2_10is_torch_npu_availableloggingcCs$|jdg|jddR}||S)a A local implementation of the PyTorch indexing operation `tensor[indices]` on the first axis, after flattening the first two dimensions of the tensor. This is functionally equivalent to FA2's `index_first_axis` and replaces the need to import it. N)reshapeshape)tensorindicesreshaped_tensorrh/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/modeling_flash_attention_utils.py_index_first_axis%src Cs|dur||n|}|jdtjd}|jdtjd}tj|dd}|}ttj |dtjdd}t ||||||fS)a FA3-compatible unpad_input function. Arguments: hidden_states: (batch, seqlen, ...) attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid. unused_mask: (batch, seqlen), bool / int, 1 means the element is allocated but unused. Return: hidden_states: (total_nnz, ...), where total_nnz = number of tokens selected in attention_mask + unused_mask. indices: (total_nnz), the indices of masked tokens from the flattened input sequence. cu_seqlens: (batch + 1), the cumulative sequence lengths, used to index into hidden_states. max_seqlen_in_batch: int seqused: (batch), returns the number of tokens selected in attention_mask + unused_mask. Nr dimdtypeFas_tuplerrr) sumtorchint32nonzeroflattenmaxitemFpadcumsumr) hidden_statesattention_mask unused_mask all_masksseqlens_in_batchused_seqlens_in_batchrmax_seqlen_in_batch cu_seqlensrrr_fa3_unpad_input1s r-cCsL|jdd}tj||g|R|j|jd}|||<|j||g|RS)a FA3-compatible pad_input function. Arguments: hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask. indices: (total_nnz), the indices that represent the non-masked tokens of the original padded input sequence. batch: int, batch size for the padded sequence. seqlen: int, maximum sequence length for the padded sequence. Return: hidden_states: (batch, seqlen, ...) rNdevicer)rrzerosr/rview)r%rbatchseqlenroutputrrr_fa3_pad_inputPs "r5)flash_attn_func)flash_attn_varlen_func) pad_input) unpad_input)apply_rotary_embTr )npu_apply_rotary_emb)npu_flash_attn_func)npu_flash_attn_varlen_funcF flash_attn__func _varlen_funcunpad_input_fa pad_input_fa window_sizecCs"trdStr dStrdSdS)z5Determine whether flash-attention can be used or not.TF)rrr rrrris_flash_attn_availablesrEcCs4trdStr t Strddlm}|SdS)zBDetermine whether flash-attention uses top-left or down-right maskFr'is_npu_fa2_top_left_aligned_causal_mask)rrrr integrations.npu_flash_attentionrGrFrrr!flash_attn_supports_top_left_masks rIr&returncCsV|jdtjd}tj|dd}|}ttj |dtjdd}|||fS)aq Retrieves indexing data required to repad unpadded (ragged) tensors. Arguments: attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: indices (`torch.Tensor`): The indices of non-masked tokens from the flattened input sequence. cu_seqlens (`torch.Tensor`): The cumulative sequence lengths, used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). max_seqlen_in_batch (`int`): Maximum sequence length in batch. r rFrrr) rrrrrr r!r"r#r$)r&r)rr+r,rrr_get_unpad_datas rK query_layer key_layer value_layer query_lengthcCs(t|\}}}|jd|jd} kr4|ddd| ddddf|ddd| ddddf}}|j\} } } } t||}t||}|| krUt||}|}|}|}n3|dkrsd}tj| dtj|jd}|dd}|d}n|dd| df}|||^}}}}}||||||f||ffS)a Unpads query, key, and values tensors, using a single dimension for all tokens even though they belong to different batches. This function is used instead of `flash_attn.bert_padding.unpad_input` in order to avoid the recomputation of the same intermediary tensors for query, key, value tensors. Arguments: query_layer (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. query_length (`int`): Target length. unpad_input_func: The function to use for unpadding the input tensors. Return: query_layer (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rr N)rr/)rKrrrarangerr/squeeze)rLrMrNr&rOunpad_input_func indices_k cu_seqlens_kmax_seqlen_in_batch_kseq_len batch_size kv_seq_lennum_key_value_headshead_dim cu_seqlens_qmax_seqlen_in_batch_q indices_q_rrr _upad_inputs6*B     r_cCs|d|d|d}|d|d|d}|d|d|d}|}tj|d|jtjd}t||dktj ||jtjdf}| d}||||||f||ffS)aI This function returns necessary arguments to call `flash_attn_varlen_func`. All three query, key, value states will be flattened. Cumulative lengths of each examples in the batch will be extracted from position_ids. NOTE: ideally cumulative lengths should be prepared at the data collator stage Arguments: query (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). position_ids (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: query (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). r rr.r) r1size contiguousrrrPr/rcatrr )querykeyvalue position_idsr] cu_seq_lens max_lengthrrr*_prepare_flash_attention_from_position_ids,s   rjcOstdtt|i|S)NzThe function `prepare_fa2_from_position_ids` in `transformers.modeling_flash_attention_utils` is deprecated and will be removed in a future version. Please use `_prepare_flash_attention_from_position_ids` instead.)warningswarn FutureWarningrj)argskwargsrrrprepare_fa2_from_position_ids^s rprdrerf target_dtypecCs\|dur |||fS|j}|tjkr)td|d||}||}||}|||fS)aG PEFT usually casts the layer norms in float32 for training stability reasons therefore the input hidden states gets silently casted in float32. Hence, we need cast them back in float16 / bfloat16 just to be sure everything works as expected. This might slowdown training & inference so it is recommended to not cast the LayerNorms! Args: query (`torch.Tensor`): Input query states to be passed to Flash Attention API key (`torch.Tensor`): Input key states to be passed to Flash Attention API value (`torch.Tensor`): Input value states to be passed to Flash Attention API target_dtype (`torch.dtype`, *optional*): The dtype to convert the attention tensors to. Conversion can be ignored by not providing the target dtype. NzThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)rrfloat32logger warning_onceto)rdrerfrq input_dtyperrrfa_peft_integration_checkfs      rxz2.4.1 query_states key_states value_states is_causaldropoutrg softmax_scalesliding_windowuse_top_left_masksoftcap deterministic cu_seq_lens_q cu_seq_lens_k max_length_q max_length_kattn_implementationc( Ks|durt}t}t}t}t}n|dkrt}t}t}t}d}n|dkr,t }t }t }t }d}| s1|}n|o6|dk}t oC| duoC|jd| k}|rLd| | fini}|rZ|dkrYtd n||d <trw| durstdurqtjd d d kat} | |d<| dur| |d<t||||\}}}|duo|jddko|dup|dkotj|dddk }tdd| |||fD}|dur|jd}t||||||\}}}}} }!| \}"}#|!\}$}%||||f|"|#|$|%||d|}&||&|||}'n~|s|r_|d}| dus|durt||||\}}}}} }!| \} }|!\}}n'|d|d|d}|d|d|d}|d|d|d}||||f| |||||d|}'|'|d|'d|'d}'n ||||f||d|}'t|'t rv|'dS|'S)aq Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, then computes the attention scores and pad the final attention scores. Args: query_states (`torch.Tensor`): Input query states to be passed to Flash Attention API key_states (`torch.Tensor`): Input key states to be passed to Flash Attention API value_states (`torch.Tensor`): Input value states to be passed to Flash Attention API attention_mask (`torch.Tensor`, *optional*): The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the position of padding tokens and 1 for the position of non-padding tokens. dropout (`float`): Attention dropout softmax_scale (`float`, *optional*): The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim) use_top_left_mask (`bool`, defaults to `False`): flash_attn<2.1 generates top-left aligned causal mask, while what is needed here is bottom-right alignment, that was made default for flash_attn>=2.1. This attribute is used to handle this difference. softcap (`float`, *optional*): Softcap for the attention logits, used e.g. in gemma2. deterministic (`bool`, *optional*): Determines if the deterministic option introduced in flash_attn>=2.4.1 is enabled. attn_implementation (`str`, *optional*): The attention implementation to use. If None, will default to the one based on the environment. Nflash_attention_3Tflash_attention_2FrrDryzCFlash Attention 3 does not support dropout. Setting dropout to 0.0. dropout_pFLASH_ATTENTION_DETERMINISTIC01rrrr )rcss|]}|duVqdSNr).0kwargrrr s z+_flash_attention_forward..)r[rT max_seqlen_q max_seqlen_krcausalr`)rr)!r7r6r8r9HAS_FA3flash_attn_3_varlen_funcflash_attn_3_func pad_input_fa3unpad_input_fa3flash_attn_2_varlen_funcflash_attn_2_func pad_input_fa2unpad_input_fa2_flash_supports_window_sizerrtru flash_241deterministic_gosenvirongetrxrdiffallr_rarjr r1 isinstancetuple)(rzr{r|r&rOr}r~rgrrrrrrrrrrqrro_flash_attn_varlen_func_flash_attn_func _pad_input _unpad_input_is_fa3ruse_sliding_windows flash_kwargsis_fa2_with_position_idsis_fa2_with_varlen_kwargsrWr]rh max_seq_lensr[rTr\rUattn_output_unpad attn_outputrrr_flash_attention_forwards1    &           rc@sFeZdZUdZeejed<eejed<eeed<eeed<dS)FlashAttentionKwargsa Keyword arguments for Flash Attention with Compile. Attributes: cumulative_seqlens_q (`torch.LongTensor`, *optional*) Gets cumulative sequence length for query state. cumulative_seqlens_k (`torch.LongTensor`, *optional*) Gets cumulative sequence length for key state. max_length_q (`int`, *optional*): Maximum sequence length for query state. max_length_k (`int`, *optional*): Maximum sequence length for key state. cumulative_seqlens_qcumulative_seqlens_krrN) __name__ __module__ __qualname____doc__rr LongTensor__annotations__intrrrrrIs  r)totalr) ryNNNFNNNNNNNN)Hinspectrrktypingrrrtorch.nn.functionalnn functionalr"utilsrrrrr r get_loggerrrtr6rr-r5 FA_VERSION flash_attnrr7rflash_attn.bert_paddingr8rr9rflash_attn.layers.rotaryr:HAS_FA2rHr;r<r=flash_attn_interfacerrrrrglobalsrlist signature parametersrErITensorrrrKr_rjrprrxrrboolfloatrstrrrrrrrs               " P2  )       7