o wi,@spdZddlmZmZddlZddlmZddlmZm Z ddl m Z m Z er1ddl mZmZmZe eZGd d d Z d%d ejd ejdejdejfddZeejefZ     d&dejdeedeeeefdeeddf ddZdejdedejfddZ   d'dejjd ejd ejdejdeejdfd eed!eed"eejdeejejffd#d$Z dS)(a7 Partially inspired by torchtune's flex attention implementation Citation: @software{torchtune, title = {torchtune: PyTorch's finetuning library}, author = {torchtune maintainers and contributors}, url = {https//github.com/pytorch/torchtune}, license = {BSD-3-Clause}, month = apr, year = {2024} } )OptionalUnionN)version)is_torch_flex_attn_availablelogging)_torch_versionis_torchdynamo_compiling) BlockMaskcreate_block_maskflex_attentioncsJeZdZdZdZdZdZfddZej j ddddZ d d Z Z S) WrappedFlexAttentionzh We are doing a singleton class so that flex attention is compiled once when it's first called. NFcs|jdur t||_|jSN) _instancesuper__new__)clsargskwargs __class__e/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/integrations/flex_attention.pyr6s zWrappedFlexAttention.__new__) recursivecCsX|jr||jkr*||_ttjdkr|rtjtddd|_ ntt|_ d|_dSdS)z> Initialize or update the singleton instance. z2.6.0Fzmax-autotune-no-cudagraphs)dynamicmodeTN) _is_flex_compiledtrainingrparser base_versiontorchcompiler _compiled_flex_attention)selfrrrr__init__<s  zWrappedFlexAttention.__init__cCs|jSr)r")r#rrr__call__NszWrappedFlexAttention.__call__)__name__ __module__ __qualname____doc__rrr"rr compilerdisabler$r% __classcell__rrrrr -s   r FquerykeyvaluereturncKs(tst|nt}||||fi|Sr)r r r )r-r.r/rrflex_attention_compiledrrrcompile_friendly_flex_attentionRs r2Tattention_mask_2dattention_chunk_sizeoffsets is_causalr c sj\}}|s |}|s |}tjjjdd|fdj}|dur2ddd|fddfdd } fd d } |sL| n|durRn| |duri|d|dfd d } n} t | |d|||ddS)aG IMPORTANT NOTICE: This function is deprecated in favor of using the mask primitives in `masking_utils.py`, and will be removed in a future version without warnings. New code should not use it. It is only kept here for BC for now, while models using it are being patched accordingly. Create a block (causal) document mask for a batch of sequences, both packed and unpacked. Create Block (causal) logic and passing it into :func:`torch.nn.attention.flex_attention.create_block_mask`. The resultant BlockMask is a compressed representation of the full (causal) block mask. BlockMask is essential for performant computation of flex attention. See: https://pytorch.org/blog/flexattention/ Args: attention_mask_2d (torch.Tensor): Attention mask for packed and padded sequences of shape (batch_size, total_seq_len). e.g. For unpacked sequence: [[1, 1, 1, 1, 0, 0, 0], [1, 1, 1, 1, 1, 0, 0]] For packed sequence: [[1, 1, 1, 2, 2, 2, 0], [1, 1, 2, 2, 2, 3, 3]] Returns: BlockMask r)r/padNcs@||k}||f||fk}||fdk}||@|@}|S)z Defines the logic of a block causal mask by combining both a standard causal mask and a block diagonal document mask. See :func:`~torchtune.modules.attention_utils.create_block_causal_mask` for an illustration. rr) batch_idxhead_idxq_idxkv_idx causal_mask document_mask padding_mask final_maskr3 document_idsrrcausal_mask_mods  z4make_flex_block_causal_mask..causal_mask_modcs.||f||fk}||||}||@S)zU Combines the chunk mask with the causal mask for chunked attention. r)r:r;r<r= chunk_maskcausal_doc_mask)rD chunk_idxsrrchunk_causal_mask_modsz:make_flex_block_causal_mask..chunk_causal_mask_modcs4||f||fk}||fdk}||@}|S)zp Utilizes default attention mask to enable encoder and encoder-decoder attention masks. rr)r:r;r<r=r?r@rArBrrdefault_mask_modsz5make_flex_block_causal_mask..default_mask_modcs|}|}||||Srr)r:r;r<r=offset_q offset_kv) kv_offsetmask_mod_maybe_combinedq_offsetrrmask_modsz-make_flex_block_causal_mask..mask_modT)rOBHQ_LENKV_LENdevice_compile) shaper nn functionalr7rTclonefill_cumsumr ) r3r4 query_length key_lengthr5r6 batch_size total_seq_lenrTrHrIrOr)r3rDrGrCrLrMrNrmake_flex_block_causal_maskhs< "  r` hidden_statesn_repcCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r8N)rVexpandreshape)rarbbatchnum_key_value_headsslenhead_dimrrr repeat_kvs 0rimoduleattention_maskscalingsoftcap head_maskc s,dur td|dddkrtdd} dt|tr!|} n|dur:ddddddd|jdffdd} d } |jd } | | d @dksmt||jd |jd }t||jd |jd }d } |d d} t|||| | | || d |j d \}}| |j }| d d }||fS)Nzm`flex_attention` does not support `head_mask`. Please set your attention to `eager` if you want this feature.dropoutgrz`flex_attention` does not support `dropout`. Please use it with inference only (`model.eval()`) or turn off the attention dropout in the respective config.cs^dur t|}dur||d||}dur-|||dd}|S)Nr)r tanh)scorer:r;r<r=rn score_maskrmrr score_modsz)flex_attention_forward..score_modTr8Fkernel_options)ru block_mask enable_gqascalerv return_lserr)logger warning_onceget ValueError isinstancer rVrir2rtodtype transpose contiguous)rjr-r.r/rkrlrmrnrrwrurxnum_local_query_headsrv attn_outputattention_weightsrrsrflex_attention_forwardsL  &   r)F)NNNNT)NNN)!r)typingrrr packagingrutilsrrutils.import_utilsrr !torch.nn.attention.flex_attentionr r r get_loggerr&r{r Tensorr2intOffsettupleboolr`rirWModulefloatrrrrrsx  )  o