o ̳is' @s0UddlZddlmZmZmZmZddlZddlmZddlm Z ddl m Z m Z e Z ejed<e rfddlmZmZmZdd ZeZejjd d d ejd ejdejdedejf ddZeejefZnejZdeejdejfddZdeejdejfddZdeejdefddZdefddZdS)N)CallableListOptionalUnion)nn)_SUPPORTS_FLEX_ATTENTION) get_loggerlog_once_log) BlockMaskcreate_block_maskflex_attentionc CsztjtddWStyC}z/td|dztjtdddWWYd}~Sty>}z td|dd}~wwd}~ww) NF)dynamicz,Compiling flex_attention failed with error 'z%'. Retrying with mode='max-autotune'.z max-autotune)rmodez-Compiling flex_attention failed with error: 'z', Updating your pytorch version to nightlies may solve it, or you can setin your config dataset.packed=False to avoid using flex attention.)torchcompiler Exceptionr info)erU/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/modules/attention_utils.pycompile_flex_attentions"  rF) recursiveqkv block_maskreturncCst||||dS)Nr)flex_attention_compiled)rrrrrrrcompile_friendly_flex_attention2sr seq_lenscCsNt|}g}t|D]}tddt||D}||q t|}|S)a Convert a batch tensor of seq lens into integer IDs denoting sample ownership. For example, seq_lens = [2, 3, 1] would return [0, 0, 1, 1, 1, 2]. Args: seq_lens (List[torch.Tensor]): Sequence lengths of samples in each pack in the batch, shape (batch_size, n), where n is the max number of sequences in a pack and can vary across packs. Returns: Tensor: Document IDs of shape (batch_size, max_seq_len). cSs(g|]\}}tj|f|tj|jdqS)dtypedevice)rfulllongr$.0iseq_lenrrr Usz3_get_document_ids_from_seq_lens..)lenrangercat enumerateappendstack)r! batch_sizebatch_document_ids sample_idx document_idsrrr_get_document_ids_from_seq_lens@s    r6cCsJg}t|}t|D]}ddt||D}|tj|q t|S)a  Given a batch tensor of seq lens defining the lengths of samples in each pack, Construct a 2D block causal mask for each pack in the batch. For example, if a single sample's seq_lens is [3, 2, 1], the mask would be:: mask = [ [1, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0], [0, 0, 0, 1, 0, 0], [0, 0, 0, 1, 1, 0], [0, 0, 0, 0, 0, 1], ] Args: seq_lens (List[torch.Tensor]): Sequence lengths of samples in each pack in the batch, shape (batch_size, n), where n is the max number of sequences in a pack and can vary across packs. Returns: Tensor: Block causal mask of shape (batch_size, max_seq_len, max_seq_len). c Ss,g|]\}}ttj||tj|jdqSr")rtrilonesboolr$r'rrrr+zs z,create_block_causal_mask..)r,r-r/r0r block_diagr1)r!batch_block_attn_masksr2r4block_attn_masksrrrcreate_block_causal_mask_s   r=csJtr t|j\}}dfdd}t||d||ddSt|dS)a Create a block causal document mask for a batch of packed sequences. If flex attention is supported by the current hardware, block causal logic and passing this into :func:`torch.nn.attention.flex_attention.create_block_mask`. The resultant BlockMask is a compressed representation of the full block causal mask. If on an older version, a standard 2D block causal mask is created and returned. Args: seq_lens (List[torch.Tensor]): Sequence lengths of samples in each pack in the batch, shape (batch_size, n), where n is the max number of sequences in a pack and can vary across packs. Returns: _MaskType: BlockMask or Tensor if torch version < 2.5.0. cudacs(||k}||f||fk}||@S)a Defines the logic of a block causal mask by combining both a standard causal mask and a block diagonal document mask. See :func:`~torchtune.modules.attention_utils.create_block_causal_mask` for an illustration. r)bhq_idxkv_idx causal_mask document_maskr5rrmask_modsz*packed_block_causal_mask..mask_modN)r$)r!)rr6shapetocreate_block_causal_mask_flexr=)r!r2 max_seq_lenrFrrErpacked_block_causal_masks     rKcCsptrdtjdtjdtjdttdtdtdtjfdd }|Sdtjdtjdtjdttdtdtdtjfd d }|S) aE Helper function to decide when to call flex attention or SDPA. It will use flex attention if ALL of the following conditions are met, otherwise it will default to SDPA: - torch version >= 2.5.0 - we are sample packing, therefore mask is a BlockMask - torch.cuda.get_device_capability() >= (7, 5) rrrmask dropout_p is_causalrcSsvt|trttdtjd|dkrtdt||||dS|dur/|dddddddf}tj j ||||||dS)NzOUsing flex attention for attention computation since a BlockMask was passed in.)levelgzCFlex attention does not support dropout. Please set dropout to 0.0.r attn_maskrMrN) isinstancer r r loggingDEBUG ValueErrorr r functionalscaled_dot_product_attentionrrrrLrMrNrrr_attention_calls4  z0_sdpa_or_flex_attention.._attention_callcSs<|dur|dddddddf}tjj||||||dS)NrP)rrVrWrXrrrrYs )rrTensorr _MaskTypefloatr9)rYrrr_sdpa_or_flex_attentionsB  G r]) rStypingrrrrrrtorchtune.utils._import_guardrtorchtune.utils._loggingrr r Logger__annotations__!torch.nn.attention.flex_attentionr r rIr rrcompilerdisablerZr r[r6r=rKr]rrrrsJ      & 4