o ©Ì³iþEã@sÄddlZddlmZddlZddlmmZddlmZddl m Z ddl m Z mZddlm Z e e¡ZGdd„dejƒZGdd „d ejƒZd ejjd dfd d „Zd ejjd ejjfdd„ZdS)éN)ÚOptional)Únn)ÚKVCache)Ú _MaskTypeÚ_sdpa_or_flex_attentionc!sìeZdZdZddddddddœdeded ed ed ejd ejd ejdejdeejdeejdeejdeedede de ddf ‡fdd„Z dede j deddfdd„Zdd„Zdddœd e jd!e jd"eed#ee jde jf d$d%„Z‡ZS)&ÚMultiHeadAttentionuN NOTE: torch.export.export() friendly MultiHeadAttention, modified from torchtune.modules.attention.MultiHeadAttention Major differences: - Rewrite `if y is None` to torch.cond(). - Logic becomes `if all values of y are NaN`, to make torch.compile() happy. - No input mutations in both false and true branches, so we need to copy kv values back into kv cache after torch.cond(). - Added a SDPA module - SDPA module includes transpose and expanding kv dimensions. - Makes it easy to swap with custom SDPAs that are needed by the users of exported program. - Uses new kv cache - This potentially can be merged with torchtune.modules.kv_cache. - Changed += to .add_ to avoid mutating module attributes. - Added clone() method. Multi-headed attention layer with support for grouped query attention (GQA) introduced in https://arxiv.org/abs/2305.13245v1. GQA is a version of multiheaded attention (MHA) which uses fewer key/value heads than query heads by grouping n query heads for each key and value head. Multi-Query Attention is an extreme version where we have a single key and value head shared by all query heads. Following is an example of MHA, GQA and MQA with num_heads = 4 (credit for the documentation: `litgpt.Config `_). :: ┌───â”┌───â”┌───â”┌───┠┌───┠┌───┠┌───┠│ v ││ v ││ v ││ v │ │ v │ │ v │ │ v │ └───┘└───┘└───┘└───┘ └───┘ └───┘ └───┘ │ │ │ │ │ │ │ ┌───â”┌───â”┌───â”┌───┠┌───┠┌───┠┌───┠│ k ││ k ││ k ││ k │ │ k │ │ k │ │ k │ └───┘└───┘└───┘└───┘ └───┘ └───┘ └───┘ │ │ │ │ ┌──┴──┠┌──┴──┠┌────┬──┴─┬────┠┌───â”┌───â”┌───â”┌───┠┌───â”┌───â”┌───â”┌───┠┌───â”┌───â”┌───â”┌───┠│ q ││ q ││ q ││ q │ │ q ││ q ││ q ││ q │ │ q ││ q ││ q ││ q │ └───┘└───┘└───┘└───┘ └───┘└───┘└───┘└───┘ └───┘└───┘└───┘└───┘ ◀──────────────────▶ ◀──────────────────▶ ◀──────────────────▶ MHA GQA MQA n_kv_heads =4 n_kv_heads=2 n_kv_heads=1 Args: embed_dim (int): embedding dimension for the model num_heads (int): number of query heads. For MHA this is also the number of heads for key and value num_kv_heads (int): number of key and value heads. User should ensure ``num_heads % num_kv_heads == 0``. For standard MHA set ``num_kv_heads == num_heads``, for GQA ``num_kv_heads < num_heads``, and for MQA set ``num_kv_heads == 1``. head_dim (int): dimension of each head, calculated by ``embed_dim // num_heads``. q_proj (nn.Module): projection layer for query. k_proj (nn.Module): projection layer for key. v_proj (nn.Module): projection layer for value. output_proj (nn.Module): projection layer for output. pos_embeddings (Optional[nn.Module]): positional embeddings layer, e.g. RotaryPositionalEmbeddings. q_norm (Optional[nn.Module]): normalization layer for query, e.g. RMSNorm. For decoding, this is applied before updating from kv_cache. This means it will only support token wide normalization and not batch or sequence wide normalization. k_norm (Optional[nn.Module]): normalization layer for key, must be set if q_norm is. kv_cache (Optional[KVCache]): KVCache object used to cache key and value max_seq_len (int): maximum sequence length supported by the model. This is needed to compute the RoPE Cache. Default: 4096. is_causal (bool): sets the default mask to causal when no mask is provided attn_dropout (float): dropout value passed onto the scaled_dot_product_attention function. Default value is 0.0. Raises: ValueError: If ``num_heads % num_kv_heads != 0`` ValueError: If ``embed_dim % num_heads != 0`` ValueError: If ``attn_dropout < 0`` or ``attn_dropout > 1`` ValueError: if q_norm is defined without k_norm or vice versa NiTç)Úpos_embeddingsÚq_normÚk_normÚkv_cacheÚ max_seq_lenÚ is_causalÚ attn_dropoutÚ embed_dimÚ num_headsÚ num_kv_headsÚhead_dimÚq_projÚk_projÚv_projÚ output_projr r r r r rrÚreturnc s"tƒ ¡||dkrtd|›d|›dƒ‚||dkr'td|›d|›dƒ‚|dks/|dkr7td|›d ƒ‚t| ƒt| ƒArCtd ƒ‚||_||_||_||_||_| |_ ||_ | |_ ||_ ||_ ||_||_| |_| |_| |_tƒ|_t|j|j|j|jr|jnd |j |j|j d |_d |_dS)Nrz num_heads (z%) must be divisible by num_kv_heads (ú)z embed_dim (z") must be divisible by num_heads (ézattn_dropout (z) must be between 0.0 and 1.0z!q and k norm must be set togetherr)rrrrrÚ attention_fnr F)ÚsuperÚ__init__Ú ValueErrorÚboolrrrrrr rr rrrrr r r rÚ_attention_callÚSDPAÚtrainingÚ_sdpaÚ cache_enabled)Úselfrrrrrrrrr r r r r rr©Ú __class__©úW/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/modules/_export/attention.pyrfsV  ÿÿ ÿÿù zMultiHeadAttention.__init__Ú batch_sizeÚdtypecCsF|jdur t d¡dSt|||j|j|dd|_|j|j_d|_dS)aQSetup key value caches for attention calculation. If called after kv_cache is already setup, this will be skipped. Args: batch_size (int): batch size for the caches. dtype (torch.dtype): dtype for the caches. max_seq_len (int): maximum sequence length model will be run with. NzWKey value caches are already setup. You cannot call ``setup_caches()`` twice. Skipping.F)r*r rrr+Útranspose_cacheT)r ÚloggerÚwarningÚInferenceKVCacherrr#r$)r%r*r+r r(r(r)Ú setup_cache°s ÿú  zMultiHeadAttention.setup_cachecCs |jdur tdƒ‚|j ¡dS)zReset the key value caches.Nz>Key value caches are not setup. Call ``setup_caches()`` first.)r Ú RuntimeErrorÚreset©r%r(r(r)Ú reset_cacheÌs ÿzMultiHeadAttention.reset_cache)ÚmaskÚ input_posÚxÚyr5r6cs(|j\‰}}ˆ |¡}ˆjˆj}| ˆ|ˆj|ˆj¡}ˆjdur)ˆj|ˆd}ˆjdur3ˆ |¡}‡‡‡fdd„‰‡fdd„} ‡‡fdd„} ˆjdur\|dusUJd ƒ‚ˆ|ƒ\} } n(t   t   |¡  ¡  ¡| | |f¡\} } } ˆjj | ¡ˆjj | ¡ˆjj | ¡ˆj|| | ˆ||d }ˆ |¡S) a* Args: x (torch.Tensor): input tensor with shape [b x s_x x d] for the query y (torch.Tensor): second input tensor with shape [b x s_y x d], is the input for k and v. For self attention, x=y. If all values are NaN, we read from kv cache. mask (Optional[_MaskType]): Used to mask the scores after the query-key multiplication and before the softmax. Either: A boolean tensor with shape ``[b x s x s]``, ``[b x s x self.encoder_max_cache_seq_len]``, or ``[b x s x self.encoder_max_cache_seq_len]`` if using KV-cacheing with encoder/decoder layers. A value of True in row ``i`` and column ``j`` means token ``i`` attends to token ``j``. A value of False means token ``i`` does not attend to token ``j``. If no mask is specified, a causal mask is used by default. A :class:`~torch.nn.attention.flex_attention.BlockMask` for document masking in a packed sequence created via `create_block_mask `_. We use :func:`~torch.nn.attention.flex_attention.flex_attention` when computing attention with block masks. Default is None. input_pos (Optional[torch.Tensor]): Optional tensor which contains the position ids of each token. During training, this is used to indicate the positions of each token relative to its sample when packed, shape [b x s]. During inference, this indicates the position of the current token. If none, assume the index of the token is its position id. Default is None. Returns: torch.Tensor: output tensor with attention applied Notation used for tensor shapes: - b: batch size - s_x: sequence length for x - s_y: sequence length for y - n_h: num heads - n_kv: num kv heads - d: embed dim - h_d: head dim N©r6csv|jd}ˆ |¡}ˆ |¡}| ˆ|dˆj¡}| ˆ|dˆj¡}ˆjdur-ˆj|ˆd}ˆjdur7ˆ |¡}||fS)Nréÿÿÿÿr9)ÚshaperrÚviewrr r )r8Ús_yÚkÚv)Úbr6r%r(r)Ú calculate_kvs      z0MultiHeadAttention.forward..calculate_kvcsˆj ¡}|j|j|jfS©N)r ÚcloneÚk_cacheÚv_cacheÚ cache_pos)r8r r3r(r)Útrue_fn's z+MultiHeadAttention.forward..true_fncs2ˆ|ƒ\}}ˆj ¡}| ||¡|j|j|jfSrB)r rCÚupdaterDrErF)r8r>r?r )rAr%r(r)Úfalse_fn+s   z,MultiHeadAttention.forward..false_fnzAMust provide y input or use kv_cache to enable streaming decoding)r5)r;rrrr<rr r r ÚtorchÚcondÚisnanÚallÚitemrDÚcopy_rErFr#r)r%r7r8r5r6Ús_xÚ_ÚqÚq_per_kvrGrIr>r?rFÚoutputr()r@rAr6r%r)ÚforwardÔs0 .        ÿ ÿ zMultiHeadAttention.forward)Ú__name__Ú __module__Ú __qualname__Ú__doc__ÚintrÚModulerrrÚfloatrrJr+r0r4ÚTensorrrUÚ __classcell__r(r(r&r)rs‚[ïýüûúùø ÷ ö õ ô óòñðïîJÿÿÿ þ úþýûúùrcspeZdZdZdedededededdf ‡fd d „ Z dd ej d ej d ej dedede e dej fdd„Z ‡Z S)r!zr TorchTune's SDPA which can be optimized and can be swapped out for a more efficient implementations. rrrrrrNcsFtƒ ¡||_||_||_|j|j|_||_||_||_||_ dSrB) rrrrrrSrrÚ _attention_fnr )r%rrrrrrr r&r(r)rMs  z SDPA.__init__rRr>r?ÚbszÚseq_lenr5c Cs¶| dd¡}| dd¡}| dd¡}|j|jkr8dd|jddf}| d¡ |¡ dd¡}| d¡ |¡ dd¡}|j|||||j|j duoK|duoK|j d}| dd¡  ¡  ||d¡S)Nrér:)r5Ú dropout_pr) Ú transposerrrSÚ unsqueezeÚexpandÚflattenr_rr rÚ contiguousr<) r%rRr>r?r`rar5Ú expand_shaperTr(r(r)rUas   ú z SDPA.forwardrB)rVrWrXrYrZr\rrrJr]rrrUr^r(r(r&r)r!Gs>þýüûú ÷ùþýüûúùør!ÚmodulercCsx| ¡D]5\}}t|tjƒr5t||t|j|j|j|j|j |j |j |j |j |j|j|j|j|j|jdƒqt|ƒqdS)N)rrrrrrrrr r r r r rr)Únamed_childrenÚ isinstanceÚTorchTuneAttentionrÚsetattrrrrrrrrrr r r r r rrÚreplace_mha_with_inference_mha)rjÚnameÚchildr(r(r)Ú_replace_mha_with_inference_mha…s2 ñý èrrcCs t|ƒ|S)z˜ Replace TorchTune's MHA with an inference friendly version of MHA that separates out the inference-related parts for further optimization. )rr)rjr(r(r)ro¡sro)ÚloggingÚtypingrrJÚtorchtune.modules.attentionÚmodulesÚ attentionrmrÚ"torchtune.modules._export.kv_cacherr/Ú!torchtune.modules.attention_utilsrrÚtorchtune.modules.kv_cacheÚ getLoggerrVr-r[rr!rrror(r(r(r)Ús     4>