o ̳i38@srddlZddlmZddlZddlmmZddlmZddlm Z ddl m Z e e ZGdddejZdS)N)Optional)nn) _MaskType)KVCachec'seZdZdZddddddddddd ded ed ed ed ejd ejdejdejdeejdeejdeejdeedede de deedee deeddf&fddZ dede j deddfdd Zd!d"Z d*ddd#d$e jd%ee jd&eed'ee jde jf d(d)ZZS)+Gemma2Attentionaz Adapated from official Google Pytorch Implementation: https://github.com/google/gemma_pytorch/blob/80881c2e6e797ef1913a4a705d4b40394791cc58/gemma/model.py#L213 to match torchtune style. A new attention had to be added since nn.functional.scaled_dot_product_attention does allow soft capping Args: embed_dim (int): embedding dimension for the model num_heads (int): number of query heads. For MHA this is also the number of heads for key and value num_kv_heads (int): number of key and value heads. User should ensure ``num_heads % num_kv_heads == 0``. For standard MHA set ``num_kv_heads == num_heads``, for GQA ``num_kv_heads < num_heads``, and for MQA set ``num_kv_heads == 1``. head_dim (int): dimension of each head, calculated by ``embed_dim // num_heads``. q_proj (nn.Module): projection layer for query. k_proj (nn.Module): projection layer for key. v_proj (nn.Module): projection layer for value. output_proj (nn.Module): projection layer for output. pos_embeddings (Optional[nn.Module]): positional embeddings layer, e.g. RotaryPositionalEmbeddings. q_norm (Optional[nn.Module]): normalization layer for query, e.g. RMSNorm. For decoding, this is applied before updating from kv_cache. This means it will only support token wide normalization and not batch or sequence wide normalization. k_norm (Optional[nn.Module]): normalization layer for key, must be set if q_norm is. kv_cache (Optional[KVCache]): KVCache object used to cache key and value max_seq_len (int): maximum sequence length supported by the model. This is needed to compute the RoPE Cache. Default: 4096. is_causal (bool): sets the default mask to causal when no mask is provided attn_dropout (float): dropout value passed onto the scaled_dot_product_attention function. This argument is ignored if the self.training is False. Default value is 0.0. sliding_window_size (Optional[int]): size of the sliding window if None no sliding window is applied softcapping (Optional[float]): capping value used for soft caping, if None no capping is performed query_pre_attn_scalar (Optional[int]): value used for pre attention normalisation, if None head_dim is used instead Raises: ValueError: If ``num_heads % num_kv_heads != 0``, **or** if ``embed_dim % num_heads != 0``, **or** if ``attn_dropout < 0`` or ``attn_dropout > 1``, **or** if ``q_norm`` is defined without k_norm or vice versa NiTggI@) pos_embeddingsq_normk_normkv_cache max_seq_len is_causal attn_dropoutsliding_window_size softcappingquery_pre_attn_scalar embed_dim num_heads num_kv_headshead_dimq_projk_projv_proj output_projrrr r r r r rrrreturncst||dkrtd|d|d||dkr'td|d|d|dks/|dkr7td|d t| t| ArCtd ||_||_||_||_||_| |_ ||_ | |_ ||_ ||_ ||_||_| |_| |_| |_||_||_|dur|d |_n|jd |_d |_dS) Nrz num_heads (z%) must be divisible by num_kv_heads ()z embed_dim (z") must be divisible by num_heads (zattn_dropout (z) must be between 0.0 and 1.0z!q and k norm must be set togethergF)super__init__ ValueErrorboolrrrr rr r r rrrrrr rrrscaling cache_enabled)selfrrrrrrrrrrr r r r r rrr __class__V/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/gemma2/_attention.pyr<sL      zGemma2Attention.__init__ batch_sizedtypecCs:|jdur tddSt|||j|j|d|_d|_dS)aQSetup key value caches for attention calculation. If called after kv_cache is already setup, this will be skipped. Args: batch_size (int): batch size for the caches. dtype (torch.dtype): dtype for the caches. max_seq_len (int): maximum sequence length model will be run with. NzWKey value caches are already setup. You cannot call ``setup_caches()`` twice. Skipping.)r'r rrr(T)r loggerwarningrrrr!)r"r'r(r r%r%r& setup_caches  zGemma2Attention.setup_cachecCs |jdur td|jdS)zzReset the key value caches. Raises: RuntimeError: if key value caches are not already setup. Nz>Key value caches are not setup. Call ``setup_caches()`` first.)r RuntimeErrorreset)r"r%r%r& reset_caches zGemma2Attention.reset_cache)mask input_posxyr/r0cCs8|durt|tjstd|j\}}}|dur|jdnd}||} |j|j} | |||j| |j } |j durB|j | |d} | dd} |j durR| | } |durk|j dus^|jsbtd|j j} |j j} n||} ||} | ||d|j } |j dur|j | |d} | |||jd|j } | |||jd|j } |j|jkr| |||j| |j } | |||j| |j } | ||d|j } | ||d|j } | dd} | dd} |jdur|| } |j dur|jr|j | | \} } | |jt| | dd } |durttj||ftjd |j}|j tjkr+t!|"d d}|j#durSt$|}t%|d|j#dt||j#d}t!|dk|d }|&d kr_|'d}|j(durt| |j(} t)| } | |j(} | |} t*j+| ,dd -| } t| | } | dd.||d} |/| S) a Args: x (torch.Tensor): input tensor with shape [b x s_x x d] for the query y (Optional[torch.Tensor]): second input tensor with shape [b x s_y x d], is the input for k and v. For self attention, x=y. Optional only with kv_cache enabled. mask (Optional[_MaskType]): Used to mask the scores after the query-key multiplication and before the softmax. Either: A boolean tensor with shape ``[b x s x s]``, ``[b x s x self.encoder_max_cache_seq_len]``, or ``[b x s x self.encoder_max_cache_seq_len]`` if using KV-cacheing with encoder/decoder layers. A value of True in row ``i`` and column ``j`` means token ``i`` attends to token ``j``. A value of False means token ``i`` does not attend to token ``j``. If no mask is specified, a causal mask is used by default. A :class:`~torch.nn.attention.flex_attention.BlockMask` for document masking in a packed sequence created via `create_block_mask `_. We use :func:`~torch.nn.attention.flex_attention.flex_attention` when computing attention with block masks. Default is None. input_pos (Optional[torch.Tensor]): Optional tensor which contains the position ids of each token. During training, this is used to indicate the positions of each token relative to its sample when packed, shape [b x s]. During inference, this indicates the position of the current token. If none, assume the index of the token is its position id. Default is None. Raises: NotImplementedError: If ``mask`` is provided, but mask is not an instance of ``torch.Tensor``. ValueError: If no ``y`` input and ``kv_cache`` is not enabled. Returns: torch.Tensor: output tensor with attention applied Notation used for tensor shapes: - b: batch size - s_x: sequence length for x - s_y: sequence length for y - n_h: num heads - n_kv: num kv heads - d: embed dim - h_d: head dim Nz5Block masks are not implemeted yet, use packed=False.rr)r0zAMust provide y input or use kv_cache to enable streaming decoding)sizer(g s