o ̳i@s6ddlmZddlZddlmZGdddejZdS))OptionalN)nnc sxeZdZdZ  ddedededdffd d Zd d Zddeddfd dZ dde j de e j de j fddZ Z S)Qwen2RotaryPositionalEmbeddingsau RoPE Embeddings used in the Qwen2 model. Ref: https://huggingface.co/Qwen/Qwen2-7B-Instruct This class is not numerically equivalent to the RoPE Embedding module used by Llama2 and Llama3. Args: dim (int): Embedding dimension. This is usually set to the dim of each head in the attention module computed as ``embed_dim`` // ``num_heads`` max_seq_len (int): Maximum expected sequence length for the model, if exceeded the cached freqs will be recomputed base (float): The base for the geometric progression used to compute the rotation angles .Adim max_seq_lenbasereturnNcs(t||_||_||_|dSN)super__init__rr r rope_init)selfrrr  __class__a/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/qwen2/_positional_embeddings.pyr s  z(Qwen2RotaryPositionalEmbeddings.__init__cCsRd|jtd|jdd|jd|j}|jd|dd||jdS)Ng?rthetaF persistent)r torcharangerfloatregister_bufferbuild_rope_cacher)rrrrrr+s&z)Qwen2RotaryPositionalEmbeddings.rope_initcCsltj||jj|jjd}td||j}tj||gdd}tj|| gdd}|j d|dddS)N)dtypedevicez i, j -> ijrcacheFr) rrrrreinsumrcatcossinr)rrseq_idx idx_thetafreqsr!rrrr3sz0Qwen2RotaryPositionalEmbeddings.build_rope_cachex input_posc Cs|d}|d}|dur|jd|n|j|}|d|d|d}|dd|f|j}|d|df|j}|dd|jddf}|d|jdddf} tj| |fdd} ||| |} | |S)ae Args: x (torch.Tensor): input tensor with shape [b, s, n_h, h_d] input_pos (Optional[torch.Tensor]): Optional tensor which contains the position ids of each token. During training, this is used to indicate the positions of each token relative to its sample when packed, shape [b, s]. During inference, this indicates the position of the current token. If none, assume the index of the token is its position id. Default is None. Returns: Tensor: output tensor with RoPE applied Notation used for tensor shapes: - b: batch size - s: sequence length - n_h: num heads - h_d: head dim TODO: The implementation below can be made more efficient for inference. rNr.r ) sizer!viewtorshaperr#type_as) rr)r*seq_lenhead_dim rope_cacher$r%x1x2rotatedx_outrrrforwardDs   z'Qwen2RotaryPositionalEmbeddings.forward)rr)rr )__name__ __module__ __qualname____doc__intrr rrrTensorrr8 __classcell__rrrrrs. r)typingrrrModulerrrrrs