o @TitT@s ddlZddlmmZddlmZddlmZddlmZeZ z ddl m Z m Z dZ Wn ey9d Z Ynwdd ed ed efddZdejdejfddZdejdejdejfddZGdddejZGdddejZGdddejZGdddejZdS) N)nn) get_logger) AdaLNZero)flash_attn_funcflash_attn_with_kvcacheTF@dimendthetacCs`d|td|dd|d|}tj||jtjd}t||}tt||}|S)Ng?rr)devicedtype)torcharangefloatr float32outerpolar ones_like)r r r freqst freqs_cisrP/home/ubuntu/.local/lib/python3.10/site-packages/linacodec/module/transformer.pyprecompute_freqs_ciss * rrxcsb|jddkrksJJ|j|jd|jdfks Jfddt|jD}|j|S)Nrrcs,g|]\}}|dks|dkr|ndqS)rr).0idndimrr %s,z)reshape_for_broadcast..)r"shape enumerateview)rrr$rr!rreshape_for_broadcast!s  r'returncCsTt|jg|jddddR}t||}t||d}||S)Nrr) rview_as_complexrreshaper$r' view_as_realflattentype_as)rrx_x_outrrrapply_rotary_emb)s,  r1cseZdZ    ddededededBdeded ed effd d Zd ededejdBdej dejdBf ddZ ddejdejdBdejdBdedeje eje ejejffBf ddZ dejde ejejfdejdede eje ejejfff ddZ ZS) AttentionFr n_headsdropout window_sizeNqkv_bias proj_biasuse_flash_attentioncausalc st||_|||_tj|||j|d|_tj|||j|d|_tj|||j|d|_tj||j||d|_ |jd|_ ||_ |du|_ |j r]|ddksXJd|d|_ ||_||_dS)Nbiasgrrz,Window size must be odd for local attention.)super__init__r3head_dimrLinearwqwkwvwoscaler4use_local_attentionwindow_per_sider8r9) selfr r3r4r5r6r7r8r9 __class__rrr=1s      zAttention.__init__bszseqlenmaskr r(cCs|js |dur dStj||ftj|d}|jrt|}|jr0tj||j d}tj||jd}|d |dd}|durd|j d|krL|j d|ksPJd| dkr`|d |dd}||@}|d  d|j dd}|S) zMCreate attention mask combining provided mask and local attention constraintsN)rr )diagonalrrz.Mask must be square and match sequence length.rr) rEronesboolr9triltriurF unsqueezeexpandr$r r3)rGrJrKrLr attn_maskrrr create_maskRs$  zAttention.create_maskrr return_kvc Csn|j\}}}||||||}} } ||||j|j}| |||j|j} | |||j|j} |durMt||d|d}t| |d|d} |jryt ry|dusZJd|j rc|j |j fnd} t || | |j ro|jnd|j| |jd} n&|||||j} tj|dd| dd| dd| |j|jd dd} | ||d } || } |r| | | ffS| S) aForward pass for multi-head attention. Args: x (torch.Tensor): Input tensor of shape (bsz, seqlen, dim). freqs_cis (torch.Tensor, optional): Precomputed rotary frequencies. mask (torch.Tensor, optional): Attention mask. return_kv (bool): Whether to return KV pairs for caching. Returns: output (torch.Tensor): Output tensor of shape (bsz, seqlen, dim). new_kv (tuple, optional): KV pairs if return_kv is True. Nrz3Flash attention does not support arbitrary masking.)rrg) dropout_p softmax_scaler5r9rr)rUrYrDr)r$r@rArBr&r3r>r1r8FLASH_ATTN_AVAILABLErErFrtrainingr4rDr9rVr Fscaled_dot_product_attention transpose contiguousrC)rGrrrLrWrJrK_xqxkxvr5outputrUrrrforwardvsH "       zAttention.forwardkv_cache start_poscCsP|j\}}}|dksJd||||||}} } ||||j|j}| |||j|j} | |||j|j} t|||||d}t| ||||d} |\} } | | f} tj | | gdd} tj | | gdd} |j r~t r~t || | |j d}ntj|dd| dd| dd|j ddd}|||d}||| fS) ad Forward pass with KV cache for efficient inference. Only used for inference. Args: x (torch.Tensor): Input tensor for the current step. Shape: (bsz, 1, dim) kv_cache: A tuple of (key_cache, value_cache) from previous steps. start_pos (int): The starting position of the new token in the sequence. freqs_cis (torch.Tensor): Precomputed rotary frequencies. Returns: output (torch.Tensor): Output tensor after attention. Shape: (bsz, 1, dim) new_kv (tuple): Updated KV cache including the new key and value. rz8KV cache method is designed for single-token generation.rX)r )rZr)rDr)r$r@rArBr&r3r>r1rcatr8r[rrDr]r^r_r`rC)rGrrgrrhrJrKrarbrcrdk_cachev_cachenew_kvrerrrforward_with_caches: "    zAttention.forward_with_cache)FFFF)F)__name__ __module__ __qualname__intrrPr=rTensorr rVtuplerfrm __classcell__rrrHrr20sp ! ) @r2c s:eZdZdededededBffdd Zdd ZZS) FeedForwardr hidden_dim multiple_offfn_dim_multiplierNcs|ttd|d}|durt||}|||d|}tj||dd|_tj||dd|_tj||dd|_dS)Nrr)rFr:)r<r=rqrr?w1w2w3)rGr rvrwrxrHrrr=s  zFeedForward.__init__cCs |t||||SN)rzr]siluryr{)rGrrrrrfs zFeedForward.forward)rnrorprqrr=rfrtrrrHrrusrucseZdZ    ddedededededBded edBd ed ed edBd ededeffdd Z    ddejdejdBdejdBdejdBdede ejejfdBdedBdeje eje ejejffBfddZ Z S)TransformerBlockNFr r3r6r7r5rwrxr4norm_epsadanorm_condition_dimr8use_adaln_zeror9c stt||||| ||| d|_t|d|||d|_| |_|jr?| dus+Jdt|| | dd|_t|| | dd|_ dSt j || d|_t j || d|_ dS)N)r r3r4r5r8r6r7r9)r rvrwrx3condition_dim must be provided when using AdaLNZeroTeps return_gater) r<r=r2 attentionru feed_forwardrrattention_normffn_normr LayerNorm)rGr r3r6r7r5rwrxr4rrr8rr9rHrrr= s0  zTransformerBlock.__init__rrrL conditionrWrgrhr(cCs|jr|dus Jd|j||d\}} n||}d} |dur0|dur0|j||||\} } n|r>|j|||dd\} } n||||} |jrO|| | } n|| } |jr`|j| |d\} }n|| } || }|jrt| ||}n| |}| dur|| fS|S)a Forward pass for a single Transformer block. Args: x (torch.Tensor): Input tensor of shape (bsz, seqlen, dim). freqs_cis (torch.Tensor, optional): Precomputed rotary frequencies. mask (torch.Tensor, optional): Attention mask. condition (torch.Tensor, optional): Conditioning tensor for AdaLNZero. return_kv (bool): Whether to return KV pairs for caching. kv_cache (tuple, optional): KV cache for efficient inference. start_pos (int, optional): Starting position for KV cache. Returns: out (torch.Tensor): Output tensor of shape (bsz, seqlen, dim). new_kv (tuple, optional): New KV pairs if return_kv is True or kv_cache is provided. N/condition must be provided when using AdaLNZerorTrW)rrrrmrr)rGrrrLrrWrgrh attn_normed attn_gaterlattn_outh ffn_normedffn_gateffn_outoutrrrrf9s.   zTransformerBlock.forward)NFFF)NFNN) rnrorprqrPrr=rrrrsrfrtrrrHrr~ sj      2 r~c)sVeZdZ                  d1d ed ed ededededBdededBdededededededBdedBdedBdedededef(fdd Zed!efd"d#Zd$e j fd%d&Z d$e j fd'd(Z     d2d)e jd*e jdBd+e jdBd,ed-eee je jfdBd.edBd!e jee jeee je jffBfd/d0ZZS)3 Transformer FN皙?h㈵>TAr n_layersr3r6r7r5rwrxr4ruse_rope rope_theta max_seq_len input_dim output_dimrr8ruse_xavier_initr9cst||_||_| |_||_t|_t |D]}|j t |||||| ||| ||||d q|jrG|dus=Jdt ||| dd|_ ntj|| d|_ |durYt||nt|_|durht||nt|_|durs|n||_| rt||| d| |_td| d| d |d |d |jj nd|_|durtd ||jrtd ||rtd|rtd||j||jdSdS)N) r r3r5rwrxr4r6r7rrr8rr9rFrrrzUsing RoPE with theta=z, max_seq_len=z, dim=z , n_heads=z, freqs_cis shape=z'Using local attention with window size z0Using AdaLNZero conditioning with condition_dim=z@Using Flash Attention for memory-efficient attention computationz-Using Xavier initialization for linear layers)r<r=r r3rrr ModuleListlayersrangeappendr~rnormrr?Identity input_proj output_proj output_dim_rrloggerdebugr$apply _init_weights_init_adaln_zero)rGr rr3r6r7r5rwrxr4rrrrrrrr8rrr9layer_idrHrrr=~sl      zTransformer.__init__r(cCs|jSr|)r)rGrrrrszTransformer.output_dimmodulecCs>t|tjrtj|j|jdurtj|jdSdSdSr|) isinstancerr?initxavier_normal_weightr;zeros_rGrrrrrs  zTransformer._init_weightscCs:t|trtj|jdjtj|jdjdSdS)Nr)rrrrrcondition_projrr;rrrrrs zTransformer._init_adaln_zerorrLrrWrgrhc Cst|j\}}} |jr|dusJd|jdurR|dur|dn|} | |jjdkrFtd| d|jjddt|j|j| d|j|_|j |j |_|j} nd} | |}g} t |j D]:\} }|dur|dur||| |||| |d \}}| |q`|r||| ||d d \}}| |q`||| ||}q`|jr|j||d \}}n||}||}| r|| fS|S) a Forward pass for the Transformer model. Args: x (torch.Tensor): Input tensor of shape (bsz, seqlen, input_dim). mask (torch.Tensor, optional): Attention mask. condition (torch.Tensor, optional): Conditioning tensor for AdaLNZero. return_kv (bool): Whether to return KV pairs for caching. kv_cache (list, optional): List of KV caches for each layer for efficient inference. start_pos (int, optional): Starting position for KV cache. Returns: output (torch.Tensor): Output tensor of shape (bsz, seqlen, output_dim). new_kv_list (list, optional): List of new KV pairs for each layer if return_kv is True or kv_cache is provided. NrrrzInput sequence length z! exceeds precomputed RoPE length z. Recomputing freqs_cis.r)rgrhTrr)r$rrrwarningrr r3rtor rr%rrrr)rGrrLrrWrgrhrJrK_dim expected_lenr new_kv_listrlayerrlrarerrrrfs<       zTransformer.forward)rrrFFNrNrrTrrNNNFFTF)NNFNN)rnrorprqrPrr=propertyrrModulerrrrrlistrsrfrtrrrHrr}s     R "r)r )rtorch.nn.functionalr functionalr]utilr adaln_zerorr flash_attnrrr[ ImportErrorrqrrrrr'r1rr2rur~rrrrrs(    Er