o iC@sddlmZddlmZddlZddlmZddlmmZ ddlm Z ddl m Z ddl mZddlmZz ddlmZmZWn eyNd \ZZYnwGd d d ejZGd d d ejZdS))partial)OptionalN)Tensor)StochasticDepth)MHA)Mlp) layer_norm_fnRMSNormNNcsfeZdZddejejddddddddddffdd Zd ddZ   dd ed e efd d Z Z S)BlockNTFcst||_| |_| |_| |_|jr|jsJd|dur'tt|dd}|dur3ttd|d}|||_ |||_ t | dd|_ |||_ |||_t|jtjsf|||_t | dd|_|||_|jrtdusqJd t|j tjtfrt|j tjsJ|r|j D]}d |_qt|d r|jD]}d |_q|r|j D]}d |_qt|d r|jD] }d |_qdSdSdS) a For prenorm=True, this Block has a slightly different structure compared to a regular prenorm Transformer block. The standard block is: LN -> MHA -> Dropout -> Add -> LN -> MLP -> Dropout -> Add. [Ref: https://arxiv.org/abs/2002.04745] Here we have: Dropout -> Add -> LN -> MHA -> Dropout -> Add -> LN -> MLP, returning both the hidden_states (output of the MLP) and the residual. This is for performance reasons, as we can fuse the dropout, add and LayerNorm. The residual needs to be provided (except for the very first block). For prenorm=False, this Block has the same structure as a regular postnorm Transformer block: MHA -> Dropout -> Add -> LN -> MLP -> Dropout -> Add -> LN. return_residual: whether each of the sub-layers (mixer and mlp) will return the residual. This is for performance reason: for post-norm architecture, returning the input allows us to fuse the backward of nn.Linear with the residual connection. z5residual_in_fp32 is only compatible with prenorm=TrueN@ num_headshidden_featuresrow)modeTriton is not installedTnorm2)super__init__prenormfused_dropout_add_lnreturn_residualresidual_in_fp32rrrmixerdropout1r drop_path1norm1mlp isinstancennIdentitydropout2 drop_path2rr LayerNormr Dropout parameters_sequence_parallelhasattr_shared_params)selfdim mixer_clsmlp_clsnorm_cls dropout_clsrresid_dropout1resid_dropout2rr&rrrsequence_parallelmark_shared_paramsp __class__N/home/ubuntu/vllm_env/lib/python3.10/site-packages/flash_attn/modules/block.pyrsP #         zBlock.__init__cK|jj||fd|i|SNdtyperallocate_inference_cacher- batch_size max_seqlenr>kwargsr:r:r;r@izBlock.allocate_inference_cache hidden_statesresidualc Cs|jr|js/|||}|dur||n|}||j|jjjd}|jr.|t j }n@|jj dks8|j s;d}n|t j |jdd|j|jd}t||jj|jj||jj|j r`|jj nd|d|jt|jtd \}}|durui}|dur}||d <|j|fi|}|dur|dd|f}t|jtjs |js|||}|dur||n|}||j|jjjd}|jr|t j }n@|jj dks|j sd}n|t j |jdd|j|jd}t||jj|jj||jj|j r|jj nd|d|jt|jtd \}}||}||fS|dusJ|j|fi|dur!|ni}|jr-|\}}|jsF|||||j|jjjd}n?|jj dksQ|j sTd}n|t j |jdd|j|jd}t||jj|jj||jj|j rz|jj nd|d t|jtd }t|jtjs||} |jr| \} }|js|||| |j|jjjd}|S|jj dks|j sd}n|t j | jdd| j| jd}t| |jj|jj||jj|j r|jj nd|d t|jtd }|S) aPass the input through the encoder layer. Args: hidden_states: the sequence to the encoder layer (required). residual: if postnorm, residual=None, If prenorm, hidden_states = Attn/MLP(LN(residual)) mixer_subset: for cross-attention only. If not None, will take a subset of x before applying the query projection. Useful for e.g., ViT where we only care about the CLS token in the last layer. Nr>r)devicer>r T)rGeps dropout_prowscalerr is_rms_norm mixer_subsetF)rGrKrLrMrrN)rrrrr toweightr>rtorchfloat32r7trainingonesshaperJrbiasrKr"r rr!r#r$r&r%rr) r-rFrGrO mixer_kwargsdropped rowscale1 rowscale2 mixer_outmlp_outr:r:r;forwardls             z Block.forwardNNNN) __name__ __module__ __qualname__r#r'r(rr@rrr^ __classcell__r:r:r8r;r s2 Sr c sleZdZdZddejejdddddddf fdd ZdddZ   dd e d e e d e e fd d Z Z S) ParallelBlockziThe attention (mixer) and MLP blocks are done in parallel, similar to GPT-J, GPT-NeoX, and PaLM. Nr Fc sHt||_| |_| |_|durtt|dd}|dur&ttd|d}|||_|||_ |||_ |||_ |||_ |jsG|||_ |jrdtdusRJdt|j tjtfrbt|j tjsdJ| r|j D]} d| _qkt|dr|j D]} d| _q{| r|j D]} d| _qt|dr|j D] } d| _qdSdSdS) a This Block has a slightly different structure compared to a regular prenorm Transformer block. The standard block is: LN -> MHA / MLP -> Dropout -> Add. [Ref: https://arxiv.org/abs/2002.04745] Here we have: Dropout -> Add -> LN -> MHA / MLP, returning both the hidden_states (output1 of the MHA / MLP) and the residual. This is for performance reasons, as we can fuse the dropout, add and LayerNorm. The residual needs to be provided (except for the very first block). Nr rrrrTr)rr tied_normrrrrrrrr r!r%rrr"r#r'r r(r)r*r+r,)r-r.r/r0r1r2r3r4rfrrr5r6r7r8r:r;rsF          zParallelBlock.__init__cKr<r=r?rAr:r:r;r@LrEz&ParallelBlock.allocate_inference_cachehidden_states1hidden_states2rGc CsP|jsQ||}|dur ||}|dur|||n||}n |dur(||n|}||j|jjjd}|jsE||j|jjjdn|}|j rP|t j }n>|js\|jj|jj fnd\}}t ||jj|jj |||||jj|jrv|jjndd|j t|jtd ^}} }|jr|}n| \}|duri}|j|fi|}||}|||fS)aPass the input through the encoder layer. Args: hidden_states1: the output of the previous attention (mixer) or embedding layer. hidden_states2: the output of the previous MLP layer (if None, will use hidden_states1). residual. NrHr r T) rGx1weight1bias1rKrLrrrN)rrr%r rPrQr>rfrrrRrSrWrrKrTr7r"r rr!) r-rgrhrGrXdropped1dropped2weight2bias2restr:r:r;r^OsR       zParallelBlock.forwardr_r`) rarbrc__doc__r#r'r(rr@rrr^rdr:r:r8r;res2 Dre) functoolsrtypingrrRtorch.nnr#torch.nn.functional functionalFrtorchvision.opsrflash_attn.modules.mharflash_attn.modules.mlpr flash_attn.ops.triton.layer_normrr ImportErrorModuler rer:r:r:r;s"         o