o TiZ@sBddlmZddlZddlmZddlmZGdddejZdS)N) distributed)SparsityConfigcsbeZdZdZedddddffdd ZeZd d Zd d Z d dZ dddZ dddZ Z S)SparseSelfAttentiona9Implements an efficient Sparse Self Attention of Transformer layer based on `Generative Modeling with Sparse Transformers`: https://arxiv.org/abs/1904.10509 For more information please see, TODO DeepSpeed Sparse Transformer. For usage example please see, TODO DeepSpeed Sparse Transformer Tutorial. ) num_headsaddmulics>t||_|j|}|d|d|_||_||_dS)atInitialize the sparse self attention layer. Arguments: sparsity_config: optional: this parameter determines sparsity pattern configuration; it is based on SparsityConfig class. key_padding_mask_mode: optional: a string determining if key padding mask needs to be added, `add`, or be multiplied, `mul`. attn_mask_mode: optional: a string determining if attention mask needs to be added, `add`, or be multiplied, `mul`. max_seq_length: optional: the maximum sequence length this sparse attention module will be applied to; it controls the size of the master_layout. master_layoutTN)super__init__sparsity_config make_layoutregister_buffer_need_layout_synchronizationkey_padding_mask_modeattn_mask_mode)selfr rrmax_seq_lengthr  __class__h/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/ops/sparse_attention/sparse_self_attention.pyr s    zSparseSelfAttention.__init__cCsv|jrtrtj|jddd|_||jjdkr'td|d|jjd||jj}|jdd|d|fS)Nr)srcFzSequence Length, z&, needs to be dividable by Block size !.) rdistis_initialized broadcastr r block ValueErrorcpu)rL num_blocksrrr get_layout2s zSparseSelfAttention.get_layoutc Csddlm}ddlm}|tjvr;||}|||jjdddd}|||jjdddd}|||jj}|||ftj|<tj|S) Nr)MatMul)SoftmaxsddFT)trans_atrans_bdsd) %deepspeed.ops.sparse_attention.matmulr#&deepspeed.ops.sparse_attention.softmaxr$ropsr"r r) rHr r#r$sparsity_layoutsparse_dot_sdd_ntsparse_dot_dsd_nnsparse_softmaxrrrget_ops@s     zSparseSelfAttention.get_opscCs,|\}}}}||kr|ddddS|S)Nr)sizepermute)rxr bszrseq_lenhead_dimrrrtranspose_key_for_scoresRsz,SparseSelfAttention.transpose_key_for_scoresFcCsD||}|r|}t|dddD]}|j|d}q|S|S)Nr2r)dim)typer=rangesqueeze)rqtyper7is_key_padding_maskxdimdrrrtranspose_mask_for_sparseXs z-SparseSelfAttention.transpose_mask_for_sparseNc Cs|jtjks Jd|\}}} } ||| }|j|jks$|j|jkr(td|dur5|j|j|dd}|dur@||j|}||| \} } } t | d}| ||}| ||||||j |j d}| ||}|S)a4Applies forward phase of sparse self attention Arguments: query: required: query tensor key: required: key tensor value: required: value tensor rpe: optional: a tensor same dimension as x that is used as relative position embedding key_padding_mask: optional: a mask tensor of size (BatchSize X SequenceLength) attn_mask: optional: a mask tensor of size (SequenceLength X SequenceLength); currently only 2D is supported key_padding_mask_mode: optional: a boolean determining if key_padding_mask needs to be added or multiplied attn_mask_mode: optional: a boolean determining if attn_mask needs to be added or multiplied Return: attn_output: a dense tensor containing attention context znsparse attention only supports training in fp16 currently, please file a github issue if you need fp32 supportz(only self-attention is supported for nowNT)rBg)scalerpekey_padding_mask attn_maskrr) dtypetorchhalfr5r;shapeNotImplementedErrorrEr1floatrr)rquerykeyvaluerGrHrIr8rtgt_lenr:r.r/r0scalingattn_output_weights attn_outputrrrforwardbs,    zSparseSelfAttention.forward)F)NNN)__name__ __module__ __qualname____doc__rr dictr+r"r1r;rErW __classcell__rrrrr s   r) torch.nnnnrKrrdeepspeed.ops.sparse_attentionrModulerrrrrs