o Tiߥ@sddlZddlZGdddZGdddeZGdddeZGdd d eZGd d d eZGd d d eZGdddeZdS)Nc@s*eZdZdZd ddZddZdd Zd S) SparsityConfigaAbstract Configuration class to store `sparsity configuration of a self attention layer`. It contains shared property of different block-sparse sparsity patterns. However, each class needs to extend it based on required property and functionality. FcCs*||_||_||_|r||_dSd|_dS)aInitialize the Sparsity Pattern Config. For usage example please see, TODO DeepSpeed Sparse Transformer Tutorial Arguments: num_heads: required: an integer determining number of attention heads of the layer. block: optional: an integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices. In which this parameter defines size of such blocks, `Block X Block`. different_layout_per_head: optional: a boolean determining if each head should be assigned a different sparsity layout; default is false and this will be satisfied based on availability. N) num_headsblockdifferent_layout_per_headnum_layout_headsselfrrrr b/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/ops/sparse_attention/sparsity_config.py__init__s zSparsityConfig.__init__cCsL||jdkrtd|d|jd||j}tj|j||ftjd}|S)aECreate layout tensor for the given sequence length Arguments: seq_len: required: an integer determining number of attention heads of the layer. Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) for sparsity layout of all head; initialized with zero rzSequence Length, z&, needs to be dividable by Block size !)dtype)r ValueErrortorchzerosrint64)r seq_len num_blockslayoutr r r setup_layouts   zSparsityConfig.setup_layoutcCs8|js|dddddf|d|jddddf<|S)aIf all heads require same sparsity layout, it propagate first head layout to all heads Arguments: layout: required: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head; may not be completely set at this step Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head rNr)rr)r rr r r %check_and_propagate_first_head_layout0s .z4SparsityConfig.check_and_propagate_first_head_layoutNrF)__name__ __module__ __qualname____doc__r rrr r r r r s   rcs*eZdZdZdfdd ZddZZS) DenseSparsityConfigzConfiguration class to store `Dense` configuration. In reality, this is not sparse and all blocks are used. We keep it for the sake of comparison and comprehension. rFcst|||dS)a!Initialize the Dense Sparsity Pattern Config. In reality, this is not sparse and all blocks are used. We keep it for the sake of comparison and comprehension. Arguments: num_heads: required: an integer determining number of attention heads of the layer. seq_len: required: an integer determining number of attention heads of the layer. different_layout_per_head: optional: this is just for the sake of consistency with other sparsity formats; can ignore it for DenseSparsityConfig N)superr r  __class__r r r Ds zDenseSparsityConfig.__init__cCs(||}d|ddddddf<|S)aSet 1 to all blocks of the layout meaning the pattern is dense; not sparse. Arguments: seq_len: required: an integer determining the underling sequence length; must be <= max sequence length Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head; for dense everything is 1 rN)r)r rrr r r make_layoutPs zDenseSparsityConfig.make_layoutr)rrrrr r" __classcell__r r r r r?s rcsHeZdZdZ       dfdd Zd d Zd d Zd dZZS)FixedSparsityConfigaIConfiguration class to store `Fixed` sparsity configuration. For more details about this sparsity config, please see `Generative Modeling with Sparse Transformers`: https://arxiv.org/abs/1904.10509; this has been customized. This class extends parent class of `SparsityConfig` and customizes it for `Fixed` sparsity. rFr bidirectionalc st|||||_||dkrtd|d|d||_|dkr+|dkr+td||_|dkr8|r8td||_|d krE|sEtd |||kr^td |d |d |d||d ||_dS)a Initialize `Fixed` Sparsity Pattern Config. For usage example please see, TODO DeepSpeed Sparse Transformer Tutorial Arguments: num_heads: required: an integer determining number of attention heads of the layer. block: optional: an integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices. In which this parameter defines size of such blocks, `Block X Block`. different_layout_per_head: optional: a boolean determining if each head should be assigned a different sparsity layout; default is false and this will be satisfied based on availability. num_local_blocks: optional: an integer determining the number of blocks in local attention window. num_global_blocks: optional: an integer determining how many consecutive blocks in a local window is used as the representative of the window for global attention. attention: optional: a string determining attention type. Attention can be `unidirectional`, such as autoregressive models, in which tokens attend only to tokens appear before them in the context. Considering that, the upper triangular of attention matrix is empty as above figure. Or it can be `bidirectional`, such as BERT, in which tokens can attend to any other tokens before or after them. Then, the upper triangular part of the attention matrix is mirror of the lower triangular in the above figure. horizontal_global_attention: optional: a boolean determining if blocks that are global representative of a local window, also attend to all other blocks. This is valid only if attention type is `bidirectional`. Looking at the attention matrix, that means global attention not only includes the vertical blocks, but also horizontal blocks. num_different_global_patterns: optional: an integer determining number of different global attentions layouts. While global attention can be fixed by which block/s are representative of any local window, since there are multi-heads, each head can use a different global representative. For example, with 4 blocks local window and global attention size of 1 block, we can have 4 different versions in which the first, Second, third, or forth block of each local window can be global representative of that window. This parameter determines how many of such patterns we want. Of course, there is a limitation based on num_local_blocks and num_global_blocks. rz$Number of blocks in a local window, z0, must be dividable by number of global blocks, runidirectionalr&;only "uni/bi-directional" attentions are supported for now!Ionly "bi-directional" attentions can support horizontal global attention!rzNumber of different layouts cannot be more than one when you have set a single layout for all heads! Set different_layout_per_head to True.z;Number of layout versions (num_different_global_patterns), zZ, cannot be larger than number of local window blocks divided by number of global blocks, z / z = N) rr num_local_blocksrnum_global_blocksNotImplementedError attentionhorizontal_global_attentionnum_different_global_patterns) r rrrr*r+r-r.r/r r r r es,      zFixedSparsityConfig.__init__cCsr|jd}td||jD]*}t||j|}t||D]}t||jdkr(|dn|D] }d||||f<q+qq |S)aSets local attention layout used by the given head in the sparse attention. Arguments: h: required: an integer determining head index layout: required: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head; may not be completely set at this step Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head in which local layout is set rrr')shaperanger*minr-)r hrriendrowcolr r r set_local_layouts  z$FixedSparsityConfig.set_local_layoutc Cs|jd}|jd||j|j}|||j}t|||jD]*}|jdkr)dn|}d|||d|||jf<|jrJd|||||jddf<q ||krt||||j}||j}|jdkredn|}d|||d||f<|jrd||||ddf<|S)a-Sets global attention layout used by the given head in the sparse attention. Currently we set global blocks starting from the last block of a local window to the first one. That means if a local window consists of 4 blocks and global attention size is one block, we use block #4 in each local window as global. If we have different layout per head, then other heads will get #3, #2, and #1. And if we have more heads (and different layout has set) than num of global attentions, multiple head may have same global attentions. Note) if horizontal_global_attention is set, global blocks will be set both horizontally and vertically. Arguments: h: required: an integer determining head index layout: required: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head; may not be completely set at this step Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head in which global layout is set rr&rN)r0r*r/r+r1r-r.r2) r r3rrfirst_global_block_idxr5r4 first_rowstartr r r set_global_layouts&  z%FixedSparsityConfig.set_global_layoutcCB||}td|jD]}|||}|||}q ||}|S)aWGenerates `Fixed` sparsity layout used by each head in the sparse attention. Arguments: seq_len: required: an integer determining number of attention heads of the layer. Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing `Fixed` sparsity layout of all head r)rr1rr8r<rr rrr3r r r r"   zFixedSparsityConfig.make_layout)rFr%rr&Fr) rrrrr r8r<r"r#r r r r r$_s40r$csVeZdZdZddddgdgdddffdd Zd d Zd d ZddZddZZ S)VariableSparsityConfigaOConfiguration class to store `Variable` sparsity configuration. This layout is an extension of FixedSparsityConfig in which: - user can set random layout; default value is zero means no random block - user can provide a list of local block sizes - user can provide a list of global block indices. For more details about `Fixed` sparsity config, please see `Generative Modeling with Sparse Transformers`: https://arxiv.org/abs/1904.10509; this has been customized. This class extends parent class of `SparsityConfig` and customizes it for `Fixed` sparsity. rFrr%Nr&c st|||||_||_||_|durIt|t|kr,tdt|dt|dtt||D]\} \} } | | krHtd| d| dq3||_ |dkrX|dkrXt d ||_ |dkre| retd | |_ dS) a Initialize `Variable` Sparsity Pattern Config. For usage example please see, TODO DeepSpeed Sparse Transformer Tutorial Arguments: num_heads: required: an integer determining number of attention heads of the layer. block: optional: an integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices. In which this parameter defines size of such blocks, `Block X Block`. different_layout_per_head: optional: a boolean determining if each head should be assigned a different sparsity layout; default is false and this will be satisfied based on availability. Currently this sparsity config can only assign single layout to all heads; needs to be extended for different layout per head. num_random_blocks: optional: an integer determining the number of random blocks in each block row. local_window_blocks: optional: a list of integers determining the number of blocks in each local attention window. It assumes first number determines # of blocks in the first local window, second the second window, ..., and the last number determines the number of blocks in the remaining local windows. global_block_indices: optional: a list of integers determining which blocks are considered as global attention. Given indices, determine the blocks that all other token blocks attend to and they attend to all other token blocks. Default value is only index 0. Notice that if global_block_end_indices parameter is set, this parameter is used as starting index of each global window. global_block_end_indices: optional: a list of integers determining end indices of global window blocks. By default this is not used. But if it is set, it must have the same size of global_block_indices parameter, and combining this two parameters, for each index i, blocks from global_block_indices[i] to global_block_end_indices[i] (exclusive) are considered as global attention. num_global_blocks: optional: an integer determining how many consecutive blocks in a local window is used as the representative of the window for global attention. attention: optional: a string determining attention type. Attention can be `unidirectional`, such as autoregressive models, in which tokens attend only to tokens appear before them in the context. Considering that, the upper triangular of attention matrix is empty as above figure. Or it can be `bidirectional`, such as BERT, in which tokens can attend to any other tokens before or after them. Then, the upper triangular part of the attention matrix is mirror of the lower triangular in the above figure. horizontal_global_attention: optional: a boolean determining if blocks that are global representative of a local window, also attend to all other blocks. This is valid only if attention type is `bidirectional`. Looking at the attention matrix, that means global attention not only includes the vertical blocks, but also horizontal blocks. N#Global block start indices length, 3, must be same as global block end indices length, rGlobal block start index, /, must be smaller than global block end index, r'r&r(r)) rr num_random_blockslocal_window_blocksglobal_block_indiceslenr enumeratezipglobal_block_end_indicesr,r-r.) r rrrrErFrGrKr-r._ start_idxend_idxr r r r s,  zVariableSparsityConfig.__init__cCsb|jd}||jkrtd|jd|dtd|D]}ttd||j}d||||f<q|S)Sets random attention layout used by the given head in the sparse attention. Note) By default, it assumes there will be a unique random block layout for all heads; unless `different_layout_per_head` parameter is set in which each head can have a different random layout. Arguments: h: required: an integer determining head index layout: required: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head; may not be completely set at this step Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head in which random layout is set rNumber of random blocks, :, must be smaller than overall number of blocks in a row, rr)r0rErr1randomsample)r r3rrr6rnd_colsr r r set_random_layout/s z(VariableSparsityConfig.set_random_layoutc Cs|jd}d}d}|jD]/}||7}t||}t||D]}t||jdkr)|dn|D] }d||||f<q,q||7}q t|||D])} t| ||}t| |D]}t| |jdkr]|dn|D] }d||||f<q`qPqB|S)aSets local attention layout used by the given head in the sparse attention. Arguments: h: required: an integer determining head index layout: required: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head; may not be completely set at this step Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head in which local layout is set rrr')r0rFr2r1r-) r r3rrstart_block_idx end_block_idx block_sizer6r7r4r r r r8Es&      z'VariableSparsityConfig.set_local_layoutc Cs|jd}|jdur4|jD]$}||kr1|jrd|||ddf<|jdkr&dn|}d|||d|f<q |Stt|j|jD]1\}\}}||krnt||}|jrZd||||ddf<|jdkradn|}d|||d||f<q=|S)Sets global attention layout used by the given head in the sparse attention. Arguments: h: required: an integer determining head index layout: required: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head; may not be completely set at this step Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head in which global layout is set rNr&r)r0rKrGr.r-rIrJr2) r r3rridxr:rLrMrNr r r r<bs&    z(VariableSparsityConfig.set_global_layoutcCN||}td|jD]}|||}|||}|||}q ||}|S)a]Generates `Variable` sparsity layout used by each head in the sparse attention. Arguments: seq_len: required: an integer determining number of attention heads of the layer. Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing `Variable` sparsity layout of all head r)rr1rrUr8r<rr>r r r r"    z"VariableSparsityConfig.make_layout) rrrrr rUr8r<r"r#r r r r r@s 5%r@csNeZdZdZ      dfdd Zd d Zd d Zd dZddZZ S)BigBirdSparsityConfiga5Configuration class to store `BigBird` sparsity configuration. For more details about this sparsity config, please see `Big Bird: Transformers for Longer Sequences`: https://arxiv.org/pdf/2007.14062.pdf This class extends parent class of `SparsityConfig` and customizes it for `BigBird` sparsity. rFrr&csDt|||||_||_||_|dkr|dkrtd||_dS)aInitialize the BigBird Sparsity Pattern Config. For usage example please see, TODO DeepSpeed Sparse Transformer Tutorial Arguments: num_heads: required: an integer determining number of attention heads of the layer. block: optional: an integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices. In which this parameter defines size of such blocks, `Block X Block`. different_layout_per_head: optional: a boolean determining if each head should be assigned a different sparsity layout; default is false and this will be satisfied based on availability. num_random_blocks: optional: an integer determining the number of random blocks in each block row. num_sliding_window_blocks: optional: an integer determining the number of blocks in sliding local attention window. num_global_blocks: optional: an integer determining how many consecutive blocks, starting from index 0, are considered as global attention. Global block tokens will be attended by all other block tokens and will attend to all other block tokens as well. attention: optional: a string determining attention type. Attention can be `unidirectional`, such as autoregressive models, in which tokens attend only to tokens appear before them in the context. Considering that, the upper triangular of attention matrix is empty as above figure. Or it can be `bidirectional`, such as BERT, in which tokens can attend to any other tokens before or after them. Then, the upper triangular part of the attention matrix is mirror of the lower triangular in the above figure. r'r&r(N)rr rEnum_sliding_window_blocksr+r,r-)r rrrrEr_r+r-r r r r s zBigBirdSparsityConfig.__init__cCs~|jd}||jkrtd|jd|dtd|D]!}|jdkr'td|ntd|d}t||j}d||||f<q|S)rOrrPrQrrr&)r0rErr1r-rRrS)r r3rrr6 sample_rangerTr r r rUs "z'BigBirdSparsityConfig.set_random_layoutcC||jd}||jkrtd|jd|d|jd}td|D]}td||}t||d|}d|||||f<q |SaSets sliding local attention layout used by the given head in the sparse attention. Arguments: h: required: an integer determining head index layout: required: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head; may not be completely set at this step Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head in which local sliding window layout is set r!Number of sliding window blocks, rQrrr0r_rr1maxr2r r3rrwr6r;r5r r r set_sliding_window_layout  z/BigBirdSparsityConfig.set_sliding_window_layoutcCst|jd}||jkrtd|jd|dd||d|jddf<d||ddd|jf<|jdkr8t|}|S)rYrzNumber of global blocks, rQrrNr')r0r+rr-rtril)r r3rrr r r set_global_layout_itcs   z+BigBirdSparsityConfig.set_global_layout_itccCr[)a[Generates `BigBird` sparsity layout used by each head in the sparse attention. Arguments: seq_len: required: an integer determining number of attention heads of the layer. Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing `BigBird` sparsity layout of all head r)rr1rrUrirlrr>r r r r"r\z!BigBirdSparsityConfig.make_layout)rFrr^rr&) rrrrr rUrirlr"r#r r r r r]s r]csHeZdZdZddddgddffdd Zd d Zd d ZddZZS)BSLongformerSparsityConfigaConfiguration class to store edited `Longformer` sparsity configuration. Note) this is a block-sparse version of the Longformer which is slightly different than original Longformer; which is element-wise sparsity. For more details about this sparsity config, please see `Longformer: The Long-Document Transformer`: https://arxiv.org/pdf/2004.05150.pdf This class extends parent class of `SparsityConfig` and customizes it for `Longformer` sparsity. rFr^rNr&c st|||||_||_||_|durIt|t|kr,tdt|dt|dtt||D]\}\} } | | krHtd| d| dq3||_ dS)aQInitialize the edited `Longformer` Sparsity Pattern Config. For usage example please see, TODO DeepSpeed Sparse Transformer Tutorial Arguments: num_heads: required: an integer determining number of attention heads of the layer. block: optional: an integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices. In which this parameter defines size of such blocks, `Block X Block`. different_layout_per_head: optional: a boolean determining if each head should be assigned a different sparsity layout; default is false and this will be satisfied based on availability. num_sliding_window_blocks: optional: an integer determining the number of blocks in sliding local attention window. global_block_indices: optional: a list of integers determining which blocks are considered as global attention. Given indices, determine the blocks that all other token blocks attend to and they attend to all other token blocks. Default value is only index 0. Notice that if global_block_end_indices parameter is set, this parameter is used as starting index of each global window. global_block_end_indices: optional: a list of integers determining end indices of global window blocks. By default this is not used. But if it is set, it must have the same size of global_block_indices parameter, and combining this two parameters, for each index i, blocks from global_block_indices[i] to global_block_end_indices[i] (exclusive) are considered as global attention. attention: optional: a string determining attention type. Attention can be `unidirectional`, such as autoregressive models, in which tokens attend only to tokens appear before them in the context. Considering that, the upper triangular of attention matrix is empty as above figure. Or it can be `bidirectional`, such as BERT, in which tokens can attend to any other tokens before or after them. Then, the upper triangular part of the attention matrix is mirror of the lower triangular in the above figure. NrArBrrCrD) rr r_rGr-rHrrIrJrK) r rrrr_rGrKr-rLrMrNr r r r +s  z#BSLongformerSparsityConfig.__init__cCrarbrergr r r riTrjz4BSLongformerSparsityConfig.set_sliding_window_layoutcCs|jd}|jdur'|jD]}||kr%d|||ddf<d||dd|f<q n/tt|j|jD]%\}\}}||krUt||}d||||ddf<d||dd||f<q0|jdkr`t|}|S)rYrNr') r0rKrGrIrJr2r-rrk)r r3rrrZrLrMrNr r r r<ls"      z,BSLongformerSparsityConfig.set_global_layoutcCr=)ajGenerates edited `Longformer` sparsity layout used by each head in the sparse attention. Arguments: seq_len: required: an integer determining number of attention heads of the layer. Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing `BSLongformer` sparsity layout of all head r)rr1rrir<rr>r r r r"r?z&BSLongformerSparsityConfig.make_layout) rrrrr rir<r"r#r r r r rm"s )#rmcs2eZdZdZd fdd ZddZd d ZZS) LocalSlidingWindowSparsityConfigzConfiguration class to store `Local Sliding Window` sparsity configuration - a purely-local sliding window attention. This class extends parent class of `SparsityConfig` and customizes it for `Local` sparsity. rr^r'cst||||_||_dS)akInitialize the Local Sliding Window Sparsity Pattern Config. For usage example please see, TODO DeepSpeed Sparse Transformer Tutorial Arguments: num_heads: required: an integer determining number of attention heads of the layer. block: optional: an integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices. In which this parameter defines size of such blocks, `Block X Block`. num_sliding_window_blocks: optional: an integer determining the number of blocks in sliding local attention window. attention: optional: a string determining attention type. Attention can be `unidirectional`, such as autoregressive models, in which tokens attend only to tokens appear before them in the context. Considering that, the upper triangular of attention matrix is empty as above figure. Or it can be `bidirectional`, such as BERT, in which tokens can attend to any other tokens before or after them. Then, the upper triangular part of the attention matrix is mirror of the lower triangular in the above figure. N)rr r_r-)r rrr_r-r r r r s  z)LocalSlidingWindowSparsityConfig.__init__cCs|jd}||jkrtd|jd|d|jd}td|D]$}td||}|jdkr7t||d|n|d}d|||||f<q |S)aSets sliding local attention layout used by the given head in the sparse attention. Arguments: h: required: an integer determining head index layout: required: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head; may not be completely set at this step Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing sparsity layout of all head in which local sliding window layout is set rrcrQrrdrr&)r0r_rr1rfr-r2rgr r r ris  $z:LocalSlidingWindowSparsityConfig.set_sliding_window_layoutcCs6||}td|jD]}|||}q ||}|S)afGenerates `Local Sliding Window` sparsity layout used by each head in the sparse attention. Arguments: seq_len: required: an integer determining number of attention heads of the layer. Return: layout: a tensor of dimension (num_heads, num_blocks, num_blocks) containing `BigBird` sparsity layout of all head r)rr1rrirr>r r r r"s  z,LocalSlidingWindowSparsityConfig.make_layout)rr^r')rrrrr rir"r#r r r r rns rn) rrRrrr$r@r]rmrnr r r r s5 -