o Ti 0@s8ddlZddlmZddlmZmZ GdddZdS)N) functional)BertSparseSelfAttentionSparsityConfigc@sleZdZdZeddZeddZeeddfdd Zeeddfd d Z ed d Z eddZ dS)SparseAttentionUtilsaThis class provides some utility functions that are use integrating sparse attention into transformer models. Such utilities include extending position embeddings, replacing current self-attention layer with sparse attention, padding sequences to multiple of block size, etc. cCs t|dr+|jjjjd}||ksJtd||}|jjjj|d|jjjj_nVt|dr}|j jjjj \}}|d8}td||}||ksJJ|d7}|j jjj ||}d}t |D]}|j jjjdd||||<||7}q^||j jjj_nt d||j_td|||S) aThis function extends the position embedding weights of a model loaded from a checkpoint. It assumes the new max position is bigger than the original max length. Arguments: model: required: a transformer model max_position: required: an integer determining new position embedding size Return: model: updated model; in which position embedding weights have been extended based on new size bertrrobertaNz}Please extend "extend_position_embedding" function to support your model type. It currently only supports "bert" & "roberta"!z Extended position embeddings to )hasattrr embeddingsposition_embeddingsweightsizemaxrepeatdatarshape new_emptyrange ValueErrorconfigmax_position_embeddingsprint)model max_positionoriginal_max_positionextend_multiples embed_sizeextended_position_embeddingkir!i/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/ops/sparse_attention/sparse_attention_utils.pyextend_position_embeddings<        z.SparseAttentionUtils.extend_position_embeddingcCs"||_||jd<td||S)aThis function updates the position embedding length of a tokenizer to a new max position. Arguments: tokenizer: required: a transformer tokenizer max_position: required: an integer determining new position embedding size Return: tokenizer: updated tokenizer; in which model maximum length has been extended based on new size model_max_lengthz+updated tokenizer model max imum length to )r$ init_kwargsr) tokenizerrr!r!r"!update_tokenizer_model_max_length?s z6SparseAttentionUtils.update_tokenizer_model_max_length) num_headscCsdt|dr||j_||j|jjj||St|dr.|d|j_||j|jjj||Std)aThis function replaces the self attention layers in model encoder with sparse self attention. It currently supports bert and roberta model and can be easily extended to any other models following similar steps here. For sparsityConfig, refer to the config class. Arguments: model: required: a transformer model max_position: required: an integer determining new position embedding size sparsity_config: optional: this parameter determines sparsity pattern configuration; it is based on SparsityConfig class Return: model: updated model; in which self attention layer has been replaced with DeepSpeed Sparse Self Attention layer. rrr zPlease extend "update_model_self_attention_to_sparse_self_attention" function to support your model type. It currently only supports "bert" & "roberta"!) r rr=replace_self_attention_layer_with_sparse_self_attention_layerrencoderlayerrr)rrsparsity_configr!r!r"7replace_model_self_attention_with_sparse_self_attentionPs  zLSparseAttentionUtils.replace_model_self_attention_with_sparse_self_attentioncCsD|D]}t||}|jjj|_|jjj|_|jjj|_||j_q|S)aMThis function replaces the self attention layers in attention layer with sparse self attention. For sparsityConfig, refer to the config class. Arguments: config: required: transformer model config layers: required: transformer model attention layers sparsity_config: optional: this parameter determines sparsity pattern configuration; it is based on SparsityConfig class Return: layers: updated attention layers; in which self attention layers have been replaced with DeepSpeed Sparse Self Attention layer. )r attentionselfquerykeyvalue)rlayersr-r,deepspeed_sparse_self_attnr!r!r"r*rs     zRSparseAttentionUtils.replace_self_attention_layer_with_sparse_self_attention_layerc Cs|dur|jn|jdd\}} || ||} | dkrh|dur8|j|| f|tjd} || } tj|| gdd}|durFtj|d| f|d}|durTtj|d| f|d}tj|d| fdd}tj|d| fdd}| |||||fS) a This function pads input tokens and attention mask on sequence length dimension to be multiple of block size. This is a requirement for Sparse Transformer in which the self attention layer works on sequences of length multiple of block size. It needs to be called in your model, such as BertModel, right before you calculate the embedding outputs. Note) 1- instead of passing your embedding layer to this function, you can simply add this function to your model. It can be more simplified if given attention_mask and/or token_type_ids are none. 2- you need to call unpad function before returning your model output to unpad the encoder sequence output. Arguments: block_size: required: an integer determining the block size of sparsity config. pad_token_id: required: an integer determining the pad token from the model config; such as bert.config.pad_token_id. input_ids: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary attention_mask: a torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. token_type_ids: a torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). position_ids: a torch.LongTensor of shape [batch_size, sequence_length] with the indices of positions of each input sequence tokens in the position embeddings. inputs_embeds: an optional torch.FloatTensor of shape [batch_size, sequence_length, hidden_size] that contains embedded representation and can be passed instead of input_ids directly. model_embeddings: an optional object. If inputs_embeds are not none, this will be your model embeddings such as BertEmbeddings from your model such as BertModel. You can move this function inside your model and use self.embeddings instead of passing this parameter. Return: pad_len: an integer determining how much inputs have been padded to transfer sequence length dimension to multiple of block size. input_ids: if input_ids are not none padded input_ids otherwise none. attention_mask: if attention_mask is not none padded attention_mask otherwise none. token_type_ids: if token_type_ids are not none padded token_type_ids otherwise none. position_ids: if position_ids are not none padded position_ids otherwise none. inputs_embeds: if inputs_embeds are not none padded inputs_embeds otherwise none. Nr)dtype)dim)r3F)rnew_fulltorchlongcatFpad) block_size input_idsattention_masktoken_type_ids position_ids inputs_embeds pad_token_idmodel_embeddings batch_sizeseq_lenpad_len pad_input_idspad_inputs_embedsr!r!r"pad_to_block_sizes z&SparseAttentionUtils.pad_to_block_sizecCs"|dkr|ddd| f}|S)aThis function unpads sequence output if inputs of the model were padded. This is a requirement for Sparse Transformer in which the self attention layer works on sequences of length multiple of block size. It needs to be called in your model, such as BertModel, right before you return the model outputs. Arguments: pad_len: required: an integer determining how much model inputs have been padded to transfer sequence length dimension to multiple of block size. sequence_output: required: sequence output of the encoder layer. Return: sequence_output: unpaded sequence output of the encoder layer. rNr!)rJsequence_outputr!r!r"unpad_sequence_outputsz*SparseAttentionUtils.unpad_sequence_outputN) __name__ __module__ __qualname____doc__ staticmethodr#r'rr.r*rMrOr!r!r!r"rs  *  !  1r)r;torch.nnrr>deepspeed.ops.sparse_attentionrrrr!r!r!r"s