o ߥi@s:dZddlZddlZddlZddlmmZddlmZ ddl m Z Gdddejj Z Gdddejj ZGd d d ejj Zejjd d Zd dZGdddejj ZGdddejj ZGdddejj ZddZddZGdddejj ZGdddejj ZGdddejj ZGdd d ejj ZdS)!z Transformer.N)FusedLayerNorm)mpucs&eZdZfddZdddZZS)PositionalEmbeddingcs>tt|||_ddtd|d|}|d|dS)Ni'@inv_freq)superr__init__ hidden_sizetorcharangeregister_buffer)selfr r __class__`/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/mglm/model/transformer.pyr s zPositionalEmbedding.__init__NcCsft||j}tj||gdd}|dur(|dddddf|ddS|dddddfS)Ndim)r gerrcatsincosexpand)rpos_seqbsz sinusoid_inppos_embrrrforward$s  zPositionalEmbedding.forwardN)__name__ __module__ __qualname__r r __classcell__rrrrrs  rcs4eZdZdZ d fdd ZddZddZZS) ParallelCrossAttentionz.Parallel cross-attention layer for TransformerNcstt||dur |}t}t|||_t|||_t|||_tj ||d|d|_ tj |d|dd|d|_ t j ||_tj||d|d|_t j ||_tjrftjjt_tjjt_dSdS)NF gather_output init_methodstrider(r)Tinput_is_parallelr))r r&r rget_model_parallel_world_sizedividehidden_size_per_partitionhidden_size_per_attention_head!num_attention_heads_per_partitionColumnParallelLinearquery key_valuer nnDropoutattention_dropoutRowParallelLineardenseoutput_dropout deepspeed checkpointing is_configuredget_cuda_rng_tracker checkpoint)rr num_attention_headsattention_dropout_proboutput_dropout_probr)output_layer_init_method world_sizerrrr 1sH   zParallelCrossAttention.__init__cC6|dd|j|jf}|j|}|ddddSz_Transpose a 3D tensor [b, s, np*hn] into a 4D tensor with size [b, np, s, hn]. Nrrr*rsizer3r2viewpermutertensornew_tensor_shaperrr_transpose_for_scores` z,ParallelCrossAttention._transpose_for_scorescCs(||}||}t|d\}}||}||} ||} t|| dd} | t |j } |durCt | |dd|} tj j dd| } t || } Wdn1sbwYt| | } | dddd } | dd|jf}| j|} || }||}|S) Nr*rg@?rrrrI)r5r6rsplit_tensor_along_last_dimrQr matmul transposemathsqrtr2mulr7Softmaxr@forkr9rM contiguousrKr1rLr;r<)r hidden_statesencoder_states cross_maskmixed_query_layer mixed_x_layermixed_key_layermixed_value_layer query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputrrrr js>             zParallelCrossAttention.forwardr!)r"r#r$__doc__r rQr r%rrrrr&.s / r&csReZdZdZ    d fdd ZddZedd d Z    dd d ZZ S)ParallelSelfAttentionaParallel self-attention layer for GPT2. Self-attention layer takes input with size [b, s, h] where b is the batch size, s is the sequence length, and h is the hidden size and creates output of the same size. Arguments: hidden_size: total hidden size of the layer (h). num_attention_heads: number of attention heads (n). Note that we require n to be divisible by number of GPUs used to parallelize the model. Also, we require hidden size to be divisible by n. attention_dropout_prob: dropout probability for the attention scores. init_method: weight initialization. output_layer_init_method: output layer initialization. If None, use `init_method`. We use the following notation: h: hidden_size n: num_attention_heads p: number of partitions np: n/p hp: h/p hn: h/n b: batch size s: sequence length NFrTc stt|||_|dur|}t} t|| |_t|||_t|| |_ ||_ | |_ tj |d|dd|d|_ |rHtj ||d|d|_tj||_tj||d|d|_tj||_tjrqtjjt_tjjt_dSdS)NrIFr+r'Tr-)r rnr performerrr/r0r1r2r3relative_encodingattention_scaler4query_key_valuerelativer r7r8r9r:r;r<r=r>r?r@rA) rr rBrCrDr)rErprorqrFrrrr sP   zParallelSelfAttention.__init__cCrGrHrJrNrrrrQrRz+ParallelSelfAttention._transpose_for_scorescCstjg|dd|ddR|j|jd}tj||gdd}|jg|dd|dd|dR}|ddddddf|}|rvt|d|df}|t ||d|dddddddf}|S)NrSrdevicedtyperrr) r zerosrKrurvrrLview_asonestril)x zero_triuzero_padx_paddedryrrr _rel_shifts&4 4z ParallelSelfAttention._rel_shiftcCs\|d}|dur||}t|d\} } } n!t||fd} || }t|d\} } } | dd| df} || } || }|| }|jr||}||}| | d}t || dd}| | d}t || dd}| |}||}|t |j}n/|jdkrt | t |j| ddt |j|j}nt | | ddt |j}t||}|jdkr|jdddd}||8}||j9}|d d|}tjjdd |}t ||}Wdn1swYt ||}|dd dd}|dd|jf}|j|}||}||}|S) NrrIrrSrTT)rkeepdimrgrr*)rKrrrrUr rrQrprs unsqueezerVrWrrXrYr2rqrZmaxr7r[r@r\r9rMr]r1rLr;r<)rr^ ltor_maskposition_embeddingsr_w_biasr_r_biasmem query_lengthrbrarcrdrrerfrgrelative_layer rw_head_qac_score rr_head_qbd_scorerhmax_attention_scoresrirjrkrlrrrr s                     zParallelSelfAttention.forwardNFFrTFNNNN) r"r#r$rmr rQ staticmethodrr r%rrrrrns 6 rncCs*d|dtd|dd||S)zOpenAI's gelu implementation.g?rTgQ63E?gHm?)r tanhr{rrr gelu_implfs  rcCst|Sr!)rrrrrgelunsrcs,eZdZdZ dfdd ZddZZS) ParallelMLPaMLP for GPT2. MLP will take the input with h hidden state, project it to 4*h hidden dimension, perform gelu transformation, and project the state back into h hidden dimension. At the end, dropout is also applied. Arguments: hidden_size: The hidden size of the self attention. output_dropout_prob: dropout probability for the outputs after self attention and final output. init_method: initialization method used for the weights. Note that all biases are initialized to zero and layernorm weight are initialized to one. output_layer_init_method: output layer initialization. If None, use `init_method`. Ncs\tt||dur |}tj|d|d|d|_tjd||d|d|_tj ||_ dS)NFr'Tr-) r rr rr4 dense_h_to_4hr: dense_4h_to_hr r7r8dropout)rr rDr)rErrrr s zParallelMLP.__init__cCs*||}t|}||}||}|Sr!)rrrr)rr^intermediate_parallelrlrrrr s   zParallelMLP.forwardr!r"r#r$rmr r r%rrrrrrs rcs0eZdZdZ dfdd Z dddZZS)ParallelDecoderLayerA single layer transformer for GPT2. We use the following notation: h: hidden size n: number of attention heads b: batch size s: sequence length Transformore layer takes input with size [b, s, h] and returns an output of the same size. Arguments: hidden_size: The hidden size of the self attention. num_attention_heads: number of attention head in the self attention. attention_dropout_prob: dropout probability of the attention score in self attention. output_dropout_prob: dropout probability for the outputs after self attention and final output. layernorm_epsilon: epsilon used in layernorm to avoid division by zero. init_method: initialization method used for the weights. Note that all biases are initialized to zero and layernorm weight are initialized to one. output_layer_init_method: output layers (attention output and mlp output) initialization. If None, use `init_method`. Ncstt||dur |}t||d|_t||||||d|_t||d|_t||||||d|_ t||d|_ t ||||d|_ dS)NepsrE) r rr LayerNorminput_layernormrnself_attentionpost_self_layernormr&cross_attentionpost_attention_layernormrmlp)rr rBrCrDlayernorm_epsilonr)rErrrr s@   zParallelDecoderLayer.__init__c Cs^||}|||}||}||}||||} || } || }||} | | } | Sr!)rrrrrr) rr^r_rr`layernorm_outputself_attention_outputself_layernorm_inputself_layernorm_outputattention_outputlayernorm_input mlp_outputrlrrrr s    zParallelDecoderLayer.forwardr!rrrrrrs #4rcs<eZdZdZ    d fdd Z    d ddZZS) ParallelTransformerLayerrNFrTc shtt||dur |}t||d|_t|||||||| | d |_t||d|_t||||d|_ dS)NrrErprorqr) r rr rrrn attentionrrr) rr rBrCrDrr)rErprorqrrrr 0s0   z!ParallelTransformerLayer.__init__c Cs\||}|dur||nd}|||||||}||} || }||} | | } | Sr!)rrrr) rr^rrrrrrrrrrlrrrr Zs   z ParallelTransformerLayer.forwardrrrrrrrrs#-rcsfdd}|S)z!Init method based on N(0, sigma).ctjjj|ddSNrmeanstdr r7initnormal_rOsigmarrinit_zz#unscaled_init_method..init_r)rrrrrunscaled_init_methodws rcs"|td|fdd}|S)z3Init method based on N(0, sigma/sqrt(2*num_layers).rcrrrrrrrrrz!scaled_init_method..init_)rXrY)r num_layersrrrrscaled_init_methods rcsPeZdZdZ         dfdd Z  dd d Zdd dZZS)GPT2ParallelTransformeraGPT-2 transformer. This module takes input from embedding layer and it's output can be used directly by a logit layer. It consists of L (num-layers) blocks of: layer norm self attention residual connection layer norm mlp residual connection followed by a final layer norm. Arguments: num_layers: Number of transformer layers. hidden_size: The hidden size of the self attention. num_attention_heads: number of attention head in the self attention. attention_dropout_prob: dropout probability of the attention score in self attention. output_dropout_prob: dropout probability for the outputs after self attention and final output. checkpoint_activations: if True, checkpoint activations. checkpoint_num_layers: number of layers to checkpoint. This is basically the chunk size in checkpoitning. layernorm_epsilon: epsilon used in layernorm to avoid division by zero. init_method_std: standard deviation of the init method which has the form N(0, std). use_scaled_init_for_output_weights: If Ture use 1/sqrt(2*num_layers) scaling for the output weights ( output of self attention and mlp). rh㈵>{Gz?TFrTc stt||_| |_| |_||_ |_ |_ r rJd| r(t |t j ||_ |_||_ rt|_t}t|_t||_t j t |j|j|_d|j_t j t |j|j|_d|j_t |j|jWdn1swYn7|rt j |d|_t j |d|_t j j j!|jj"ddnt j ||_t j j j!|jj"dd f ddt j #fddt$|D|_%t&d |_'t(j)*rt(j)j+t_+t(j)j,t_,dSdS) NTrrrc s> rttdStt d S)Nrr)rrrr) rCrqr init_method_stdrrBrDrErorpuse_decoder_layerrr get_layers, z3GPT2ParallelTransformer.__init__..get_layercsg|]}qSrr).0_)rrr sz4GPT2ParallelTransformer.__init__..r)-r rr r checkpoint_activationscheckpoint_num_layersmax_memory_lengthrorrr r7r8embedding_dropoutrpblock_position_encodingrrrr/r0r2r3 ParameterTensorrmodel_parallelrno_gradzero_ Embeddingblock_position_embeddingsrrweight ModuleListrangelayersrfinal_layernormr=r>r?r@rA)rrr rBmax_sequence_lengthrembedding_dropout_probrCrDrrrr"use_scaled_init_for_output_weightsrprrorrqrFr) rCrqrr rrrBrDrErorprrr s         z GPT2ParallelTransformer.__init__Ncsdd\}|r|ddnd} || } t|dkp(t|k} jr:s2Jd| dks:Jd| rXrB|n|} dfdd } jsW| || | d}n|dddddd| | df}jrtj| dd d jjd } |} |}n)j r|dddf|dddf}} |}|j r |}| fd d j dksˆrȈgngfdd}jr"d}tj}j}||kr!js|gn||g}jr||jjg7}|r |||||7}tj||||g|R||7}||ksnEtjD]?\}}js3|gn||g}jrE||jjg7}|rL||nd}||d|ij dks^req'}j dksur}j|d|fS)Nr*rrzFattention_mask should be a scalar to indicate the seperation position.zDo not support transformer-xl.csd||f}t|}rd|dddd|f<n)|dd}tj||j|jddd}||ddk}|| d |d}|dkr^|dd}tj ||f|fdd}| d}|S)Nrrrrtr*r) new_onesr rzrr rurvrL masked_fillr expand_asr) seq_lengthsep memory_lengthmidsmask) batch_sizer^ is_scalarrrbuild_mask_matrix2s2  z:GPT2ParallelTransformer.forward..build_mask_matrix)rrgrtcsr|S|Sr!)detach)_hidden_states) detach_memoryrr check_detachesz5GPT2ParallelTransformer.forward..check_detachcsfdd}|S)Ncsj}|d|dd}}jr#|dd|dd}}n |dd|dd}}t|D]%\}}|r>||nd}||g|Rd|i}jdksRrY|q4|S)Nrrrr)rrp enumeraterappend)inputslayers_x_mems_ilayermem_i_)rend mem_layers return_memoryrstartrrcustom_forwardqszGGPT2ParallelTransformer.forward..custom..custom_forwardr)rrr)rrrr)rrrcustomosz/GPT2ParallelTransformer.forward..customr)r)r)rKr numelroitemrpr rurvrrrrrrlenrrrrrrrArrr update_mems)rr^ position_idsattention_mask memory_statesr_rrrr key_lengthis_seprrposition_sequencerblock_position_idsrrlr chunk_lengthargsrrmem_irlr)rrrr^rrrrrr s              zGPT2ParallelTransformer.forwardc Cs|r |ddnd}|dd}||}|st|j|}g}tt|D]2}||kr=|||dd| dfq&|tj||dd| |df||fddq&|S)Nrrr)rKminrrrrr r) rhiddensmemsrrrnew_memory_lengthnew_memsrrrrrs" " z#GPT2ParallelTransformer.update_mems) rrrTFFFFrT)NNFTr)r"r#r$rmr r rr%rrrrrs&-t rcs8eZdZdZdejffdd ZddZddZZ S) BertParallelSelfAttentionaParallel self-attention layer for BERT. Self-attention layer takes input with size [b, s, h] where b is the batch size, s is the sequence lenght, and h is the hidden size and creates output of the same size. Arguments: hidden_size: total hidden size of the layer (h). num_attention_heads: number of attention heads (n). Note that we require n to be divisible by number of GPUs used to parallelize the model. Also, we require hidden size be divisible by n. dropout_prob: dropout probability for the attention scores. output_parallel: If true, no all-gather is done on the output and the output values will be per partition. We use the following notation: h: hidden_size n: num_attention_heads p: number of partitions np: n/p hp: h/p hn: h/n b: batch size s: sequence length Fcstt|||_||_||_||_t}t |||_ t |||_ t |||_ tj |d|dd|d|_tj||_tjrQtjjt_tjjt_dSdS)NrIFr+)r rr r rB dropout_proboutput_parallelrr/r0r1r2r3r4rrr r7r8rr=r>r?r@rA)rr rBrrr)rFrrrr s2  z"BertParallelSelfAttention.__init__cCrGrHrJrNrrrrQrRz/BertParallelSelfAttention._transpose_for_scorescCs||}t|d\}}}||}||}||} tt|j} t|| | dd| } | |7} tj j dd| } t || } Wdn1sWwYt| | } | dddd} | dd|jf}| j|} |jr| }|St| }|S)NrIrrSrrr*r)rrrrUrQrXrYr2r rVrWr7r[r@r\rrMr]rKr1rLr!gather_from_model_parallel_region)rr^rrbrarcrdrerfrg norm_factorrhrirjrkrlrrrr s<        z!BertParallelSelfAttention.forward) r"r#r$rmrxavier_normal_r rQr r%rrrrrs# rcs2eZdZdZddejffdd ZddZZS)BertParallelTransformerOutputz[The output layer used after self attention and intermediate parts of transformer layer.g-q=FcsBtt|tj||||d|_tj||_ t ||d|_ dS)Nr-r) r rr rr:r;r r7r8rr layernorm)r input_size output_sizerrr.r)rrrr 7sz&BertParallelTransformerOutput.__init__cCs*||}||}||}||}|Sr!)r;rr)rr^ input_tensorrrrrr Hs   z%BertParallelTransformerOutput.forward r"r#r$rmrrr r r%rrrrr3srcs.eZdZdZejffdd ZddZZS)BertParallelTransformerLayera_A single layer transformer for Bert. We use the following notation: h: hidden size n: number of attention heads b: batch size s: sequence length Transformore layer takes input with size [b, s, h] and returns an output of the same size. Arguments: hidden_size: The hidden size of the self attention. intermediate_size: size of the intermediate state after self attention. In both BERT and GPT this is set to be 4 times the hidden size. num_attention_heads: number of attention head in the self attention. attention_dropout_prob: dropout probability of the attention score in self attention. output_dropout_prob: dropout probability for the outputs after self attention and final output. intermediate_activation_fn: activation function for output of intermediate. layernorm_epsilon: epsilon used in layernorm to avoid division by zero. init_method: initialization method used for the weights. Note that all biases are initialized to zero and layernorm weight are initialized to one. c sltt|t|||d|d|_t||||d|d|_tj||d|d|_ ||_ t||||d|d|_ dS)NT)rr))rr.r)Fr') r r"r rrr self_outputrr4 intermediateintermediate_activation_fnrl) rr intermediate_sizerBrCrDr%rr)rrrr ps>  z%BertParallelTransformerLayer.__init__cCs<|||}|||}||}||}|||}|Sr!)rr#r$r%rl)rr^rattention_output_parallelattention_self_outputintermediate_output_parallel layer_outputrrrr s z$BertParallelTransformerLayer.forwardr!rrrrr"Ps '*r")rmrXr=r torch.nn.initr7r#apex.normalization.fused_layer_normrr megatron_utilrModulerr&rnjitscriptrrrrrrrrrrr"rrrrs2   lM 5ld  5v