o i_ @sTdZddlZddlmZddlmZddlZ e e e j j ZGdddejZdS)z-Class Declaration of Transformer's Attention.Ncs,eZdZdZd fdd Zd dd ZZS) MultiHeadAttentionaMulti Head Attention Layer. Args: n_units (int): Number of input units. h (int): Number of attention heads. dropout (float): Dropout rate. initialW: Initializer to initialize the weight. initial_bias: Initializer to initialize the bias. :param int h: the number of heads :param int n_units: the number of features :param float dropout_rate: dropout rate 皙?Ncstt|||dksJdt|}|Htj||||d||dd|_tj||||d||dd|_ tj||||d||dd|_ tj||||d||dd|_ Wdn1sewY|||_ ||_ ||_d|_dS)zInitialize MultiHeadAttention.rg?)scale)initialW initial_biasN)superr__init__npsqrt init_scopeLLinearlinear_qlinear_klinear_v linear_outd_khdropoutattn)selfn_unitsrrrrstvd __class__e/home/ubuntu/.local/lib/python3.10/site-packages/espnet/nets/chainer_backend/transformer/attention.pyr sB    zMultiHeadAttention.__init__c Csp|j}|dur/|||d|j|j}|||d|j|j}|||d|j|j}n'|||d|j|j}|||d|j|j}|||d|j|j}tt |dd| ddddt |j} |dur|j |g|jdd}t|| || jtd} tj| dd|_t|j|j} t| t |dd} t | ddd|j|j} || S) aCore function of the Multi-head attention layer. Args: e_var (chainer.Variable): Variable of input array. s_var (chainer.Variable): Variable of source array from encoder. mask (chainer.Variable): Attention mask. batch (int): Batch size. Returns: chainer.Variable: Outout of multi-head attention layer. Nrr)axisf)xprreshaperrrrFmatmulswapaxes transposer r stackwherefullshape MIN_VALUEsoftmaxrrr) re_vars_varmaskbatchr$QKVscoresp_attnxrrrforward?s& $ zMultiHeadAttention.forward)rrNN)NNr)__name__ __module__ __qualname____doc__r r: __classcell__rrrrr s#r)r>chainerchainer.functions functionsr& chainer.linkslinksr numpyr floatfinfofloat32minr.Chainrrrrrs