o ei0@sfdZddlZddlmZddlZddlmZGdddejZGdddejZGd d d ejZ dS) zThis module mixes information from different tokens via HyperMixing. It can be viewed as a linear-time drop-in replacement for (self-)attention. source: https://arxiv.org/abs/2203.03691 Authors * Florian Mai 2023 * Juan Pablo Zuluaga 2023 N)Optional)nncseZdZdZ    ddedededed ed ed d ffd d ZddZ  ddee j dee j deedee j fddZ Z S) HyperMixingaThis class implements multi-head HyperMixing. It is an implementation of the token-mixing component in HyperMixer, a linear time drop-in replacement for self-attention. In contrast to the original HyperMixer, this module supports multiple heads, which improves the expressiveness of the model while decreasing the number of parameters. Reference: https://arxiv.org/abs/2203.03691 Arguments --------- input_output_dim : int number of features in keys, queries, and values hypernet_size : int determines the size of the hidden layer of the token-mixing MLP. tied : bool If True, then the generated weight matrices of the token-mixing MLP are tied. num_heads : int parallel token-mixing MLPs. fix_tm_hidden_size : bool If True, the hidden-layer size is equal to hypernet_size rather than hypernet_size / num_heads. max_length : int Maximum number of input tokens. Needed for generating sufficiently large position embeddings. Example ------- >>> import torch >>> inputs = torch.rand([8, 60, 512]) >>> net = HyperMixing(512, 2048, num_heads=8) >>> outputs, attn = net(inputs, inputs, inputs) >>> outputs.shape torch.Size([8, 60, 512]) F input_output_dim hypernet_sizetied num_headsfix_tm_hidden_size max_lengthreturnNcs\t||_t|||||d|_t|_t||_ ||_ ddl m }||||_ dS)N)r r keep_output_sizer)PositionalEncoding)super__init__r HyperNetworkhyperrGELU activation LayerNorm layer_normr 0speechbrain.lobes.models.transformer.Transformerrpositional_encoding)selfrrr r r r r __class__Z/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/nnet/hypermixing.pyr4s     zHyperMixing.__init__cCs,t||}||}t||dd}|S)z(function to stick MLP1 together manuallyr)torchbmm transpose)routW1W2rrrr_mlp_pass_from_componentsSs z%HyperMixing._mlp_pass_from_componentsT attn_maskkey_padding_maskreturn_attn_weightspos_embscCs*|}|d} |d} |durt|d} || }|||} || \} }|dur>| | d} || d}|dd}|| |j |j |j | f}| | |j | df} || |j | df}| || ||j }|| |j | f}|dd}| |}tj| | | f|jd}||fS)a The signature of this method is deliberately chosen to be the same as for sb.nnet.attention.MultiHeadAttention for compatibility within SpeechBrain. NOTE: key, value, attn_mask and pos_embs have no effect. Query is used for all three. Thus, the module should only be used to replace self-attention at the moment. Arguments ---------- query : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. key : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. Currently unused. All value : torch.Tensor (B, S, E) where S is the source sequence length, B is the batch size, E is the embedding dimension. Currently unused. attn_mask : torch.Tensor, optional NOTE: Currently has NO effect. key_padding_mask : torch.Tensor, optional (B, S) where B is the batch size, S is the source sequence length. If a ByteTensor is provided, the non-zero positions will be ignored while the position with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged. return_attn_weights: torch.Tensor, optional NOTE: Currently has NO effect. pos_embs: torch.Tensor, optional NOTE: Currently has NO effect. Outputs ------- attn_output : torch.Tensor (B, L, E) where L is the target sequence length, B is the batch size, E is the embedding dimension. attn_output_weights : torch.Tensor (B, L, S) where B is the batch size, L is the target sequence length, S is the source sequence length. NOTE: always returns all zeros. rrNr)device)sizer logical_not unsqueezefloatrrr"reshaper rr&rrzerosr,)rquerykeyvaluer'r(r)r*r#bsizeseq_len float_mask hyp_inputr$r%dummy_att_weightsrrrforwardZs>8       zHyperMixing.forward)FrFr)NNTN) __name__ __module__ __qualname____doc__intboolrr&rr Tensorr; __classcell__rrrrrsF% rcsFeZdZdZ   ddededdffd d Zd ejfd d ZZ S)raThis class implements The HyperNetwork. It is an approach of using a one network, also known as a hypernetwork, to generate the weights for another network. Here, it is used to generate the labels of linear layers. Reference: https://arxiv.org/abs/1609.09106 Arguments ---------- input_output_dim : int Dimension of the linear layers hypernet_size: Dimension of the HyperNetwork tied : bool, optional Define whether weights of layer 1 and layer 2 are shared num_heads: int, optional Number of heads, akin to heads in MultiHeadAttention keep_output_size: bool, optional Set whether to keep the same output size independent of number of heads FrTrrr NcsRtt|||_t|||||d|_|jr|j|_dSt|||||d|_dS)N) output_sizenum_mlpsr)rrrr ParallelMLPsw1_genw2_gen)rrrr r rrrrrs$  zHyperNetwork.__init__ input_tensorcCs.||}|jr|}||fS||}||fS)aForward computation for a HyperNetwork. Arguments ---------- input_tensor : [batchsize, max_positions, d] The HyperNetwork is supposed to generate an MLP of the form W_2(GELU(W1 x)), where W1 : N -> k and W2 : k -> N, so it has to return tensors W1 and W2 Outputs ------- W1 : torch.Tensor Generated weights of Layer 1 W2 : torch.Tensor Generated weights of Layer 2 )rGr rH)rrIr$r%rrrr;s  zHyperNetwork.forward)FrT) r<r=r>r?r@rr rBr;rCrrrrrsrcs4eZdZdZ   d d fdd Zdd ZZS) rFaClass that implements the MultiHead HyperMixer or HyperConformer. Arguments ---------- input_size : int Dimension of the linear layers hidden_size: int Dimension of the hidden layer output_size : int Dimension of the HyperNetwork num_mlps : int Number of heads, akin to heads in MultiHeadAttention keep_output_size : bool, optional Set whether to keep the same output size independent of number of heads NrTr csFtt||dur |}||_||_||dksJ||dks#J||dks+J||}|s5||}||}||_||_||_t t ||||_ t t |||_ t t ||||_t t |||_tjj|j tddtjj|j tddtjj|jtddtjj|jtddt|_dS)Nrg@)gain)rrFroriginal_in_sizeoriginal_out_size input_sizerDrEr Parameterr empty fc1_weights fc1_biases fc2_weights fc2_biasesinitxavier_uniform_mathsqrtrr)rrM hidden_sizerDrErrrrr#s8  zParallelMLPs.__init__cCs||d}|d}||||j|jf}td||j|jdd}| |}td||j |j dd}|S)zPerforms the forward computation of multi parallel MLPs. Arguments ---------- x : tensor Input tensor Outputs ------- x : torch.Tensor return output tensor rrzblmf,mhf->bmlhrzbmlh,mfh->bmlf) r-r1rErMr einsumrPrQr/rrRrS)rxr6r7rrrr;Ss   zParallelMLPs.forward)NrT)r N)r<r=r>r?rr;rCrrrrrFs0rF) r?rVtypingrr rModulerrrFrrrrs  5L