o %ݫiM3@sdZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z ddl mZGdd d ejZGd d d ejZGd d d ejZdS)aBranchformer implementation. Ref: "Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding" Source: Some parts of the code may be adapted from ESPNet. Authors * Titouan Parcollet 2023 )OptionalN)ConvolutionalSpatialGatingUnit)MultiheadAttention RelPosMHAXL) HyperMixing) LayerNormcs:eZdZdZddejejddffdd Zdd ZZ S) ConvolutionBranchaThis is an implementation of the convolution branch in Branchformer. The default structure is: LN -> Channel Proj -> GeLU -> (CNN Spatial Gating) -> Channel Proj -> Dropout Arguments --------- input_size : int The expected size of the feature (channel) dimension. linear_units: int, optional Number of neurons in the hidden linear units. kernel_size: int, optional Kernel size of non-bottleneck convolutional layer. activation: torch.nn.Module, optional Activation function used after pre projection. gate_activation: torch.nn.Module, optional Activation function used at the gate of the CSGU module. dropout: float, optional Dropout rate. use_linear_after_conv: bool, optional If True, will apply a linear transformation of size input_size//2 Example ------- >>> x = torch.rand((8, 60, 512)) >>> net = ConvolutionBranch(512, 1024) >>> output = net(x) >>> output.shape torch.Size([8, 60, 512]) FcsJtt|||_t|d||_||_t|||||d|_dS)N) input_size kernel_sizedropoutuse_linear_after_conv activation) super__init__nnLinearpre_channel_projpost_channel_projrrcsgu)selfr linear_unitsrrgate_activationrr __class__e/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/transformer/Branchformer.pyr7s  zConvolutionBranch.__init__cCs(|||}||}||}|S)zT Arguments ---------- x: torch.Tensor -> (B, T, D) )rrrr)rxrrrforwardNs  zConvolutionBranch.forward) __name__ __module__ __qualname____doc__rGELUIdentityrr! __classcell__rrrrrs"rc sheZdZdZdddejdddejdf fdd Z   dd ee j d ee j d ee j fd dZ Z S)BranchformerEncoderLayeraThis is an implementation of Branchformer encoder layer. Arguments --------- d_model : int The expected size of the input embedding. nhead : int Number of attention heads. kernel_size : int, optional Kernel size of convolution model. kdim : int, optional Dimension of the key. vdim : int, optional Dimension of the value. activation: torch.nn.Module Activation function used in each Conformer layer. dropout : int, optional Dropout for the encoder. attention_type: str, optional type of attention layer, e.g. regularMHA for regular MultiHeadAttention. csgu_linear_units: int, optional Number of neurons in the hidden linear units of the CSGU Module. gate_activation: torch.nn.Module, optional Activation function used at the gate of the CSGU module. use_linear_after_conv: bool, optional If True, will apply a linear transformation of size input_size//2 Example ------- >>> import torch >>> x = torch.rand((8, 60, 512)) >>> pos_embs = torch.rand((1, 2*60-1, 512)) >>> net = BranchformerEncoderLayer(nhead=8, d_model=512, kernel_size=3) >>> output = net(x, pos_embs=pos_embs) >>> output[0].shape torch.Size([8, 60, 512]) r Nr rr Fc st|dkrt|||||d|_n|dkr"t|||dd|_n|dkr2t||dd|dd|_t||| || || d |_tj |d ||_ t ||_ t ||_t ||_dS) N regularMHA)nheadd_modelrkdimvdimrF) num_heads embed_dimrmask_pos_future hypermixing)input_output_dim hypernet_sizetiedr/fix_tm_hidden_size)r rrrrrrr )rrr mha_layerrrrconvolution_branchtorchrr merge_projr norm_mhsa norm_convDropoutr) rr,r+rr-r.rrattention_typecsgu_linear_unitsrrrrrrsJ     z!BranchformerEncoderLayer.__init__src_masksrc_key_padding_maskpos_embsc Cs~|}|}||}|j||||||d\}}||}||}||}||}|||tj||gdd}||fS)a Arguments ---------- x : torch.Tensor The sequence to the encoder layer. src_mask : torch.Tensor, optional The mask for the src sequence. src_key_padding_mask : torch.Tensor, optional The mask for the src keys per batch. pos_embs: torch.Tensor, torch.nn.Module, optional Module or tensor containing the input sequence positional embeddings ) attn_maskkey_padding_maskrC)dim)r<r8rr=r9r;r:cat)rr rArBrCx1x2 self_attnrrrr!s"     "z BranchformerEncoderLayer.forward)NNN r"r#r$r%rr&r'rrr:Tensorr!r(rrrrr)\s,*<r)c sneZdZdZdddejdddejdddf fdd Z    dd ee j d ee j d ee j fd dZ Z S)BranchformerEncoderaThis class implements the Branchformer encoder. Arguments --------- num_layers : int Number of layers. d_model : int Embedding dimension size. nhead : int Number of attention heads. kernel_size : int, optional Kernel size of convolution model. kdim : int, optional Dimension of the key. vdim : int, optional Dimension of the value. activation: torch.nn.Module Activation function used in each Confomer layer. dropout : int, optional Dropout for the encoder. attention_type: str, optional type of attention layer, e.g. regularMHA for regular MultiHeadAttention. csgu_linear_units: int, optional Number of neurons in the hidden linear units of the CSGU Module. gate_activation: torch.nn.Module, optional Activation function used at the gate of the CSGU module. use_linear_after_conv: bool, optional If True, will apply a linear transformation of size input_size//2. output_hidden_states: bool, optional Whether the model should output the hidden states as a list of tensor. layerdrop_prob: float The probability to drop an entire layer. Example ------- >>> import torch >>> x = torch.rand((8, 60, 512)) >>> pos_emb = torch.rand((1, 2*60-1, 512)) >>> net = BranchformerEncoder(1, 512, 8) >>> output, _ = net(x, pos_embs=pos_emb) >>> output.shape torch.Size([8, 60, 512]) >>> import torch >>> x = torch.rand((8, 60, 512)) >>> pos_emb = torch.rand((1, 2*60-1, 512)) >>> net = BranchformerEncoder(1, 512, 8, output_hidden_states=True) >>> output, attn_list, hidden_list = net(x, pos_embs=pos_emb) >>> hidden_list[0].shape torch.Size([8, 60, 512]) >>> len(hidden_list) 2 r Nr rr Fc sbttj f ddt|D|_tdd|_||_ |_ | |_ dS)Ncs*g|]}t  d qS)) r+r,r-r.rrrr?r@rr)r)).0i rr?r@r,rrr-rr+rr.rr 9s z0BranchformerEncoder.__init__..gư>)eps) rrr:r ModuleListrangelayersrnormlayerdrop_probr?output_hidden_states)r num_layersr,r+rr-r.rrr?r@rrrYrXrrQrr%s  zBranchformerEncoder.__init__rArBrCc Cs|dusJd|jdkr|durtd|}|jdkr$tt|j}g}|jr,|g} t|jD]*\} } |j rD|jdksD|| |jkr[| ||||d\}} | | |jr[| |q1| |}|jri||| fS||fS)a Arguments --------- src : torch.Tensor The sequence to the encoder layer. src_mask : torch.Tensor, optional The mask for the src sequence. src_key_padding_mask : torch.Tensor, optional The mask for the src keys per batch. pos_embs: torch.Tensor, torch.nn.Module, Module or tensor containing the input sequence positional embeddings If custom pos_embs are given it needs to have the shape (1, 2*S-1, E) where S is the sequence length, and E is the embedding dimension. dynchunktrain_config : None This configuration is unsupported for this encoder. Returns ------- output : torch.Tensor The output of the Conformer. attention_lst : list The attention values. hidden_state_lst : list, optional The output of the hidden layers of the encoder. Only works if output_hidden_states is set to true. Nz3Dynamic Chunk Training unsupported for this encoderrzThe chosen attention type for the Branchformer is RelPosMHAXL. For this attention type, the positional embeddings are mandatoryr )rArBrC) r? ValueErrorrXr:randlenrVrY enumeratetrainingappendrW) rsrcrArBrCdynchunktrain_configoutput keep_probs attention_lsthidden_state_lstrP enc_layer attentionrrrr!OsB #       zBranchformerEncoder.forward)NNNNrLrrrrrNs2<-rN)r%typingrr:torch.nnr$speechbrain.lobes.models.convolutionrspeechbrain.nnet.attentionrrspeechbrain.nnet.hypermixingrspeechbrain.nnet.normalizationrModulerr)rNrrrrs     E