o %ݫi&@sdZddlZddlmZmZddlmZddlmZddl m Z m Z Gdddej j ZGd d d eZGd d d ej j ZdS) zThis is a module to ensemble a convolution (depthwise) encoder with or without residual connection. Authors * Jianyuan Zhong 2020 * Titouan Parcollet 2023 N)Conv1dConv2d) Sequential) LayerNorm)FilterPropertiesstack_filter_propertiescs6eZdZdZdddejjffdd ZddZZ S) ConvolutionalSpatialGatingUnitaThis module implementing CSGU as defined in: Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding" The code is heavily inspired from the original ESPNet implementation. Arguments --------- input_size: int Size of the feature (channel) dimension. kernel_size: int, optional Size of the kernel dropout: float, optional Dropout rate to be applied at the output use_linear_after_conv: bool, optional If True, will apply a linear transformation of size input_size//2 activation: torch.class, optional Activation function to use on the gate, default is Identity. Example ------- >>> x = torch.rand((8, 30, 10)) >>> conv = ConvolutionalSpatialGatingUnit(input_size=x.shape[-1]) >>> out = conv(x) >>> out.shape torch.Size([8, 30, 5]) gFc st||_||_||_|jddkrtd|d}t||_tdd|f||dd|ddd|_ |jrRt j |||_ t j jj|j jd d t j j|j jt j j|j j jt j ||_dS) Nrz"Input size must be divisible by 2!samenormalF) input_shape out_channels kernel_sizestridepaddinggroups conv_initskip_transposegư>)std)super__init__ input_sizeuse_linear_after_conv activation ValueErrorrnormrconvtorchnnLinearlinearinitnormal_weightones_biasDropoutdropout)selfrrr)rr n_channels __class__X/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/convolution.pyr2s0   z'ConvolutionalSpatialGatingUnit.__init__cCsN|jddd\}}||}||}|jr||}||}|||S)z Arguments --------- x: torch.Tensor Input tensor, shape (B, T, D) Returns ------- out: torch.Tensor The processed outputs. r )dim)chunkrrrr"rr))r*xx1x2r.r.r/forwardYs    z&ConvolutionalSpatialGatingUnit.forward) __name__ __module__ __qualname____doc__rr Identityrr6 __classcell__r.r.r,r/rs!'rcsdeZdZdZddgdgdgdgdgdeejjed d d d ffd d Z de fddZ Z S)ConvolutionFrontEndaThis is a module to ensemble a convolution (depthwise) encoder with or without residual connection. Arguments --------- input_shape: tuple Expected shape of the input tensor. num_blocks: int Number of block (default 21). num_layers_per_block: int Number of convolution layers for each block (default 5). out_channels: Optional(list[int]) Number of output channels for each of block. kernel_sizes: Optional(list[int]) Kernel size of convolution blocks. strides: Optional(list[int]) Striding factor for each block, this stride is applied at the last convolution layer at each block. dilations: Optional(list[int]) Dilation factor for each block. residuals: Optional(list[bool]) Whether apply residual connection at each block (default None). conv_module: class Class to use for constructing conv layers. activation: Callable Activation function for each block (default LeakyReLU). norm: torch class Normalization to regularize the model (default BatchNorm1d). dropout: float Dropout (default 0.1). conv_bias: bool Whether to add a bias term to convolutional layers. padding: str Type of padding to apply. conv_init: str Type of initialization to use for conv layers. Example ------- >>> x = torch.rand((8, 30, 10)) >>> conv = ConvolutionFrontEnd(input_shape=x.shape) >>> out = conv(x) >>> out.shape torch.Size([8, 8, 3, 512]) )i)r>r>r>)r r r )r r r )TTT皙?Tr Ncsbtj|dt|D]#}|jt|||||||||||| | | | d|| ||dq dS)Nr convblock_) num_layersrrrdilationresidual conv_modulerrr) layer_name conv_biasrr)rrrangeappend ConvBlock)r*r num_blocksnum_layers_per_blockr kernel_sizesstrides dilations residualsrHrrr)rJrrir,r.r/rs( zConvolutionFrontEnd.__init__returncCstdd|DS)Ncss|]}|VqdSN)get_filter_properties).0blockr.r.r/ s z.)rchildrenr*r.r.r/rWsz)ConvolutionFrontEnd.get_filter_properties) r7r8r9r:rrr LeakyReLUrrrrWr<r.r.r,r/r=rs$0&r=c sReZdZdZddddeejjdddddf fd d Zd d Z d e fddZ Z S)rMaAn implementation of convolution block with 1d or 2d convolutions (depthwise). Arguments --------- num_layers : int Number of depthwise convolution layers for this block. out_channels : int Number of output channels of this model (default 640). input_shape : tuple Expected shape of the input tensor. kernel_size : int Kernel size of convolution layers (default 3). stride : int Striding factor for this block (default 1). dilation : int Dilation factor. residual : bool Add a residual connection if True. conv_module : torch class Class to use when constructing conv layers. activation : Callable Activation function for this block. norm : torch class Normalization to regularize the model (default BatchNorm1d). dropout : float Rate to zero outputs at. conv_bias : bool Add a bias term to conv layers. padding : str The type of padding to add. conv_init : str Type of initialization to use for conv layers. Example ------- >>> x = torch.rand((8, 30, 10)) >>> conv = ConvBlock(2, 16, input_shape=x.shape) >>> out = conv(x) >>> x.shape torch.Size([8, 30, 10]) r>r FNrBTr c s$tt|d|_g|_t|D]S}||dkr|nd}|jj|||||d|| | |d |jt|||d| durJ|jj| d|d|jj| d|d|jjtj | d |dqd|_ d|_ |rt|d|_ |j j||d|d d |j j| d dtj | |_ dSdS) NrCr conv_)rrrrFrIr'rr) window_sizerrFnorm_)rIact_dropout_r)rrrrIr) rrrconvsfilter_propertiesrKrLrrr r( reduce_convdrop)r*rErrrrrFrGrHrrr)rJrrrT layer_strider,r.r/rsV     zConvBlock.__init__cCs,||}|jr|||}||}|S)z:Processes the input tensor x and returns an output tensor.)rcrerf)r*r3outr.r.r/r65s  zConvBlock.forwardrUcCs t|jSrV)rrdr\r.r.r/rW=s zConvBlock.get_filter_properties) r7r8r9r:rrr r]rr6rrWr<r.r.r,r/rMs />rM)r:rspeechbrain.nnet.CNNrrspeechbrain.nnet.containersrspeechbrain.nnet.normalizationr!speechbrain.utils.filter_analysisrrr Modulerr=rMr.r.r.r/s  _Z