o i'@sdZddlmZddlmZmZmZddlZddlZddl m Z ddl m Z ddl mZmZGdd d e ZGd d d ejjZdS) z&Sinc convolutions for raw audio input.) OrderedDict)OptionalTupleUnionN)check_argument_types) AbsPreEncoder)LogCompressionSincConvcseZdZdZ       d'd eeeefd ed ed ed ededeffdd ZddZ      d(d ed edededef ddZ ddZ d e j d!e j d"ee j e j ffd#d$Zd"efd%d&ZZS))LightweightSincConvsuLightweight Sinc Convolutions. Instead of using precomputed features, end-to-end speech recognition can also be done directly from raw audio using sinc convolutions, as described in "Lightweight End-to-End Speech Recognition from Raw Audio Data Using Sinc-Convolutions" by Kürzinger et al. https://arxiv.org/abs/2010.07597 To use Sinc convolutions in your model instead of the default f-bank frontend, set this module as your pre-encoder with `preencoder: sinc` and use the input of the sliding window frontend with `frontend: sliding_window` in your yaml configuration file. So that the process flow is: Frontend (SlidingWindow) -> SpecAug -> Normalization -> Pre-encoder (LightweightSincConvs) -> Encoder -> Decoder Note that this method also performs data augmentation in time domain (vs. in spectral domain in the default frontend). Use `plot_sinc_filters.py` to visualize the learned Sinc filters. > leakyreludropouthammingmelfs in_channels out_channelsactivation_type dropout_typewindowing_type scale_typecstsJtt|trt|}||_||_||_ ||_ ||_ ||_ ||_ tjjttjjd|_||jvrEtdt|jtjjtjjd|_||jvr`tdt|j||dS)aInitialize the module. Args: fs: Sample rate. in_channels: Number of input channels. out_channels: Number of output channels (for each input channel). activation_type: Choice of activation function. dropout_type: Choice of dropout function. windowing_type: Choice of windowing function. scale_type: Choice of filter-bank initialization scale. )rspatial dropout2dzDropout type has to be one of )rreluz!Activation type has to be one of N)rsuper__init__ isinstancestr humanfriendly parse_sizerrrrrrrtorchnnDropoutSpatialDropout Dropout2dchoices_dropoutNotImplementedErrorlistkeys LeakyReLUReLUchoices_activation_create_sinc_convsespnet_initialization_fn)selfrrrrrrr __class__O/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/asr/preencoder/sinc.pyr)s@         zLightweightSincConvs.__init__c Cst}d}t|j|dd|j|j|jd|_td|jfdtfdtj j |dd fd tj d fg}tj ||d <|}d}|j ||d d dddd|d<|}|j}dD]}|j ||ddd|d|<|}qR|j}|j ||dddd|d<tj ||_dS)Ner ) kernel_sizestrider window_funcrFiltersr BatchNormTaffineAvgPool SincConvBlockrg?)depthwise_kernel_sizedepthwise_stridepointwise_groupsavgpooldropout_probability DConvBlock1)r? )rBrC DConvBlock)rBrCrD DConvBlock5)rr rrrrfiltersrr"r# BatchNorm1d AvgPool1d Sequential gen_lsc_blockrblocks)r0rSrblockrlayerr3r3r4r.ds\    z'LightweightSincConvs._create_sinc_convsrJNr333333?FrBrCrFc Cst} |s||}} | dkr||| }} | dkstjj|||||d| d<|r5tjj||dd|d| d<|j|j| d<tjj|dd| d <|rRtjd | d <|j|j || d <tj | S) aGenerate a convolutional block for Lightweight Sinc convolutions. Each block consists of either a depthwise or a depthwise-separable convolutions together with dropout, (batch-)normalization layer, and an optional average-pooling layer. Args: in_channels: Number of input channels. out_channels: Number of output channels. depthwise_kernel_size: Kernel size of the depthwise convolution. depthwise_stride: Stride of the depthwise convolution. depthwise_groups: Number of groups of the depthwise convolution. pointwise_groups: Number of groups of the pointwise convolution. dropout_probability: Dropout probability in the block. avgpool: If True, an AvgPool layer is inserted. Returns: torch.nn.Sequential: Neural network building block. r)groups depthwiser pointwise activationTr< batchnormr?rEr) rr"r#Conv1dr-rrOrPr'rrQ) r0rrrBrCdepthwise_groupsrDrFrErTrr3r3r4rRs.     z"LightweightSincConvs.gen_lsc_blockcCsZ|j|jD]"}|D]}t|tjjkr)|jr)d|jj dd<d|j j dd<q qdS)z/Initialize sinc filters with filterbank values.g?Ng) rN init_filtersrStyper"r#rOr=weightdatabias)r0rTrUr3r3r4r/s  z-LightweightSincConvs.espnet_initialization_fninput input_lengthsreturnc CsV|\}}}}|||||}|j|}|\} } } |||| | }||fS)aApply Lightweight Sinc Convolutions. The input shall be formatted as (B, T, C_in, D_in) with B as batch size, T as time dimension, C_in as channels, and D_in as feature dimension. The output will then be (B, T, C_out*D_out) with C_out and D_out as output dimensions. The current module structure only handles D_in=400, so that D_out=1. Remark for the multichannel case: C_out is the number of out_channels given at initialization multiplied with C_in. )sizeviewrSforward) r0rdreBTC_inD_in input_frames output_frames_C_outD_outr3r3r4ris  zLightweightSincConvs.forwardcCs |j|jS)zGet the output size.)rr)r0r3r3r4 output_sizes z LightweightSincConvs.output_size)r r r rrrr)rJr NrrVF)__name__ __module__ __qualname____doc__rintrfloatrr.rRr/r"Tensorrrirs __classcell__r3r3r1r4r sf ;> 6  r csReZdZdZ  d dedeeeefffdd Z de j d e j fd d Z Z S) r%z^Spatial dropout module. Apply dropout to full channels on tensors of input (B, C, D) rVNrFshapecs:tsJt|durd}tj||_|f|_dS)zInitialize. Args: dropout_probability: Dropout probability. shape (tuple, list): Shape of input tensors. N)rr?r )rrrr"r#r&rr|)r0rFr|r1r3r4rs  zSpatialDropout.__init__xrfcCs"|j|j}||}|j|jS)z"Forward of spatial dropout module.)permuter|r)r0r}yr3r3r4ris   zSpatialDropout.forward)rVN)rtrurvrwryrrtupler)rr"rzrir{r3r3r1r4r%sr%)rw collectionsrtypingrrrr r" typeguardr%espnet2.asr.preencoder.abs_preencoderrespnet2.layers.sinc_convrr r r#Moduler%r3r3r3r4s   l