o %ݫiG0@sdZddlZddlmZddlmmZddZddZGdddej Z Gd d d ej Z Gd d d ej Z Gd ddej Z dS)z This file implements the CNN14 model from https://arxiv.org/abs/1912.10211 Authors * Cem Subakan 2022 * Francesco Paissan 2022 NcCs<tj|jt|dr|jdur|jjddSdSdS)z+Initialize a Linear or Convolutional layer.biasN)nninitxavier_uniform_weighthasattrrdatafill_)layerr R/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/Cnn14.py init_layer s   rcCs |jjd|jjddS)zInitialize a Batchnorm layer.rg?N)rr r r)bnr r r init_bnsrcs2eZdZdZfddZddZd dd ZZS) ConvBlockaThis class implements the convolutional block used in CNN14 Arguments --------- in_channels : int Number of input channels out_channels : int Number of output channels norm_type : str in ['bn', 'in', 'ln'] The type of normalization Example ------- >>> convblock = ConvBlock(10, 20, 'ln') >>> x = torch.rand(5, 10, 20, 30) >>> y = convblock(x) >>> print(y.shape) torch.Size([5, 20, 10, 15]) cstt|tj||ddddd|_tj||ddddd|_||_|dkr3t||_ t||_ n1|dkrJtj |ddd|_ tj |ddd|_ n|d kr]t d ||_ t d ||_ nt d ||dS) NrrF) in_channels out_channels kernel_sizestridepaddingrrinTaffinetrack_running_statslnrUnknown norm type {})superr__init__rConv2dconv1conv2 norm_type BatchNorm2dnorm1norm2InstanceNorm2d GroupNorm ValueErrorformat init_weight)selfrrr& __class__r r r"1sB   zConvBlock.__init__cCs,t|jt|jt|jt|jdS)zU Initializes the model convolutional layers and the batchnorm layers N)rr$r%rr(r)r/r r r r.Ws   zConvBlock.init_weightr4avgcCst|||}t|||}|dkr#tj||d}|S|dkr0tj||d}|S|dkrHtj||d}tj||d}||}|Std)akThe forward pass for convblocks in CNN14 Arguments --------- x : torch.Tensor input tensor with shape B x C_in x D1 x D2 where B = Batchsize C_in = Number of input channel D1 = Dimensionality of the first spatial dim D2 = Dimensionality of the second spatial dim pool_size : tuple with integer values Amount of pooling at each layer pool_type : str in ['max', 'avg', 'avg+max'] The type of pooling Returns ------- The output of one conv block max)rr5zavg+maxzIncorrect pooling type!) Frelu_r(r$r)r% max_pool2d avg_pool2d Exception)r/x pool_size pool_typex1x2r r r forward`s zConvBlock.forward)r3r5__name__ __module__ __qualname____doc__r"r.rA __classcell__r r r0r rs  & rcs4eZdZdZ d fdd ZddZdd ZZS) Cnn14aThis class implements the Cnn14 model from https://arxiv.org/abs/1912.10211 Arguments --------- mel_bins : int Number of mel frequency bins in the input emb_dim : int The dimensionality of the output embeddings norm_type: str in ['bn', 'in', 'ln'] The type of normalization return_reps: bool (default=False) If True the model returns intermediate representations as well for interpretation l2i : bool If True, remove one of the outputs. Example ------- >>> cnn14 = Cnn14(120, 256) >>> x = torch.rand(3, 400, 120) >>> h = cnn14.forward(x) >>> print(h.shape) torch.Size([3, 1, 256]) rFcstt|||_||_||_|dkrt||_n!|dkr)tj |ddd|_n|dkr5t d||_nt d |t dd|d |_t dd |d |_t d d |d |_t d d |d |_t d d |d |_t d ||d |_|dS)NrrTrrrr @)rrr&ii)r!rHr" return_repsl2ir&rr'norm0r*r+r,r-r conv_block1 conv_block2 conv_block3 conv_block4 conv_block5 conv_block6r.)r/mel_binsemb_dimr&rLrMr0r r r"s@  zCnn14.__init__cCst|jdS)z8 Initializes the model batch norm layer N)rrNr2r r r r.szCnn14.init_weightc Csz|dkr |d}|dd}||}|dd}|j|ddd}tj|d|jd}|j|ddd}tj|d|jd}|j |ddd}tj|d|jd}|j |ddd}tj|d|jd}|j |ddd}tj|d|jd}|j |ddd}tj|d|jd}t j|dd }t j|d d \}}t j|d d }||}|js|dS|jr|d|||ffS|d||||ffS) a The forward pass for the CNN14 encoder Arguments --------- x : torch.Tensor input tensor with shape B x C_in x D1 x D2 where B = Batchsize C_in = Number of input channel D1 = Dimensionality of the first spatial dim D2 = Dimensionality of the second spatial dim Returns ------- Outputs of CNN14 encoder rrr3r5)r=r>g?)ptrainingr)dimr4)rY unsqueeze transposerNrOr7dropoutrXrPrQrRrSrTtorchmeanr6rLrM) r/r<x4_outx3_outx2_outx1_outr?_r@r r r rAs4      z Cnn14.forward)rFFrBr r r0r rHs 'rHcs.eZdZdZ dfdd Zd ddZZS) CNN14PSIa This class estimates a mel-domain saliency mask Arguments --------- dim : int Dimensionality of the embeddings Example ------- >>> from speechbrain.lobes.models.Cnn14 import Cnn14 >>> classifier_embedder = Cnn14(mel_bins=80, emb_dim=2048, return_reps=True) >>> x = torch.randn(2, 201, 80) >>> _, hs = classifier_embedder(x) >>> psimodel = CNN14PSI(2048) >>> xhat = psimodel.forward(hs) >>> print(xhat.shape) torch.Size([2, 1, 201, 80]) rJcstt||ddd|_t|d|ddd|_t||ddd|_t|d|ddd|_t||d dd|_t|d |d dd|_ t||d dd |_ t|dd dd |_ t d|_ dS)Nrr3rr4r4rgrgrgrrgrrrrgT)r!r"rConvTranspose2dconvt1convt2convt3convt4convt5convt6convt7convt8ReLUnonl)r/rYr0r r r"s zCNN14PSI.__init__Nc Cs||d}||}||d}||}||}||}||}||d}||}||}||}||}||d} || } || }||}||}||} | S)aa Forward step. Given the classifier representations estimates a saliency map. Arguments --------- hs : torch.Tensor Classifier's representations. labels : None Unused Returns ------- xhat : torch.Tensor Estimated saliency map (before sigmoid) rrr4r) rorxrprqrrrsrtrurv) r/hslabelsh1h2hh3h4h5h6xhatr r r rA$s&           zCNN14PSI.forward)rJ)NrCrDrErFr"rArGr r r0r rds rdcs*eZdZdZdfdd ZddZZS) CNN14PSI_stfta This class estimates a saliency map on the STFT domain, given classifier representations. Arguments --------- dim : int Dimensionality of the input representations. outdim : int Defines the number of output channels in the saliency map. Example ------- >>> from speechbrain.lobes.models.Cnn14 import Cnn14 >>> classifier_embedder = Cnn14(mel_bins=80, emb_dim=2048, return_reps=True) >>> x = torch.randn(2, 201, 80) >>> _, hs = classifier_embedder(x) >>> psimodel = CNN14PSI_stft(2048, 1) >>> xhat = psimodel.forward(hs) >>> print(xhat.shape) torch.Size([2, 1, 201, 513]) rJrcstt||ddd|_t|d|ddd|_t||ddd|_t|d|ddd|_t||ddd d|_t|d |dd dd|_ t|d|dd d d |_ t|d|d dd d|_ t|d |ddd|_ t d|_dS)Nrrhrr4rergri)rrjr3rkrrl)rrjrm)rr4)rrj)rrgrT)r!r"rrnrorprqrrrsrtrurvconvt9rwrx)r/rYoutdimr0r r r"hs zCNN14PSI_stft.__init__c Cs||d}||}||d}||}||}||}||}||d}||}||}||}||}||d}||}||}||}||}||}| |} | S)a Forward step to estimate the saliency map Arguments -------- hs : torch.Tensor Classifier's representations. Returns -------- xhat : torch.Tensor An Estimate for the saliency map rrr4r) rorxrprqrrrsrtrurvr) r/ryr{r|r}r~rrrrr r r rA{s(            zCNN14PSI_stft.forward)rJrrr r r0r rQsr)rFr]torch.nnrtorch.nn.functional functionalr7rrModulerrHrdrr r r r s  hzS