o ½e¦iG0ã@s†dZddlZddlmZddlmmZdd„Zdd„ZGdd„dej ƒZ Gd d „d ej ƒZ Gd d „d ej ƒZ Gd d„dej ƒZ dS)z„ This file implements the CNN14 model from https://arxiv.org/abs/1912.10211 Authors * Cem Subakan 2022 * Francesco Paissan 2022 éNcCs<tj |j¡t|dƒr|jdur|jj d¡dSdSdS)z+Initialize a Linear or Convolutional layer.ÚbiasNç)ÚnnÚinitÚxavier_uniform_ÚweightÚhasattrrÚdataÚfill_)Úlayer©r ú\/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/lobes/models/Cnn14.pyÚ init_layer s   þrcCs |jj d¡|jj d¡dS)zInitialize a Batchnorm layer.rgð?N)rr r r)Úbnr r r Úinit_bnsrcs2eZdZdZ‡fdd„Zdd„Zd dd „Z‡ZS) Ú ConvBlockaÔThis class implements the convolutional block used in CNN14 Arguments --------- in_channels : int Number of input channels out_channels : int Number of output channels norm_type : str in ['bn', 'in', 'ln'] The type of normalization Example ------- >>> convblock = ConvBlock(10, 20, 'ln') >>> x = torch.rand(5, 10, 20, 30) >>> y = convblock(x) >>> print(y.shape) torch.Size([5, 20, 10, 15]) csÔtt|ƒ ¡tj||ddddd|_tj||ddddd|_||_|dkr3t |¡|_ t |¡|_ n1|dkrJtj |ddd|_ tj |ddd|_ n|d kr]t  d |¡|_ t  d |¡|_ nt d  |¡ƒ‚| ¡dS) N©ér©érF)Ú in_channelsÚ out_channelsÚ kernel_sizeÚstrideÚpaddingrrÚinT©ÚaffineÚtrack_running_statsÚlnrúUnknown norm type {})ÚsuperrÚ__init__rÚConv2dÚconv1Úconv2Ú norm_typeÚ BatchNorm2dÚnorm1Únorm2ÚInstanceNorm2dÚ GroupNormÚ ValueErrorÚformatÚ init_weight)Úselfrrr&©Ú __class__r r r"1sBúú ÿ ÿ zConvBlock.__init__cCs,t|jƒt|jƒt|jƒt|jƒdS)zU Initializes the model convolutional layers and the batchnorm layers N)rr$r%rr(r)©r/r r r r.Ws   zConvBlock.init_weight©ér4ÚavgcCs˜t | | |¡¡¡}t | | |¡¡¡}|dkr#tj||d}|S|dkr0tj||d}|S|dkrHtj||d}tj||d}||}|Stdƒ‚)akThe forward pass for convblocks in CNN14 Arguments --------- x : torch.Tensor input tensor with shape B x C_in x D1 x D2 where B = Batchsize C_in = Number of input channel D1 = Dimensionality of the first spatial dim D2 = Dimensionality of the second spatial dim pool_size : tuple with integer values Amount of pooling at each layer pool_type : str in ['max', 'avg', 'avg+max'] The type of pooling Returns ------- The output of one conv block Úmax)rr5zavg+maxzIncorrect pooling type!) ÚFÚrelu_r(r$r)r%Ú max_pool2dÚ avg_pool2dÚ Exception)r/ÚxÚ pool_sizeÚ pool_typeÚx1Úx2r r r Úforward`s øúÿzConvBlock.forward)r3r5©Ú__name__Ú __module__Ú __qualname__Ú__doc__r"r.rAÚ __classcell__r r r0r rs  & rcs4eZdZdZ d ‡fdd„ Zdd„Zdd „Z‡ZS) ÚCnn14a«This class implements the Cnn14 model from https://arxiv.org/abs/1912.10211 Arguments --------- mel_bins : int Number of mel frequency bins in the input emb_dim : int The dimensionality of the output embeddings norm_type: str in ['bn', 'in', 'ln'] The type of normalization return_reps: bool (default=False) If True the model returns intermediate representations as well for interpretation l2i : bool If True, remove one of the outputs. Example ------- >>> cnn14 = Cnn14(120, 256) >>> x = torch.rand(3, 400, 120) >>> h = cnn14.forward(x) >>> print(h.shape) torch.Size([3, 1, 256]) rFcsätt|ƒ ¡||_||_||_|dkrt |¡|_n!|dkr)tj |ddd|_n|dkr5t  d|¡|_nt d  |¡ƒ‚t dd|d |_t dd |d |_t d d |d |_t d d |d |_t d d |d |_t d ||d |_| ¡dS)NrrTrrrr é@)rrr&é€éii)r!rHr"Ú return_repsÚl2ir&rr'Únorm0r*r+r,r-rÚ conv_block1Ú conv_block2Ú conv_block3Ú conv_block4Ú conv_block5Ú conv_block6r.)r/Úmel_binsÚemb_dimr&rLrMr0r r r"s@ ÿÿÿÿÿÿÿ zCnn14.__init__cCst|jƒdS)z8 Initializes the model batch norm layer N)rrNr2r r r r.ÄszCnn14.init_weightc Csz| ¡dkr | d¡}| dd¡}| |¡}| dd¡}|j|ddd}tj|d|jd}|j|ddd}tj|d|jd}|j |ddd}tj|d|jd}|j |ddd}tj|d|jd}|j |ddd}tj|d|jd}|j |ddd}tj|d|jd}t j|dd }t j|d d \}}t j|d d }||}|js¥| d¡S|jr²| d¡|||ffS| d¡||||ffS) a¿ The forward pass for the CNN14 encoder Arguments --------- x : torch.Tensor input tensor with shape B x C_in x D1 x D2 where B = Batchsize C_in = Number of input channel D1 = Dimensionality of the first spatial dim D2 = Dimensionality of the second spatial dim Returns ------- Outputs of CNN14 encoder rrr3r5)r=r>gš™™™™™É?)ÚpÚtrainingr)Údimr4)rYÚ unsqueezeÚ transposerNrOr7ÚdropoutrXrPrQrRrSrTÚtorchÚmeanr6rLrM) r/r<Úx4_outÚx3_outÚx2_outÚx1_outr?Ú_r@r r r rAÊs4      z Cnn14.forward)rFFrBr r r0r rH„s ÿ'rHcs.eZdZdZ d‡fdd„ Zd dd„Z‡ZS) ÚCNN14PSIaû This class estimates a mel-domain saliency mask Arguments --------- dim : int Dimensionality of the embeddings Example ------- >>> from speechbrain.lobes.models.Cnn14 import Cnn14 >>> classifier_embedder = Cnn14(mel_bins=80, emb_dim=2048, return_reps=True) >>> x = torch.randn(2, 201, 80) >>> _, hs = classifier_embedder(x) >>> psimodel = CNN14PSI(2048) >>> xhat = psimodel.forward(hs) >>> print(xhat.shape) torch.Size([2, 1, 201, 80]) rJcsÆtƒ ¡t ||ddd¡|_t |d|ddd¡|_t ||ddd¡|_t |d|ddd¡|_t ||d dd¡|_t |d |d dd¡|_ t ||d dd ¡|_ t |dd dd ¡|_ t  d¡|_ dS)Nrr3rr4©éé©r4rgrg©érgré©rgrr©rrgT)r!r"rÚConvTranspose2dÚconvt1Úconvt2Úconvt3Úconvt4Úconvt5Úconvt6Úconvt7Úconvt8ÚReLUÚnonl)r/rYr0r r r"s zCNN14PSI.__init__Nc CsÂ| |d¡}| |¡}| |d¡}| |¡}||}| |¡}| |¡}| |d¡}| |¡}||}| |¡}| |¡}| |d¡} | | ¡} || }| |¡}| |¡}| |¡} | S)aa Forward step. Given the classifier representations estimates a saliency map. Arguments --------- hs : torch.Tensor Classifier's representations. labels : None Unused Returns ------- xhat : torch.Tensor Estimated saliency map (before sigmoid) rrr4r) rorxrprqrrrsrtrurv) r/ÚhsÚlabelsÚh1Úh2ÚhÚh3Úh4Úh5Úh6Úxhatr r r rA$s&           zCNN14PSI.forward)rJ)N©rCrDrErFr"rArGr r r0r rdþs þrdcs*eZdZdZd‡fdd„ Zdd„Z‡ZS) Ú CNN14PSI_stftaŽ This class estimates a saliency map on the STFT domain, given classifier representations. Arguments --------- dim : int Dimensionality of the input representations. outdim : int Defines the number of output channels in the saliency map. Example ------- >>> from speechbrain.lobes.models.Cnn14 import Cnn14 >>> classifier_embedder = Cnn14(mel_bins=80, emb_dim=2048, return_reps=True) >>> x = torch.randn(2, 201, 80) >>> _, hs = classifier_embedder(x) >>> psimodel = CNN14PSI_stft(2048, 1) >>> xhat = psimodel.forward(hs) >>> print(xhat.shape) torch.Size([2, 1, 201, 513]) rJrcsötƒ ¡t ||ddd¡|_t |d|ddd¡|_t ||ddd¡|_t |d|ddd¡|_t ||ddd d¡|_t |d |dd dd¡|_ t |d|dd d d ¡|_ t |d|d dd d¡|_ t |d |ddd¡|_ t  d¡|_dS)Nrrhrr4rergri)rrjr3rkrrl)rrjrm)rr4)rrj)rrgrT)r!r"rrnrorprqrrrsrtrurvÚconvt9rwrx)r/rYÚoutdimr0r r r"hs ÿÿzCNN14PSI_stft.__init__c CsÌ| |d¡}| |¡}| |d¡}| |¡}||}| |¡}| |¡}| |d¡}| |¡}||}| |¡}| |¡}| |d¡}| |¡}||}| |¡}| |¡}| |¡}|  |¡} | S)a Forward step to estimate the saliency map Arguments -------- hs : torch.Tensor Classifier's representations. Returns -------- xhat : torch.Tensor An Estimate for the saliency map rrr4r) rorxrprqrrrsrtrurvr…) r/ryr{r|r}r~rr€rr‚r r r rA{s(            zCNN14PSI_stft.forward)rJrrƒr r r0r r„Qsr„)rFr]Útorch.nnrÚtorch.nn.functionalÚ functionalr7rrÚModulerrHrdr„r r r r Ús  hzS