o ½e¦i9Aã@sÀdZddlZddlmZddlmmZddlmZGdd„dej ƒZ Gdd„dej ƒZ dd „Z Gd d „d ej ƒZ Gd d „d ej ƒZGdd„dej ƒZGdd„dej ƒZGdd„dej ƒZdS)zÖThis file implements the necessary classes and functions to implement Listen-to-Interpret (L2I) interpretation method from https://arxiv.org/abs/2202.11479v2 Authors * Cem Subakan 2022 * Francesco Paissan 2022 éN)Ú ResBlockAudiocs4eZdZdZddgd¢f‡fdd„ Zdd„Z‡ZS) ÚPsia7Convolutional Layers to estimate NMF Activations from Classifier Representations Arguments --------- n_comp : int Number of NMF components (or equivalently number of neurons at the output per timestep) T : int The targeted length along the time dimension in_emb_dims : List with int elements A list with length 3 that contains the dimensionality of the input dimensions The list needs to match the number of channels in the input classifier representations The last entry should be the smallest entry Example ------- >>> inp = [torch.ones(2, 150, 6, 2), torch.ones(2, 100, 6, 2), torch.ones(2, 50, 12, 5)] >>> psi = Psi(n_comp=100, T=120, in_emb_dims=[150, 100, 50]) >>> h = psi(inp) >>> print(h.shape) torch.Size([2, 100, 120]) édé¯)iiics¶tƒ ¡||_tjdd|_tj|dfd|_t|ƒ}tj|d|ddd|_ tj|d|ddd|_ tj||ddd|_ t  tj|d|dddt  |¡t ¡¡|_t ¡|_dS) N©ér)Ú scale_factoré)ÚsizeréÚsame©Ú kernel_sizeÚpadding)ÚsuperÚ__init__Ú in_emb_dimsÚnnÚUpsamplingBilinear2dÚupsampÚ upsamp_timeÚminÚConv2dÚc1Úc2Úout_convÚ SequentialÚ BatchNorm2dÚReLUÚconvÚact)ÚselfÚn_compÚTrÚout_c©Ú __class__©úZ/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/lobes/models/L2I.pyr&s$  ÿ ÿýz Psi.__init__c Csxd}t|jƒD]\}}||jd|j|ksJd|ƒ‚q|djd|djdks2Jd|ƒ‚|djd|djdksFJd|ƒ‚d|djd|djddkspJd|d|djd›d |djd›ƒ‚|\}}}| |¡}| |¡}| | |¡¡}| | |¡¡}t |d ¡}t |d ¡}t j |||fdd }|  |¡}|  |¡}| |  |¡¡ d¡}|S) aThis forward function returns the NMF time activations given classifier activations Arguments --------- inp: list A length 3 list of classifier input representations. Returns ------- NMF time activations zcin PSI doesn't match. The embedding dimensions need to be consistent with the list self.in_emb_dimsr zNr. of channels rrzSpatial dimension r z 1st (idx 0) element has shape z" second element (idx 1) has shape )rr rr)Úaxis)Ú enumeraterÚshaperr rrÚFÚpadÚtorchÚcatrrrÚsqueeze) r!ÚinpÚerrorÚiÚ in_emb_dimÚx1Úx2Úx3Úxr'r'r(Úforward>s4 ÿ((&ÿ"þÿ       z Psi.forward©Ú__name__Ú __module__Ú __qualname__Ú__doc__rr9Ú __classcell__r'r'r%r(rsrcs2eZdZdZd ‡fdd„ Zdd„Zd d „Z‡ZS) ÚNMFDecoderAudioaÈThis class implements an NMF decoder Arguments --------- n_comp : int Number of NMF components n_freq : int The number of frequency bins in the NMF dictionary device : str The device to run the model Example ------- >>> NMF_dec = NMFDecoderAudio(20, 210, device='cpu') >>> H = torch.rand(1, 20, 150) >>> Xhat = NMF_dec.forward(H) >>> print(Xhat.shape) torch.Size([1, 210, 150]) réÚcudacs4tƒ ¡tjdt ||¡dd|_t ¡|_dS)Ngš™™™™™¹?T)Ú requires_grad) rrrÚ Parameterr.ÚrandÚWrÚactiv)r!r"Ún_freqÚdevicer%r'r(r†s ÿzNMFDecoderAudio.__init__cCs.| |¡}| |j¡ d¡}t d||¡}|S)aŽThe forward pass for NMF given the activations H Arguments --------- H : torch.Tensor The activations Tensor with shape B x n_comp x T where B = Batchsize n_comp = number of NMF components T = number of timepoints Returns ------- output : torch.Tensor The NMF outputs rzbij, bjk -> bik)rGrFÚ unsqueezer.Úeinsum)r!ÚHÚtempÚoutputr'r'r(r9Žs zNMFDecoderAudio.forwardcCs|j}| |¡S)z(This function returns the NMF dictionary)rFrG)r!rFr'r'r(Úreturn_W¦s zNMFDecoderAudio.return_W)rrArB)r;r<r=r>rr9rOr?r'r'r%r(r@qs r@cCs^|jj}| d¡dkr-ztj |jj¡|jj  d¡WdSt y,t d|ƒYdSwdS)zˆ Applies Xavier initialization to network weights. Arguments --------- m : nn.Module Module to initialize. ÚConvéÿÿÿÿrzSkipping initialization of N) r&r;ÚfindrÚinitÚxavier_uniform_ÚweightÚdataÚbiasÚfill_ÚAttributeErrorÚprint)ÚmÚ classnamer'r'r(Ú weights_init¬s  ÿür]cs4eZdZdZ     d ‡fdd„ Zd d „Z‡ZS) Ú PsiOptimizedasConvolutional Layers to estimate NMF Activations from Classifier Representations, optimized for log-spectra. Arguments --------- dim : int Dimension of the hidden representations (input to the classifier). K : int Number of NMF components (or equivalently number of neurons at the output per timestep) numclasses : int Number of possible classes. use_adapter : bool `True` if you wish to learn an adapter for the latent representations. adapter_reduce_dim: bool `True` if the adapter should compress the latent representations. Example ------- >>> inp = torch.randn(1, 256, 26, 32) >>> psi = PsiOptimized(dim=256, K=100, use_adapter=False, adapter_reduce_dim=False) >>> h, inp_ad= psi(inp) >>> print(h.shape, inp_ad.shape) torch.Size([1, 1, 417, 100]) torch.Size([1, 256, 26, 32]) é€ré2FTcstƒ ¡||_||_|r(t|ƒ|_|r(t ||ddd¡|_t  ||ddd¡|_ t  t  ||ddd¡t  d¡t  |¡t  ||ddd¡t  ¡t  |¡t  ||ddd¡t  ¡t  |¡t  ||ddd¡t  ¡t  |¡t  |dddd¡t  ¡t d|¡t  ¡¡|_| t¡dS)Nérr r Té rA)rrÚ use_adapterÚadapter_reduce_dimrÚadapterrrÚdownÚConvTranspose2dÚuprrrÚLinearÚdecoderÚapplyr])r!ÚdimÚKÚ numclassesrcrdr%r'r(r×s6   ðzPsiOptimized.__init__cCsT|jr | |¡}n|}|jr!| |¡}| |¡}| |¡}||fS| |¡}||fS)aY Computes forward step. Arguments --------- hs : torch.Tensor Latent representations (input to the classifier). Expected shape `torch.Size([B, C, H, W])`. Returns ------- NMF activations and adapted representations. Shape `torch.Size([B, 1, T, 100])`. : torch.Tensor )rcrerdrfrhrj)r!ÚhsÚhcatÚz_q_x_stÚoutr'r'r(r9þs     þzPsiOptimized.forward)r_rr`FTr:r'r'r%r(r^¾sú'r^cs*eZdZdZd ‡fdd„ Zdd„Z‡ZS) ÚThetaaÙThis class implements a linear classifier on top of NMF activations Arguments --------- n_comp : int Number of NMF components T : int Number of Timepoints in the NMF activations num_classes : int Number of classes that the classifier works with Example ------- >>> theta = Theta(30, 120, 50) >>> H = torch.rand(1, 30, 120) >>> c_hat = theta.forward(H) >>> print(c_hat.shape) torch.Size([1, 50]) rrr`csBtƒ ¡tj|ddd|_t tj||ddtjdd¡|_dS)Nr F)rW)rl)rrrriÚhard_attrÚSoftmaxÚ classifier)r!r"r#Ú num_classesr%r'r(r/s  ÿzTheta.__init__cCs| |¡ d¡}| |¡}|S)aªWe first collapse the time axis, and then pass through the linear layer Arguments --------- H : torch.Tensor The activations Tensor with shape B x n_comp x T where B = Batchsize n_comp = number of NMF components T = number of timepoints Returns ------- theta_out : torch.Tensor Classifier output r)rtr0rv)r!rLÚ theta_outr'r'r(r9:s z Theta.forward)rrr`r:r'r'r%r(rss rscs(eZdZdZ‡fdd„Zdd„Z‡ZS)Ú NMFEncoderaThis class implements an NMF encoder with a convolutional network Arguments --------- n_freq : int The number of frequency bins in the NMF dictionary n_comp : int Number of NMF components Example ------- >>> nmfencoder = NMFEncoder(513, 100) >>> X = torch.rand(1, 513, 240) >>> Hhat = nmfencoder(X) >>> print(Hhat.shape) torch.Size([1, 100, 240]) c sZtƒ ¡t tj|ddddt ¡tjdddddt ¡tjd|dddt ¡¡|_dS)Néér r r_)rrrrÚConv1drÚconvenc)r!rHr"r%r'r(rbs  úzNMFEncoder.__init__cCs | |¡S)aL Arguments --------- X : torch.Tensor The input spectrogram Tensor with shape B x n_freq x T where B = Batchsize n_freq = nfft for the input spectrogram T = number of timepoints Returns ------- NMF encoded outputs. )r})r!ÚXr'r'r(r9ms zNMFEncoder.forwardr:r'r'r%r(ryOs  rycó,eZdZdZd ‡fdd„ Zd dd„Z‡ZS) Ú CNN14PSI_stfta† This class estimates a saliency map on the STFT domain, given classifier representations. Arguments --------- dim : int Dimensionality of the input representations. K : int Defines the number of output channels in the saliency map. Example ------- >>> from speechbrain.lobes.models.Cnn14 import Cnn14 >>> classifier_embedder = Cnn14(mel_bins=80, emb_dim=2048, return_reps=True) >>> x = torch.randn(2, 201, 80) >>> _, hs = classifier_embedder(x) >>> psimodel = CNN14PSI_stft(2048, 20) >>> xhat = psimodel.forward(hs) >>> print(xhat.shape) torch.Size([2, 20, 207]) r_rcsötƒ ¡t ||ddd¡|_t |d|ddd¡|_t ||ddd¡|_t |d|ddd¡|_t ||dddd¡|_t |d|dddd¡|_ t |d|dddd¡|_ t |d|dddd¡|_ t |d|ddd¡|_ t  d ¡|_dS) Nr rr éraér{rT)rrrÚConvTranspose1dÚconvt1Úconvt2Úconvt3Úconvt4Úconvt5Úconvt6Úconvt7Úconvt8Úconvt9rÚnonl©r!rlrmr%r'r(r•s zCNN14PSI_stft.__init__Nc Csødd„|Dƒ}| |d¡}| |¡}| |d¡}| |¡}||}| |¡}| |¡}| |d¡}| |¡}||}| |¡}| |¡}| |d¡} | | ¡} || }| |¡}| |¡}| |¡}| |¡}|  |¡} | | ¡} t   | ¡} | S)á— Forward step. Estimates NMF activations to be used to get the saliency mask. Arguments -------- hs : torch.Tensor Classifier's representations. labels : torch.Tensor Predicted labels for classifier's representations. Returns -------- xhat : torch.Tensor The estimated NMF activation coefficients cSsg|]}| d¡‘qS)rQ)Úmean)Ú.0Úhr'r'r(Ú µsz)CNN14PSI_stft.forward..rr rr ) r„rr…r†r‡rˆr‰rŠr‹rŒr,Úrelu© r!roÚlabelsÚh1Úh2r’Úh3Úh4Úh5Úh6Úxhatr'r'r(r9¤s0               zCNN14PSI_stft.forward©r_r©Nr:r'r'r%r(r€~sr€cr) ÚCNN14PSI_stft_2da‰ This class estimates the NMF activations to create a saliency map using the L2I framework Arguments --------- dim : int Dimensionality of the input representations. K : int Defines the number of output channels in the saliency map. Example ------- >>> from speechbrain.lobes.models.Cnn14 import Cnn14 >>> classifier_embedder = Cnn14(mel_bins=80, emb_dim=2048, return_reps=True) >>> x = torch.randn(2, 201, 80) >>> _, hs = classifier_embedder(x) >>> psimodel = CNN14PSI_stft_2d(2048, 20) >>> xhat = psimodel.forward(hs) >>> print(xhat.shape) torch.Size([2, 20, 207]) r_rcsötƒ ¡t ||ddd¡|_t |d|ddd¡|_t ||ddd¡|_t |d|ddd¡|_t ||ddd d¡|_t |d |dd dd¡|_ t |d|dd d d ¡|_ t |d|d dd d¡|_ t |d |ddd¡|_ t  d¡|_dS)Nr )rrar r)rrara)r‚ra)r r‚rr{)r r )rar )rr‚)r ra)rr)rr‚)r rarT)rrrrgr„r…r†r‡rˆr‰rŠr‹rŒrrrŽr%r'r(rñs ÿÿzCNN14PSI_stft_2d.__init__Nc Csô| |d¡}| |¡}| |d¡}| |¡}||}| |¡}| |¡}| |d¡}| |¡}||}| |¡}| |¡}| |d¡} | | ¡} || }| |¡}| |¡}| |¡}| |¡}|  |¡} | | ¡} |   d¡} t   | ¡} | S)rrr rr rQ) r„rr…r†r‡rˆr‰rŠr‹rŒrr,r”r•r'r'r(r9s0                zCNN14PSI_stft_2d.forwardržrŸr:r'r'r%r(r Úsr )r>r.Útorch.nnrÚtorch.nn.functionalÚ functionalr,Úspeechbrain.lobes.models.PIQrÚModulerr@r]r^rsryr€r r'r'r'r(Ús  b;\5/\