o ½e¦ibVã@sÂdZddlZddlmZddlmZddd„Zdd „ZGd d „d eƒZGd d „d eƒZ Gdd„dej ƒZ Gdd„dej ƒZ Gdd„dej ƒZ Gdd„de ƒZGdd„de ƒZGdd„dej ƒZdS)z¨This file implements the necessary classes and functions to implement Posthoc Interpretations via Quantization. Authors * Cem Subakan 2023 * Francesco Paissan 2023 éN)ÚFunctionéÚTRAINcCsÂt t d|d||¡¡ |j¡}| d¡ |jdd¡}| d¡ d||¡}||k}|dkrItj|t  |jd|¡ |j¡gdddk}|Stj|t  |jd|¡ |j¡gdddk}|S)a¥This class returns binary matrix that indicates the irrelevant regions in the VQ-dictionary given the labels array Arguments --------- labels : torch.Tensor 1 dimensional tensor of size [B] K : int Number of keys in the dictionary num_classes : int Number of possible classes N_shared : int Number of shared keys stage : str "TRAIN" or else Returns ------- irrelevant_regions : torch.Tensor Example ------- >>> labels = torch.Tensor([1, 0, 2]) >>> irrelevant_regions = get_irrelevant_regions(labels, 20, 3, 5) >>> print(irrelevant_regions.shape) torch.Size([3, 20]) gà¿gR¸…ëQà?rér©Údim) ÚtorchÚroundÚlinspaceÚtoÚdeviceÚ unsqueezeÚrepeatÚshapeÚcatÚonesÚzeros)ÚlabelsÚKÚ num_classesÚN_sharedÚstageÚ uniform_matÚlabels_expandedÚirrelevant_regions©rúZ/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/lobes/models/PIQ.pyÚget_irrelevant_regions sBÿþÿþù ÷ÿõÿþù ÷ÿ rcCs^|jj}| d¡dkr-ztj |jj¡|jj  d¡WdSt y,t d|ƒYdSwdS)z; Applies Xavier initialization to network weights. ÚConvéÿÿÿÿrzSkipping initialization of N) Ú __class__Ú__name__ÚfindÚnnÚinitÚxavier_uniform_ÚweightÚdataÚbiasÚfill_ÚAttributeErrorÚprint)ÚmÚ classnamerrrÚ weights_initPs ÿür.c@s4eZdZdZe     d dd„ƒZedd„ƒZdS) ÚVectorQuantizationaThis class defines the forward method for vector quantization. As VQ is not differentiable, it returns a RuntimeError in case `.grad()` is called. Refer to `VectorQuantizationStraightThrough` for a straight_through estimation of the gradient for the VQ operation.Né TcCst ¡}| d¡}| ¡} | d|¡} | ddd¡ d| d| d¡} |  d¡} t| |jd|||r5dndd} tj|ddd}tj| ddd d }tj ||| |  ¡d d d }|ratj || <tj |dd\}}|j| dd…Ž}|  |¡|WdƒS1s„wYdS)a[ Applies VQ to vectors `input` with `codebook` as VQ dictionary. Arguments --------- ctx : torch context The context object for storing info for backwards. inputs : torch.Tensor Hidden representations to quantize. Expected shape is `torch.Size([B, W, H, C])`. codebook : torch.Tensor VQ-dictionary for quantization. Expected shape of `torch.Size([K, C])` with K dictionary elements. labels : torch.Tensor Classification labels. Used to define irrelevant regions and divide the latent space based on predicted class. Shape should be `torch.Size([B])`. num_classes : int Number of possible classes activate_class_partitioning : bool `True` if latent space should be quantized for different classes. shared_keys : int Number of shared keys among classes. training : bool `True` if stage is TRAIN. Returns ------- Codebook's indices for quantized representation : torch.Tensor Example ------- >>> inputs = torch.ones(3, 14, 25, 256) >>> codebook = torch.randn(1024, 256) >>> labels = torch.Tensor([1, 0, 2]) >>> print(VectorQuantization.apply(inputs, codebook, labels).shape) torch.Size([3, 14, 25]) rrérrÚVALID)rrrT)rÚkeepdimgÀçð?)ÚalphaÚbetaN)rÚno_gradÚsizeÚviewÚreshaperrrÚsumÚaddmmÚtÚinfÚminÚmark_non_differentiable)ÚctxÚinputsÚcodebookrrÚactivate_class_partitioningÚ shared_keysÚtrainingÚembedding_sizeÚ inputs_sizeÚinputs_flattenrÚlabels_flattenrÚ codebook_sqrÚ inputs_sqrÚ distancesÚ_Úindices_flattenÚindicesrrrÚforward`s> -  ÿ  ûû  $ÛzVectorQuantization.forwardcCstdƒ‚)z;Handles error in case grad() is called on the VQ operation.zÝTrying to call `.grad()` on graph containing `VectorQuantization`. The function `VectorQuantization` is not differentiable. Use `VectorQuantizationStraightThrough` if you want a straight-through estimator of the gradient.)Ú RuntimeError)rAÚ grad_outputrrrÚbackward´sÿzVectorQuantization.backward©Nr0Tr0T©r!Ú __module__Ú __qualname__Ú__doc__Ú staticmethodrQrTrrrrr/]s øSr/c@s@eZdZdZe     d dd„ƒZe     d dd„ƒZdS) Ú!VectorQuantizationStraightThroughzªThis class defines the forward method for vector quantization. As VQ is not differentiable, it approximates the gradient of the VQ as in https://arxiv.org/abs/1711.00937.Nr0Tc CsXt |||||||¡}| d¡} | | |¡| | ¡tj|d| d} |  |¡} | | fS)a3 Applies VQ to vectors `input` with `codebook` as VQ dictionary and estimates gradients with a Straight-Through (id) approximation of the quantization steps. Arguments --------- ctx : torch context The context object for storing info for backwards. inputs : torch.Tensor Hidden representations to quantize. Expected shape is `torch.Size([B, W, H, C])`. codebook : torch.Tensor VQ-dictionary for quantization. Expected shape of `torch.Size([K, C])` with K dictionary elements. labels : torch.Tensor Classification labels. Used to define irrelevant regions and divide the latent space based on predicted class. Shape should be `torch.Size([B])`. num_classes : int Number of possible classes activate_class_partitioning : bool `True` if latent space should be quantized for different classes. shared_keys : int Number of shared keys among classes. training : bool `True` if stage is TRAIN. Returns ------- Quantized representation and codebook's indices for quantized representation : tuple Example ------- >>> inputs = torch.ones(3, 14, 25, 256) >>> codebook = torch.randn(1024, 256) >>> labels = torch.Tensor([1, 0, 2]) >>> quant, quant_ind = VectorQuantizationStraightThrough.apply(inputs, codebook, labels) >>> print(quant.shape, quant_ind.shape) torch.Size([3, 14, 25, 256]) torch.Size([1050]) rr©rÚindex)r/Úapplyr9Úsave_for_backwardr@rÚ index_selectÚview_as) rArBrCrrrDrErFrPrOÚ codes_flattenÚcodesrrrrQÂs"/ù  ÿ z)VectorQuantizationStraightThrough.forwardcCsrd\}} |jdr | ¡}|jdr0|j\} } |  d¡} | ¡ d| ¡} t | ¡} |  d| | ¡|| dddddfS)zz Estimates gradient assuming vector quantization as identity function. (https://arxiv.org/abs/1711.00937) )NNrrrN) Úneeds_input_gradÚcloneÚ saved_tensorsr8Ú contiguousr9rÚ zeros_likeÚ index_add_)rArSÚ grad_indicesrrrDrErFÚ grad_inputsÚ grad_codebookrPrCrGÚgrad_output_flattenrrrrTs    ÿ z*VectorQuantizationStraightThrough.backwardrU)NNTr0TrVrrrrr[¿s  øBør[cs*eZdZdZd‡fdd„ Zdd„Z‡ZS)ÚConv2dEncoder_v2ay This class implements a convolutional encoder to extract classification embeddings from logspectra. Arguments --------- dim : int Number of channels of the extracted embeddings. Example ------- >>> inputs = torch.ones(3, 431, 513) >>> model = Conv2dEncoder_v2() >>> print(model(inputs).shape) torch.Size([3, 256, 26, 32]) écs¢tƒ ¡t d|ddd¡|_t |¡|_t ||ddd¡|_t |¡|_t ||ddd¡|_ t |¡|_ t ||ddd¡|_ t |¡|_ t |ƒ|_t ¡|_dS)Nrér1)ÚsuperÚ__init__r#ÚConv2dÚconv1Ú BatchNorm2dÚbn1Úconv2Úbn2Úconv3Úbn3Úconv4Úbn4Ú ResBlockAudioÚresblockÚReLUÚnonl©Úselfr©r rrrr7s      zConv2dEncoder_v2.__init__cCs| d¡}| |¡}| |¡}| |¡}| |¡}| |¡}| |¡}| |¡}| |¡}| |¡}| |¡}|  |¡}| |¡}|  |¡}|S)zô Computes forward pass. Arguments --------- x : torch.Tensor Log-power spectrogram. Expected shape `torch.Size([B, T, F])`. Returns ------- Embeddings : torch.Tensor r) r rtrvr€rwrxryrzr{r|r~)r‚ÚxÚh1Úh2Úh3Úh4rrrrQEs             zConv2dEncoder_v2.forward)ro©r!rWrXrYrrrQÚ __classcell__rrrƒrrn&srncs(eZdZdZ‡fdd„Zdd„Z‡ZS)r}aQThis class implements a residual block. Arguments --------- dim : int Input channels of the tensor to process. Matches output channels of the residual block. Example ------- >>> res = ResBlockAudio(128) >>> x = torch.randn(2, 128, 16, 16) >>> print(x.shape) torch.Size([2, 128, 16, 16]) c sLtƒ ¡t t ||ddd¡t |¡t d¡t ||d¡t |¡¡|_dS)NérT)rqrrr#Ú SequentialrsrurÚblockrrƒrrrrxs   ûzResBlockAudio.__init__cCs|| |¡S)zïForward step. Arguments --------- x : torch.Tensor Tensor to process. Expected shape is `torch.Size([B, C, H, W])`. Returns ------- Residual block output : torch.Tensor )r)r‚r„rrrrQ‚s zResBlockAudio.forwardr‰rrrƒrr}hs  r}cs8eZdZdZ       d ‡fdd„ Zd d „Z‡ZS) ÚVectorQuantizedPSI_Audioaæ This class reconstructs log-power spectrograms from classifier's representations. Arguments --------- dim : int Dimensionality of VQ vectors. K : int Number of elements of VQ dictionary. numclasses : int Number of possible classes activate_class_partitioning : bool `True` if latent space should be quantized for different classes. shared_keys : int Number of shared keys among classes. use_adapter : bool `True` to learn an adapter for classifier's representations. adapter_reduce_dim : bool `True` if adapter should compress representations. Example ------- >>> psi = VectorQuantizedPSI_Audio(dim=256, K=1024) >>> x = torch.randn(2, 256, 16, 16) >>> labels = torch.Tensor([0, 2]) >>> logspectra, hcat, z_q_x = psi(x, labels) >>> print(logspectra.shape, hcat.shape, z_q_x.shape) torch.Size([2, 1, 257, 257]) torch.Size([2, 256, 8, 8]) torch.Size([2, 256, 8, 8]) é€éé2Trcstƒ ¡t|||||d|_||_||_|r2t|ƒ|_|r2t  ||ddd¡|_ t  ||ddd¡|_ t  t  ||ddd¡t d¡t |¡t  ||ddd¡t ¡t |¡t  ||ddd¡t ¡t |¡t  ||ddd¡t ¡t |¡t  |dddd¡¡ |_| t¡dS)N)Ú numclassesrDrErp©r1r1rr‹Té )rqrrÚ VQEmbeddingrCÚ use_adapterÚadapter_reduce_dimr}Úadapterr#rsÚdownÚConvTranspose2dÚuprŒrruÚdecoderr^r.)r‚rrr’rDrEr–r—rƒrrrr°s> û óz!VectorQuantizedPSI_Audio.__init__cCsj|jr | |¡}n|}|jr"| |¡}|j ||¡\}}| |¡}n |j ||¡\}}| |¡}|||fS)aÔ Forward step. Reconstructs log-power based on provided label's keys in VQ dictionary. Arguments --------- hs : torch.Tensor Classifier's representations. labels : torch.Tensor Predicted labels for classifier's representations. Returns ------- Reconstructed log-power spectrogram, reduced classifier's representations and quantized classifier's representations. : tuple )r–r˜r—r™rCÚstraight_throughr›rœ)r‚ÚhsrÚhcatÚz_q_x_stÚz_q_xÚx_tilderrrrQÜs     z VectorQuantizedPSI_Audio.forward)rrr‘TrTTr‰rrrƒrrŽ‘s ø,rŽcó"eZdZdZd‡fdd„ Z‡ZS)Ú VectorQuantizedPSIFocalNet_Audioal This class reconstructs log-power spectrograms from a FocalNet classifier's representations. Arguments --------- dim : int Dimensionality of VQ vectors. **kwargs : dict See documentation of `VectorQuantizedPSI_Audio`. Example ------- >>> psi = VectorQuantizedPSIFocalNet_Audio(dim=256, K=1024) >>> x = torch.randn(2, 256, 16, 16) >>> labels = torch.Tensor([0, 2]) >>> logspectra, hcat, z_q_x = psi(x, labels) >>> print(logspectra.shape, hcat.shape, z_q_x.shape) torch.Size([2, 1, 495, 593]) torch.Size([2, 256, 8, 8]) torch.Size([2, 256, 8, 8]) éc ó¶tƒjd d|i|¤Žt t ||ddd¡t ¡t |¡t ||ddd¡t ¡t |¡t ||ddd¡t ¡t |¡t ||ddd¡t ¡t |¡t |dddd¡¡ |_| t ¡dS© Nrr‹)rprr)rprr“)rpr1)r0ér© rqrrr#rŒršrrurœr^r.©r‚rÚkwargsrƒrrrró"óz)VectorQuantizedPSIFocalNet_Audio.__init__)r¥©r!rWrXrYrrrŠrrrƒrr¤ûór¤cr£)ÚVectorQuantizedPSIViT_Audioab This class reconstructs log-power spectrograms from a ViT classifier's representations. Arguments --------- dim : int Dimensionality of VQ vectors. **kwargs : dict See documentation of `VectorQuantizedPSI_Audio`. Example ------- >>> psi = VectorQuantizedPSIViT_Audio(dim=256, K=1024) >>> x = torch.randn(2, 256, 16, 16) >>> labels = torch.Tensor([0, 2]) >>> logspectra, hcat, z_q_x = psi(x, labels) >>> print(logspectra.shape, hcat.shape, z_q_x.shape) torch.Size([2, 1, 495, 593]) torch.Size([2, 256, 8, 8]) torch.Size([2, 256, 8, 8]) éc r¦r§r©rªrƒrrrr9r¬z$VectorQuantizedPSIViT_Audio.__init__)r°r­rrrƒrr¯$r®r¯cs<eZdZdZ   d ‡fdd„ Zd dd „Zd d d „Z‡ZS)r•a  Implements VQ Dictionary. Wraps `VectorQuantization` and `VectorQuantizationStraightThrough`. For more details refer to the specific class. Arguments --------- K : int Number of elements of VQ dictionary. D : int Dimensionality of VQ vectors. numclasses : int Number of possible classes activate_class_partitioning : bool `True` if latent space should be quantized for different classes. shared_keys : int Number of shared keys among classes. r‘TrcsHtƒ ¡t ||¡|_|jjj d|d|¡||_||_ ||_ dS)Ngð¿r4) rqrrr#Ú EmbeddingÚ embeddingr&r'Úuniform_r’rDrE)r‚rÚDr’rDrErƒrrrr`s  zVQEmbedding.__init__NcCs*| dddd¡ ¡}t ||jj|¡}|S)aá Wraps VectorQuantization. Computes VQ-dictionary indices for input quantization. Note that this forward step is not differentiable. Arguments --------- z_e_x : torch.Tensor Input tensor to be quantized. labels : torch.Tensor Predicted class for input representations (used for latent space quantization). Returns ------- Codebook's indices for quantized representation : torch.Tensor Example ------- >>> inputs = torch.ones(3, 256, 14, 25) >>> codebook = VQEmbedding(1024, 256) >>> labels = torch.Tensor([1, 0, 2]) >>> print(codebook(inputs, labels).shape) torch.Size([3, 14, 25]) rr1r‹r)Úpermutergr/r^r²r&)r‚Úz_e_xrÚz_e_x_ÚlatentsrrrrQqs  ÿzVQEmbedding.forwardc CsŒ| dddd¡ ¡}t ||jj ¡||j|j|j |j ¡\}}| dddd¡ ¡}t j |jjd|d}|  |¡}| dddd¡ ¡} || fS)a/ Implements the vector quantization with straight through approximation of the gradient. Arguments --------- z_e_x : torch.Tensor Input tensor to be quantized. labels : torch.Tensor Predicted class for input representations (used for latent space quantization). Returns ------- Straight through quantized representation and quantized representation : tuple Example ------- >>> inputs = torch.ones(3, 256, 14, 25) >>> codebook = VQEmbedding(1024, 256) >>> labels = torch.Tensor([1, 0, 2]) >>> quant, quant_ind = codebook.straight_through(inputs, labels) >>> print(quant.shape, quant_ind.shape) torch.Size([3, 256, 14, 25]) torch.Size([3, 256, 14, 25]) rr1r‹rr\)rµrgr[r^r²r&Údetachr’rDrErFrr`ra) r‚r¶rr·Úz_q_x_rPr¡Úz_q_x_bar_flattenÚ z_q_x_bar_Ú z_q_x_barrrrrŽs" ù  ÿ zVQEmbedding.straight_through)r‘Tr)N)r!rWrXrYrrrQrrŠrrrƒrr•Msú r•)rr)rYrÚtorch.nnr#Útorch.autogradrrr.r/r[ÚModulernr}rŽr¤r¯r•rrrrÚs   C bgB)j))