o %ݫibV@sdZddlZddlmZddlmZdddZdd ZGd d d eZGd d d eZ Gdddej Z Gdddej Z Gdddej Z Gddde ZGddde ZGdddej ZdS)zThis file implements the necessary classes and functions to implement Posthoc Interpretations via Quantization. Authors * Cem Subakan 2023 * Francesco Paissan 2023 N)FunctionTRAINcCsttd|d|||j}|d|jdd}|dd||}||k}|dkrItj|t |jd||jgdddk}|Stj|t |jd||jgdddk}|S)aThis class returns binary matrix that indicates the irrelevant regions in the VQ-dictionary given the labels array Arguments --------- labels : torch.Tensor 1 dimensional tensor of size [B] K : int Number of keys in the dictionary num_classes : int Number of possible classes N_shared : int Number of shared keys stage : str "TRAIN" or else Returns ------- irrelevant_regions : torch.Tensor Example ------- >>> labels = torch.Tensor([1, 0, 2]) >>> irrelevant_regions = get_irrelevant_regions(labels, 20, 3, 5) >>> print(irrelevant_regions.shape) torch.Size([3, 20]) ggRQ?rrdim) torchroundlinspacetodevice unsqueezerepeatshapecatoneszeros)labelsK num_classesN_sharedstage uniform_matlabels_expandedirrelevant_regionsrP/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/PIQ.pyget_irrelevant_regions sB   rcCs^|jj}|ddkr-ztj|jj|jj dWdSt y,t d|YdSwdS)z; Applies Xavier initialization to network weights. ConvrzSkipping initialization of N) __class____name__findnninitxavier_uniform_weightdatabiasfill_AttributeErrorprint)m classnamerrr weights_initPs r.c@s4eZdZdZe     d ddZeddZdS) VectorQuantizationaThis class defines the forward method for vector quantization. As VQ is not differentiable, it returns a RuntimeError in case `.grad()` is called. Refer to `VectorQuantizationStraightThrough` for a straight_through estimation of the gradient for the VQ operation.N TcCst}|d}|} |d|} |dddd| d| d} | d} t| |jd|||r5dndd} tj|ddd}tj| ddd d }tj ||| | d d d }|ratj || <tj |dd\}}|j| dd}| ||WdS1swYdS)a[ Applies VQ to vectors `input` with `codebook` as VQ dictionary. Arguments --------- ctx : torch context The context object for storing info for backwards. inputs : torch.Tensor Hidden representations to quantize. Expected shape is `torch.Size([B, W, H, C])`. codebook : torch.Tensor VQ-dictionary for quantization. Expected shape of `torch.Size([K, C])` with K dictionary elements. labels : torch.Tensor Classification labels. Used to define irrelevant regions and divide the latent space based on predicted class. Shape should be `torch.Size([B])`. num_classes : int Number of possible classes activate_class_partitioning : bool `True` if latent space should be quantized for different classes. shared_keys : int Number of shared keys among classes. training : bool `True` if stage is TRAIN. Returns ------- Codebook's indices for quantized representation : torch.Tensor Example ------- >>> inputs = torch.ones(3, 14, 25, 256) >>> codebook = torch.randn(1024, 256) >>> labels = torch.Tensor([1, 0, 2]) >>> print(VectorQuantization.apply(inputs, codebook, labels).shape) torch.Size([3, 14, 25]) rrrrVALID)rrrT)rkeepdimg?)alphabetaN)rno_gradsizeviewreshaperrrsumaddmmtinfminmark_non_differentiable)ctxinputscodebookrractivate_class_partitioning shared_keystrainingembedding_size inputs_sizeinputs_flattenrlabels_flattenr codebook_sqr inputs_sqr distances_indices_flattenindicesrrrforward`s> -      $zVectorQuantization.forwardcCstd)z;Handles error in case grad() is called on the VQ operation.zTrying to call `.grad()` on graph containing `VectorQuantization`. The function `VectorQuantization` is not differentiable. Use `VectorQuantizationStraightThrough` if you want a straight-through estimator of the gradient.) RuntimeError)rA grad_outputrrrbackwardszVectorQuantization.backwardNr0Tr0Tr! __module__ __qualname____doc__ staticmethodrQrTrrrrr/]s Sr/c@s@eZdZdZe     d ddZe     d ddZdS) !VectorQuantizationStraightThroughzThis class defines the forward method for vector quantization. As VQ is not differentiable, it approximates the gradient of the VQ as in https://arxiv.org/abs/1711.00937.Nr0Tc CsXt|||||||}|d} || ||| tj|d| d} | |} | | fS)a3 Applies VQ to vectors `input` with `codebook` as VQ dictionary and estimates gradients with a Straight-Through (id) approximation of the quantization steps. Arguments --------- ctx : torch context The context object for storing info for backwards. inputs : torch.Tensor Hidden representations to quantize. Expected shape is `torch.Size([B, W, H, C])`. codebook : torch.Tensor VQ-dictionary for quantization. Expected shape of `torch.Size([K, C])` with K dictionary elements. labels : torch.Tensor Classification labels. Used to define irrelevant regions and divide the latent space based on predicted class. Shape should be `torch.Size([B])`. num_classes : int Number of possible classes activate_class_partitioning : bool `True` if latent space should be quantized for different classes. shared_keys : int Number of shared keys among classes. training : bool `True` if stage is TRAIN. Returns ------- Quantized representation and codebook's indices for quantized representation : tuple Example ------- >>> inputs = torch.ones(3, 14, 25, 256) >>> codebook = torch.randn(1024, 256) >>> labels = torch.Tensor([1, 0, 2]) >>> quant, quant_ind = VectorQuantizationStraightThrough.apply(inputs, codebook, labels) >>> print(quant.shape, quant_ind.shape) torch.Size([3, 14, 25, 256]) torch.Size([1050]) rrrindex)r/applyr9save_for_backwardr@r index_selectview_as) rArBrCrrrDrErFrPrO codes_flattencodesrrrrQs"/   z)VectorQuantizationStraightThrough.forwardcCsrd\}} |jdr |}|jdr0|j\} } | d} |d| } t| } | d| | || dddddfS)zz Estimates gradient assuming vector quantization as identity function. (https://arxiv.org/abs/1711.00937) )NNrrrN) needs_input_gradclone saved_tensorsr8 contiguousr9r zeros_like index_add_)rArS grad_indicesrrrDrErF grad_inputs grad_codebookrPrCrGgrad_output_flattenrrrrTs     z*VectorQuantizationStraightThrough.backwardrU)NNTr0TrVrrrrr[s  Br[cs*eZdZdZdfdd ZddZZS)Conv2dEncoder_v2ay This class implements a convolutional encoder to extract classification embeddings from logspectra. Arguments --------- dim : int Number of channels of the extracted embeddings. Example ------- >>> inputs = torch.ones(3, 431, 513) >>> model = Conv2dEncoder_v2() >>> print(model(inputs).shape) torch.Size([3, 256, 26, 32]) csttd|ddd|_t||_t||ddd|_t||_t||ddd|_ t||_ t||ddd|_ t||_ t ||_t|_dS)Nrr1)super__init__r#Conv2dconv1 BatchNorm2dbn1conv2bn2conv3bn3conv4bn4 ResBlockAudioresblockReLUnonlselfrr rrrr7s      zConv2dEncoder_v2.__init__cCs|d}||}||}||}||}||}||}||}||}||}||}| |}||}| |}|S)z Computes forward pass. Arguments --------- x : torch.Tensor Log-power spectrogram. Expected shape `torch.Size([B, T, F])`. Returns ------- Embeddings : torch.Tensor r) r rtrvrrwrxryrzr{r|r~)rxh1h2h3h4rrrrQEs             zConv2dEncoder_v2.forward)ror!rWrXrYrrrQ __classcell__rrrrrn&srncs(eZdZdZfddZddZZS)r}aQThis class implements a residual block. Arguments --------- dim : int Input channels of the tensor to process. Matches output channels of the residual block. Example ------- >>> res = ResBlockAudio(128) >>> x = torch.randn(2, 128, 16, 16) >>> print(x.shape) torch.Size([2, 128, 16, 16]) c sLttt||dddt|tdt||dt||_dS)NrT)rqrrr# Sequentialrsrurblockrrrrrrxs   zResBlockAudio.__init__cCs|||S)zForward step. Arguments --------- x : torch.Tensor Tensor to process. Expected shape is `torch.Size([B, C, H, W])`. Returns ------- Residual block output : torch.Tensor )r)rrrrrrQs zResBlockAudio.forwardrrrrrr}hs  r}cs8eZdZdZ       d fdd Zd d ZZS) VectorQuantizedPSI_Audioa This class reconstructs log-power spectrograms from classifier's representations. Arguments --------- dim : int Dimensionality of VQ vectors. K : int Number of elements of VQ dictionary. numclasses : int Number of possible classes activate_class_partitioning : bool `True` if latent space should be quantized for different classes. shared_keys : int Number of shared keys among classes. use_adapter : bool `True` to learn an adapter for classifier's representations. adapter_reduce_dim : bool `True` if adapter should compress representations. Example ------- >>> psi = VectorQuantizedPSI_Audio(dim=256, K=1024) >>> x = torch.randn(2, 256, 16, 16) >>> labels = torch.Tensor([0, 2]) >>> logspectra, hcat, z_q_x = psi(x, labels) >>> print(logspectra.shape, hcat.shape, z_q_x.shape) torch.Size([2, 1, 257, 257]) torch.Size([2, 256, 8, 8]) torch.Size([2, 256, 8, 8]) 2Trcstt|||||d|_||_||_|r2t||_|r2t ||ddd|_ t ||ddd|_ t t ||dddtdt|t ||dddtt|t ||dddtt|t ||dddtt|t |dddd |_|tdS)N) numclassesrDrErpr1r1rrT )rqrr VQEmbeddingrC use_adapteradapter_reduce_dimr}adapterr#rsdownConvTranspose2duprrrudecoderr^r.)rrrrrDrErrrrrrrs>  z!VectorQuantizedPSI_Audio.__init__cCsj|jr ||}n|}|jr"||}|j||\}}||}n |j||\}}||}|||fS)a Forward step. Reconstructs log-power based on provided label's keys in VQ dictionary. Arguments --------- hs : torch.Tensor Classifier's representations. labels : torch.Tensor Predicted labels for classifier's representations. Returns ------- Reconstructed log-power spectrogram, reduced classifier's representations and quantized classifier's representations. : tuple )rrrrrCstraight_throughrr)rhsrhcatz_q_x_stz_q_xx_tilderrrrQs     z VectorQuantizedPSI_Audio.forward)rrrTrTTrrrrrrs ,rc"eZdZdZdfdd ZZS) VectorQuantizedPSIFocalNet_Audioal This class reconstructs log-power spectrograms from a FocalNet classifier's representations. Arguments --------- dim : int Dimensionality of VQ vectors. **kwargs : dict See documentation of `VectorQuantizedPSI_Audio`. Example ------- >>> psi = VectorQuantizedPSIFocalNet_Audio(dim=256, K=1024) >>> x = torch.randn(2, 256, 16, 16) >>> labels = torch.Tensor([0, 2]) >>> logspectra, hcat, z_q_x = psi(x, labels) >>> print(logspectra.shape, hcat.shape, z_q_x.shape) torch.Size([2, 1, 495, 593]) torch.Size([2, 256, 8, 8]) torch.Size([2, 256, 8, 8]) c tjd d|i|tt||dddtt|t||dddtt|t||dddtt|t||dddtt|t|dddd |_|t dS Nrr)rprr)rprr)rpr1)r0r rqrrr#rrrrurr^r.rrkwargsrrrrr"z)VectorQuantizedPSIFocalNet_Audio.__init__)rr!rWrXrYrrrrrrrrrcr)VectorQuantizedPSIViT_Audioab This class reconstructs log-power spectrograms from a ViT classifier's representations. Arguments --------- dim : int Dimensionality of VQ vectors. **kwargs : dict See documentation of `VectorQuantizedPSI_Audio`. Example ------- >>> psi = VectorQuantizedPSIViT_Audio(dim=256, K=1024) >>> x = torch.randn(2, 256, 16, 16) >>> labels = torch.Tensor([0, 2]) >>> logspectra, hcat, z_q_x = psi(x, labels) >>> print(logspectra.shape, hcat.shape, z_q_x.shape) torch.Size([2, 1, 495, 593]) torch.Size([2, 256, 8, 8]) torch.Size([2, 256, 8, 8]) c rrrrrrrrr9rz$VectorQuantizedPSIViT_Audio.__init__)rrrrrrr$rrcs<eZdZdZ   d fdd Zd dd Zd d d ZZS)ra  Implements VQ Dictionary. Wraps `VectorQuantization` and `VectorQuantizationStraightThrough`. For more details refer to the specific class. Arguments --------- K : int Number of elements of VQ dictionary. D : int Dimensionality of VQ vectors. numclasses : int Number of possible classes activate_class_partitioning : bool `True` if latent space should be quantized for different classes. shared_keys : int Number of shared keys among classes. rTrcsHtt|||_|jjjd|d|||_||_ ||_ dS)Ngr4) rqrrr# Embedding embeddingr&r'uniform_rrDrE)rrDrrDrErrrrr`s  zVQEmbedding.__init__NcCs*|dddd}t||jj|}|S)a Wraps VectorQuantization. Computes VQ-dictionary indices for input quantization. Note that this forward step is not differentiable. Arguments --------- z_e_x : torch.Tensor Input tensor to be quantized. labels : torch.Tensor Predicted class for input representations (used for latent space quantization). Returns ------- Codebook's indices for quantized representation : torch.Tensor Example ------- >>> inputs = torch.ones(3, 256, 14, 25) >>> codebook = VQEmbedding(1024, 256) >>> labels = torch.Tensor([1, 0, 2]) >>> print(codebook(inputs, labels).shape) torch.Size([3, 14, 25]) rr1rr)permutergr/r^rr&)rz_e_xrz_e_x_latentsrrrrQqs  zVQEmbedding.forwardc Cs|dddd}t||jj||j|j|j |j \}}|dddd}t j |jjd|d}| |}|dddd} || fS)a/ Implements the vector quantization with straight through approximation of the gradient. Arguments --------- z_e_x : torch.Tensor Input tensor to be quantized. labels : torch.Tensor Predicted class for input representations (used for latent space quantization). Returns ------- Straight through quantized representation and quantized representation : tuple Example ------- >>> inputs = torch.ones(3, 256, 14, 25) >>> codebook = VQEmbedding(1024, 256) >>> labels = torch.Tensor([1, 0, 2]) >>> quant, quant_ind = codebook.straight_through(inputs, labels) >>> print(quant.shape, quant_ind.shape) torch.Size([3, 256, 14, 25]) torch.Size([3, 256, 14, 25]) rr1rrr\)rrgr[r^rr&detachrrDrErFrr`ra) rrrrz_q_x_rPrz_q_x_bar_flatten z_q_x_bar_ z_q_x_barrrrrs"    zVQEmbedding.straight_through)rTr)N)r!rWrXrYrrrQrrrrrrrMs r)rr)rYrtorch.nnr#torch.autogradrrr.r/r[Modulernr}rrrrrrrrs   C bgB)j))