o wiD@sddlZddlmZmZmZmZddlZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZmZmZmZmZddlmZGd d d eZGd d d eZGd ddeZGdddeZGdddeZGdddeZGdddeZ GdddeZ!dS)N)CallableDictOptionalTuple)make_seq_mask_like)MultiHeadAttention)covariance_matrix) NeuralModule typecheck) AudioSignal FloatType LengthsType NeuralTypeSpectrogramType)loggingc seZdZdZ     ddededeedeed eef fd d Ze d d Z e ddZ e e de jde jfddZZS)ChannelAugmentalRandomly permute and selects a subset of channels. Args: permute_channels (bool): Apply a random permutation of channels. num_channels_min (int): Minimum number of channels to select. num_channels_max (int): Max number of channels to select. rng: Optional, random generator. seed: Optional, seed for the generator. TNpermute_channelsnum_channels_minnum_channels_maxrngseedcst|durt|n||_||_||_||_|dur,||kr,td|d|t d|j j t d|jt d|jt d|jdS)NzMin number of channels z/ cannot be greater than max number of channels Initialized %s withz permute_channels: %sz num_channels_min: %sz num_channels_max: %s) super__init__randomRandom_rngrrr ValueErrorrdebug __class____name__)selfrrrrrr q/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/audio/parts/submodules/multichannel.pyr(s zChannelAugment.__init__cCdtdtiS))Returns definitions of module input typesinputBCTrr r"r$r$r% input_typesA zChannelAugment.input_typescCr&)*Returns definitions of module output typesoutputr)r-r.r$r$r% output_typesHr0zChannelAugment.output_typesr(returncCs|jdks Jd|d}||jkrtd|d|jd|jdur&|n|j}|j|j|}tt|}|j r@|j ||d|}|dd|ddfS)Nz$Expecting input with shape (B, C, T)rzNumber of input channels (z5) is smaller than the min number of output channels ()) ndimsizer RuntimeErrorrrrandintlistrangershuffle)r"r(num_channels_inrnum_channels_outchannelsr$r$r%forwardOs     zChannelAugment.forward)TrNNN)r! __module__ __qualname____doc__boolintrrrpropertyr/r3r torchno_gradTensorrA __classcell__r$r$r#r%rs2   "rcsdeZdZdZddedeeffdd ZeddZed d Z e d e j d e j fd dZ ZS)TransformAverageConcatenateaApply transform-average-concatenate across channels. We're using a version from [2]. Args: in_features: Number of input features out_features: Number of output features References: [1] Luo et al, End-to-end Microphone Permutation and Number Invariant Multi-channel Speech Separation, 2019 [2] Yoshioka et al, VarArray: Array-Geometry-Agnostic Continuous Speech Separation, 2022 N in_features out_featurescst|dur |}|ddkrtd|tjtjj||dddtj|_tjtjj||dddtj|_ t d|j j t d|t d|dS) NrGNumber of output features should be divisible by two, currently set to Fbiasr in_features: %d out_features: %d)rrrrHnn SequentialLinearReLUtransform_channeltransform_averagerrr r!)r"rMrNr#r$r%rus   z$TransformAverageConcatenate.__init__cCr&r'r(r*r+Dr,rrr.r$r$r%r/r0z'TransformAverageConcatenate.input_typescCr&r1r2r\r^r.r$r$r%r3r0z(TransformAverageConcatenate.output_typesr(r4c Csx|j\}}}}|dddd}||}tj|ddd}|dd|d}||}tj||gdd }|dddd}|S) Args: input: shape (B, M, in_features, T) Returns: Output tensor with shape shape (B, M, out_features, T) rr5rrOTdimkeepdimrc)shapepermuterZrHmeanexpandrYcat) r"r(r*MFr,average transformr2r$r$r%rAs   z#TransformAverageConcatenate.forwardN)r!rBrCrDrFrrrGr/r3r rHrJrArKr$r$r#r%rLhs    rLc sleZdZdZddedeededeffd d Zed d Z ed dZ e de j de j fddZZS)TransformAttendConcatenateuApply transform-attend-concatenate across channels. The output is a concatenation of transformed channel and MHA over channels. Args: in_features: Number of input features out_features: Number of output features n_head: Number of heads for the MHA module dropout_rate: Dropout rate for the MHA module References: - Jukić et al, Flexible multichannel speech enhancement for noise-robust frontend, 2023 NrrMrNn_head dropout_ratecst|dur |}|ddkrtd|tjtjj||dddtj|_tjtjj||dddtj|_ t ||d|d|_ t d|jjt d|t d |t d |t d |dS) NrOrrPFrQrsn_featrtrrSrTz n_head: %dz dropout_rate: %f)rrrrHrUrVrWrXrYtransform_attendr attentionrrr r!)r"rMrNrsrtr#r$r%rs"     z#TransformAttendConcatenate.__init__cCr&r[r^r.r$r$r%r/r0z&TransformAttendConcatenate.input_typescCr&r_r^r.r$r$r%r3r0z'TransformAttendConcatenate.output_typesr(r4c Cs|j\}}}}|dddd}|||||}||}||}|j|||dd}tj||gdd}||||d}|dddd}|S) r`rr5rrONquerykeyvaluemaskrerf) rgrhreshaperYrwrxrHrkview) r"r(r*rlrmr,roattendr2r$r$r%rAs   z"TransformAttendConcatenate.forward)Nrrr)r!rBrCrDrFrfloatrrGr/r3r rHrJrArKr$r$r#r%rqs$   rqcsTeZdZdZfddZeddZeddZede j d e j fd d Z Z S) ChannelAveragePoolz&Apply average pooling across channels.csttd|jjdS)NInitialized %s)rrrrr r!r.r#r$r%rs zChannelAveragePool.__init__cCr&r[r^r.r$r$r%r/r0zChannelAveragePool.input_typescCr&r1r2r*r]r,r^r.r$r$r%r3r0zChannelAveragePool.output_typesr(r4cCstj|ddS) Args: input: shape (B, M, F, T) Returns: Output tensor with shape shape (B, F, T) rf)rHri)r"r(r$r$r%rA&s zChannelAveragePool.forward) r!rBrCrDrrGr/r3r rHrJrArKr$r$r#r%rs    rcsdeZdZdZddededeffdd Zed d Zed d Z e d e j de j fddZ ZS)ChannelAttentionPoolu#Use attention pooling to aggregate information across channels. First apply MHA across channels and then apply averaging. Args: in_features: Number of input features out_features: Number of output features n_head: Number of heads for the MHA module dropout_rate: Dropout rate for the MHA module References: - Wang et al, Neural speech separation using sparially distributed microphones, 2020 - Jukić et al, Flexible multichannel speech enhancement for noise-robust frontend, 2023 rrrMrsrtcsXt||_t|||d|_td|jjtd|td|td|dS)NrurrSz num_heads: %dz dropout_rate: %d) rrrMrrxrrr r!)r"rMrsrtr#r$r%rAs   zChannelAttentionPool.__init__cCr&r[r^r.r$r$r%r/Kr0z ChannelAttentionPool.input_typescCr&rr^r.r$r$r%r3Rr0z!ChannelAttentionPool.output_typesr(r4cCst|j\}}}}|dddd}|||||}|j|||dd}||||d}|dddd}tj|dd }|S) rrr5rrONryrer)axis)rgrhr~rxrrHri)r"r(r*rlrmr,r2r$r$r%rAYs zChannelAttentionPool.forward)rr)r!rBrCrDrFrrrGr/r3r rHrJrArKr$r$r#r%r2s   rcs$eZdZdZ          d,d ed ed eed eed edededeedeedeffdd Z e d-de j dede j fddZ de j de j fddZde j de j de j fdd Zde j de j d!e j de j fd"d#Zed$d%Zed&d'Zede j d(e j d)e j de j fd*d+ZZS)."ParametricMultichannelWienerFilteraParametric multichannel Wiener filter, with an adjustable tradeoff between noise reduction and speech distortion. It supports automatic reference channel selection based on the estimated output SNR. Args: beta: Parameter of the parameteric filter, tradeoff between noise reduction and speech distortion (0: MVDR, 1: MWF). rank: Rank assumption for the speech covariance matrix. postfilter: Optional postfilter. If None, no postfilter is applied. ref_channel: Optional, reference channel. If None, it will be estimated automatically. ref_hard: If true, estimate a hard (one-hot) reference. If false, a soft reference. ref_hard_use_grad: If true, use straight-through gradient when using the hard reference ref_subband_weighting: If true, use subband weighting when estimating reference channel num_subbands: Optional, used to determine the parameter size for reference estimation diag_reg: Optional, diagonal regularization for the multichannel filter eps: Small regularization constant to avoid division by zero References: - Souden et al, On Optimal Frequency-Domain Multichannel Linear Filtering for Noise Reduction, 2010 ?oneNTFư>:0yE>betarank postfilter ref_channelref_hardref_hard_use_gradref_subband_weighting num_subbandsdiag_regepsc sTt||_||_|jdkr"|jdkr"td|jd|jd|dvr.td|d||_| durA| dkrAtd | d | |_| dkrPtd | d | |_||_|jd krft ||||| d |_ nd|_ |jdu|_ t d|jjt d|jt d|jt d|jt d|jt d|jt d|jt d|j dS)NfullrzRank z is not compatible with beta .)Nbanz Postfilter z is not supported.zDiagonal regularization z must be positive.zEpsilon max_snr)hard hard_use_gradsubband_weightingrrrz beta: %fz rank: %sz postfilter: %sz diag_reg: %gz eps: %gz ref_channel: %sz is_mimo: %s)rrrrrrrrrReferenceChannelEstimatorSNR ref_estimatoris_mimorrr r!) r"rrrrrrrrrrr#r$r%rsB    z+ParametricMultichannelWienerFilter.__init__xrdr4cCs.tj|dddd}|r|dd}|S)zCalculate trace of matrix slices over the last two dimensions in the input tensor. Args: x: tensor, shape (..., C, C) Returns: Trace for each (C, C) matrix. shape (...) raredim1dim2)rHdiagonalsum unsqueeze)rrdtracer$r$r%rs z(ParametricMultichannelWienerFilter.tracepsdcCsF|j||j|j}|t|dtj|jd|j d}|S)zApply diagonal regularization on psd. Args: psd: tensor, shape (..., C, C) Returns: Tensor, same shape as input. redevice) rrrealrrH diag_embedronesrgr)r"rrr$r$r%apply_diag_regs *z1ParametricMultichannelWienerFilter.apply_diag_regr(filtercCs|s td|j|std|j|jdks1|d|dks1|d|dkr=td|jd|jtd| |}|S) aApply the MIMO filter on the input. Args: input: batch with C input channels, shape (B, C, F, T) filter: batch of C-input, M-output filters, shape (B, F, C, M) Returns: M-channel filter output, shape (B, M, F, T) z'Expecting complex-valued filter, found z&Expecting complex-valued input, found rrrarz Filter shape z", not compatible with input shape zbfcm,bcft->bmft) is_complex TypeErrordtyper7r8rrgrHeinsumconj)r"r(rr2r$r$r% apply_filters 2z/ParametricMultichannelWienerFilter.apply_filterpsd_ncCsl|d}td||||}t||}td|||}|}|||j}|d|}|S)a@Apply blind analytic normalization postfilter. Note that this normalization has been derived for the GEV beamformer in [1]. More specifically, the BAN postfilter aims to scale GEV to satisfy the distortionless constraint and the final analytical expression is derived using an assumption on the norm of the transfer function. However, this may still be useful in some instances. Args: input: batch with M output channels (B, M, F, T) filter: batch of C-input, M-output filters, shape (B, F, C, M) psd_n: batch of noise PSDs, shape (B, F, C, C) Returns: Filtere input, shape (B, M, F, T) References: - Warsitz and Haeb-Umbach, Blind Acoustic Beamforming Based on Generalized Eigenvalue Decomposition, 2007 razbfcm,bfci,bfij,bfjm->bmfzbfcm,bfci,bfim->bmf).N)r8rHrrsqrtabsr)r"r(rr num_inputs numerator denominatorrr$r$r% apply_bans  z,ParametricMultichannelWienerFilter.apply_bancCs$tdttdttdtdS)r'r\r)r(mask_smask_n)rrr r.r$r$r%r/(   z.ParametricMultichannelWienerFilter.input_typescCr&r_r^r.r$r$r%r31r0z/ParametricMultichannelWienerFilter.output_typesrrc Cs|j}tjj|jjdd|}|}|}t||d}t||d}|j dkrM|j r3| |}tj ||}|j|ddj}||j||j}n$|j dkri||j|} |j ra| | } tj | |}ntd|j tj|jtr|d |jfd }n%|jd ur|j|||d |j} tj|| d d d d d d fd dd }|j||d} |jdkr|j| ||d} Wd n1swY| |S)aReturn processed signal. The output has either one channel (M=1) if a ref_channel is selected, or the same number of channels as the input (M=C) if ref_channel is None. Args: input: Input signal, complex tensor with shape (B, C, F, T) mask_s: Mask for the desired signal, shape (B, F, T) mask_n: Mask for the undesired noise, shape (B, F, T) Returns: Processed signal, shape (B, M, F, T) F)enabled)rr}rT)rdrzUnexpected rank .reNWpsd_srrb)r(rr)r(rr)rrHampautocastrtypecdoubledoublerrrrlinalgsolverrrrr9jit isinstancerrFrrtorrrr) r"r(rriodtyperrrlampsd_snref_channel_tensorr2r$r$r%rA8s<       (  4z*ParametricMultichannelWienerFilter.forward) rrNNTTFNrr)F)r!rBrCrDrstrrrFrEr staticmethodrHrJrrrrrGr/r3r rArKr$r$r#r%rwsX   @""  ,rc seZdZdZ     ddededed eed ef fd d Ze d dZ e ddZ e de jde jde jde jfddZZS)raEstimate a reference channel by selecting the reference that maximizes the output SNR. It returns one-hot encoded vector or a soft reference. A straight-through estimator is used for gradient when using hard reference. Args: hard: If true, use hard estimate of ref channel. If false, use a soft estimate across channels. hard_use_grad: Use straight-through estimator for the gradient. subband_weighting: If true, use subband weighting when adding across subband SNRs. If false, use average across subbands. References: Boeddeker et al, Front-End Processing for the CHiME-5 Dinner Party Scenario, 2018 TFNrrrrrrcst||_||_||_||_|r|durtd|d|r*tj t |nd|_ |r8tj t |nd|_ t d|jjt d|jt d|jt d|jt d|t d|jdS) NzANumber of subbands must be provided when using subband_weighting=rrz hard: %dz hard_use_grad: %dz subband_weighting: %dz num_subbands: %sz eps: %e)rrrrrrrrHrU Parameterrweight_sweight_nrrr r!)r"rrrrrr#r$r%rs   z%ReferenceChannelEstimatorSNR.__init__cCs$tdttdttdtdS)r')r*r]r+r+rr^r.r$r$r%r/rz(ReferenceChannelEstimatorSNR.input_typescCr&)r1r2)r*r+)rr r.r$r$r%r3r0z)ReferenceChannelEstimatorSNR.output_typesrrrr4c Cs0|jr>td|||}td|||}tj||jjddddd}tj||j jddddd}ntd|||}td|||}|||j }dt ||j }|jdd}|j r|j dd d \}} t|d| d } |jr| ||} | S| } | S|} | S) a Args: W: Multichannel input multichannel output filter, shape (B, F, C, M), where C is the number of input channels and M is the number of output channels psd_s: Covariance for the signal, shape (B, F, C, C) psd_n: Covariance for the noise, shape (B, F, C, C) Returns: One-hot or soft reference channel, shape (B, M) z...jm,...jk,...km->...mrrfrraz...fjm,...fjk,...fkm->...m reTrbr)rrHrrrrrsoftmaxrrrlog10rmax zeros_likescatterrdetach) r"rrrpow_spow_nsnrref_soft_idxrrefr$r$r%rAs( "$ z$ReferenceChannelEstimatorSNR.forward)TTFNr)r!rBrCrDrErrFrrrGr/r3r rHrJrArKr$r$r#r%rs0  ,rc sbeZdZdZd(dededeedeffdd Zed e e e ffd d Z ed e e e ffd dZ e d)dejdejdeejd ejfddZe d*dejdededeed ejf ddZedejd ejfddZ d)dejdejdejdeejd eejf dd Zd!ejd"ejd ejfd#d$Z d+d%ejdeejdeejd ejfd&d'ZZS), WPEFilteruWA weighted prediction error filter. Given input signal, and expected power of the desired signal, this class estimates a multiple-input multiple-output prediction filter and returns the filtered signal. Currently, estimation of statistics and processing is performed in batch mode. Args: filter_length: Length of the prediction filter in frames, per channel prediction_delay: Prediction delay in frames diag_reg: Diagonal regularization for the correlation matrix Q, applied as diag_reg * trace(Q) + eps eps: Small positive constant for regularization References: - Yoshioka and Nakatani, Generalization of Multi-Channel Linear Prediction Methods for Blind MIMO Impulse Response Shortening, 2012 - Jukić et al, Group sparsity for MIMO speech dereverberation, 2015 rr filter_lengthprediction_delayrrcsnt||_||_||_||_td|jj td|jtd|jtd|jtd|jdS)Nrz filter_length: %dz prediction_delay: %dz diag_reg: %gz eps: %g) rrrrrrrrr r!)r"rrrrr#r$r%rs zWPEFilter.__init__r4cCs(tdttdttdtdddS)+Returns definitions of module output ports.r\r*Toptional)r(power input_lengthrrr r.r$r$r%r/s  zWPEFilter.input_typescCstdttdtdddS)rr\rTr)r2 output_lengthrr.r$r$r%r3s zWPEFilter.output_typesNr(rrc Cstj|dd}d||j}|j||j|jd}|j||||d\}}|j||d}|j||d} || } |durGt || dd d } | | d } | |fS) aNGiven input and the predicted power for the desired signal, estimate the WPE filter and return the processed signal. Args: input: Input signal, shape (B, C, F, N) power: Predicted power of the desired signal, shape (B, C, F, N) input_length: Optional, length of valid frames in `input`. Defaults to `None` Returns: Tuple of (processed_signal, output_length). Processed signal has the same shape as the input signal (B, C, F, N), and the output length is the same as the input length. rrfrdelay)r(weight tilde_inputr)QR)rrNreFlengthsliketime_dim valid_ones) rHrir convtensorrrestimate_correlationsestimate_filterrr masked_fill) r"r(rrrrrrGundesired_signaldesired_signal length_maskr$r$r%rA#s  zWPEFilter.forwardrrrn_stepsc Cs|jdkr td|j|j\}}}}|dur|}tjj||d|df}|d|d} | ddddddd|ddf} | S)aCreate a tensor equivalent of convmtx_mc for each example in the batch. The input signal tensor `x` has shape (B, C, F, N). Convtensor returns a view of the input signal `x`. Note: We avoid reshaping the output to collapse channels and filter taps into a single dimension, e.g., (B, F, N, -1). In this way, the output is a view of the input, while an additional reshape would result in a contiguous array and more memory use. Args: x: input tensor, shape (B, C, F, N) filter_length: length of the filter, determines the shape of the convolution tensor delay: delay to add to the input signal `x` before constructing the convolution tensor n_steps: Optional, number of time steps to keep in the out. Defaults to the number of time steps in the input tensor. Returns: Return a convolutional tensor with shape (B, C, F, n_steps, filter_length) rrz1Expecting a 4-D input. Received input with shape Nrrre)r7r9rgrHrU functionalpadunfold) clsrrrrr*r+rmNtilde_Xr$r$r%rSs &zWPEFilter.convtensorc Cs|j\}}}}}|ddddd}||||||}g}t|D]}||tt|||||d|<q!|d|fS)aReshape and permute columns to convert the result of convtensor to be equal to convmtx_mc. This is used for verification purposes and it is not required to use the filter. Args: x: output of self.convtensor, shape (B, C, F, N, filter_length) Returns: Output has shape (B, F, N, C*filter_length) that corresponds to the layout of convmtx_mc. rrOr5rrr.)rgrhr~r<npfliparange) r rr*r+rmr rrhmr$r$r%permute_convtensor}s    zWPEFilter.permute_convtensorrrc Cs|durt||ddd}||d}td||ddddddddf|}td||dddddddf|}||fS)a[ Args: input: Input signal, shape (B, C, F, N) weight: Time-frequency weight, shape (B, F, N) tilde_input: Multi-channel convolution tensor, shape (B, C, F, N, filter_length) input_length: Length of each input example, shape (B) Returns: Returns a tuple of correlation matrices for each batch. Let `X` denote the input signal in a single subband, `tilde{X}` the corresponding multi-channel correlation matrix, and `w` the vector of weights. The first output is Q = tilde{X}^H * diag(w) * tilde{X}, for each (b, f). The matrix Q has shape (C * filter_length, C * filter_length) The output is returned in a tensor with shape (B, F, C, filter_length, C, filter_length). The second output is R = tilde{X}^H * diag(w) * X, for each (b, f). The matrix R has shape (C * filter_length, C) The output is returned in a tensor with shape (B, F, C, filter_length, C). The last dimension corresponds to output channels. NreFrrzbjfik,bmfin->bfjkmnzbjfik,bmfi->bfjkm)rrrHrr)r"r(rrrrrrr$r$r%rs 0.zWPEFilter.estimate_correlationsrrc Cs|j\}}}}}}||jksJd|jd|j|||||j||}|||||j|}|jr\|jtj|ddddj|j}|t | dtj |jd|j d}tj ||} | |||||} | ddd d d } | S) ahEstimate the MIMO prediction filter as G(b,f) = Q(b,f) \ R(b,f) for each subband in each example in the batch (b, f). Args: Q: shape (B, F, C, filter_length, C, filter_length) R: shape (B, F, C, filter_length, C) Returns: Complex-valued prediction filter, shape (B, C, F, C, filter_length) z Shape of Q z is not matching filter length rarerrrrrrrOr5)rgrr~rrHrrrrrrrrrrrh) r"rrr*rmr+rrrrr$r$r%rs $*zWPEFilter.estimate_filterrcCs^|dur |dur td|dur|durtd|dur&|j||j|jd}td||}|S)aXApply a prediction filter `filter` on the input `input` as output(b,f) = tilde{input(b,f)} * filter(b,f) If available, directly use the convolution matrix `tilde_input`. Args: input: Input signal, shape (B, C, F, N) tilde_input: Convolution matrix for the input signal, shape (B, C, F, N, filter_length) filter: Prediction filter, shape (B, C, F, C, filter_length) Returns: Multi-channel signal obtained by applying the prediction filter on the input signal, same shape as input (B, C, F, N) Nz*Both inputs cannot be None simultaneously.z.Both inputs cannot be provided simultaneously.rzbjfik,bmfjk->bmfi)r9rrrrHr)r"rr(rr2r$r$r%rszWPEFilter.apply_filter)rrrp)rN)NN)r!rBrCrDrFrrrrGrrrr/r3r rHrJrA classmethodrrrrrrrKr$r$r#r%rsp$  / )  /&r)"rtypingrrrrnumpyrrH1nemo.collections.asr.parts.preprocessing.featuresr:nemo.collections.asr.parts.submodules.multi_head_attentionr(nemo.collections.audio.parts.utils.audiornemo.core.classesr r nemo.core.neural_typesr r r rr nemo.utilsrrrLrqrrrrrr$r$r$r%s(    KQX!E s