o }oiƭ@sddlmZmZmZmZddlZddlmZddlm Z ddl m Z ddl m Z mZmZmZmZmZddlmZddlmZmZdd lmZmZmZmZdd lmZGd d d eZGd ddeZ GdddeZ!GdddeZ"GdddeZ#GdddeZ$dS))DictListOptionalTupleNConformerEncoder)make_seq_mask_like)!SpectrogramToMultichannelFeatures)ChannelAttentionPoolChannelAveragePool"ParametricMultichannelWienerFilterTransformAttendConcatenateTransformAverageConcatenate WPEFilter)db2mag) NeuralModule typecheck) FloatType LengthsType NeuralTypeSpectrogramType)loggingcseZdZdZ         dd ed ed ed ed eedeededededeffdd Z e de ee ffddZ e de ee ffddZedejdejdeejejffddZZS)MaskEstimatorRNNaEstimate `num_outputs` masks from the input spectrogram using stacked RNNs and projections. The module is structured as follows: input --> spatial features --> input projection --> --> stacked RNNs --> output projection for each output --> sigmoid Reference: Multi-microphone neural speech separation for far-field multi-talker speech recognition (https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8462081) Args: num_outputs: Number of output masks to estimate num_subbands: Number of subbands of the input spectrogram num_features: Number of features after the input projections num_layers: Number of RNN layers num_hidden_features: Number of hidden features in RNN layers num_input_channels: Number of input channels dropout: If non-zero, introduces dropout on the outputs of each RNN layer except the last layer, with dropout probability equal to `dropout`. Default: 0 bidirectional: If `True`, use bidirectional RNN. rnn_type: Type of RNN, either `lstm` or `gru`. Default: `lstm` mag_reduction: Channel-wise reduction for magnitude features use_ipd: Use inter-channel phase difference (IPD) features NrTlstmrms num_outputs num_subbands num_features num_layersnum_hidden_featuresnum_input_channelsdropoutrnn_type mag_reductionuse_ipdc st|dur }t|| | d|_tjj|jj|jjd|_ | dkr5tjj ||d||d|_ n| dkrGtjj ||d||d|_ nt d| tjj|rWdnd|_tj|_tjfd d t|D|_tj|_dS) N)rr"r%r& in_features out_featuresrT) input_size hidden_sizer batch_firstr# bidirectionalgruzUnknown rnn_type: csg|] }tjjdqS)r')torchnnLinear.0_rrZ/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/audio/modules/masking.py xsz-MaskEstimatorRNN.__init__..)super__init__r featuresr0r1r2r num_channelsinput_projectionLSTMrnnGRU ValueErrorfc LayerNormnorm ModuleListrangeoutput_projectionsSigmoidoutput_nonlinearity) selfrrrr r!r"r#r-r$r%r& __class__r6r8r;?sN    zMaskEstimatorRNN.__init__returncCtdttdtdS+Returns definitions of module output ports.BCDTrSinput input_lengthrrrrKr7r7r8 input_types|  zMaskEstimatorRNN.input_typescCrOrQrRrWoutput output_lengthrrrr\r7r7r8 output_typesr^zMaskEstimatorRNN.output_typesrYrZcCs&|j||d\}}|j\}}}}|dddd}|||d}||}tjjjj || ddd  |j }|j ||\}}tjjjj|dd \} } | |j } ||| |} g} |jD]} | | } || } | dd} | | qctj| dd } t| | ddd }| |d } | | fS)aqEstimate `num_outputs` masks from the input spectrogram. Args: input: C-channel input, shape (B, C, F, N) input_length: Length of valid entries along the time dimension, shape (B,) Returns: Returns `num_outputs` masks in a tensor, shape (B, num_outputs, F, N), and output length with shape (B,) rXrrr/TF)r,enforce_sorted)r,axislengthsliketime_dim valid_ones)r<shapepermuteviewr>r0r1utilsr@pack_padded_sequencecputodeviceflatten_parameterspad_packed_sequencerErCrHrJ transposeappendstackr masked_fill)rKrYrZr5rSnum_feature_channelsrN input_packedrarbmasksoutput_projectionmask length_maskr7r7r8forwards6           zMaskEstimatorRNN.forward) rrNNrTrrN)__name__ __module__ __qualname____doc__intrfloatstrboolr;propertyrrr]rdrr0Tensorrr __classcell__r7r7rLr8r$sL   =0rc&seZdZdZ        d*d edededededededededededeeedeededeede deedeef$fdd Z e d!e ee ffd"d#Ze d!e ee ffd$d%Zed&ejd'ejd!eejejffd(d)ZZS)+MaskEstimatorFlexChannelsuEstimate `num_outputs` masks from the input spectrogram using stacked channel-wise and temporal layers. This model is using interlaved channel blocks and temporal blocks, and it can process arbitrary number of input channels. Default channel block is the transform-average-concatenate layer. Default temporal block is the Conformer encoder. Reduction from multichannel signal to single-channel signal is performed after `channel_reduction_position` blocks. Only temporal blocks are used afterwards. After the sequence of blocks, the output mask is computed using an additional output temporal layer and a nonlinearity. References: - Yoshioka et al, VarArray: Array-Geometry-Agnostic Continuous Speech Separation, 2022 - Jukić et al, Flexible multichannel speech enhancement for noise-robust frontend, 2023 Args: num_outputs: Number of output masks. num_subbands: Number of subbands on the input spectrogram. num_blocks: Number of blocks in the model. channel_reduction_position: After this block, the signal will be reduced across channels. channel_reduction_type: Reduction across channels: 'average' or 'attention' channel_block_type: Block for channel processing: 'transform_average_concatenate' or 'transform_attend_concatenate' temporal_block_type: Block for temporal processing: 'conformer_encoder' temporal_block_num_layers: Number of layers for the temporal block temporal_block_num_heads: Number of heads for the temporal block temporal_block_dimension: The hidden size of the model temporal_block_self_attention_model: Self attention model for the temporal block temporal_block_att_context_size: Attention context size for the temporal block mag_reduction: Channel-wise reduction for magnitude features mag_power: Power to apply on magnitude features use_ipd: Use inter-channel phase difference (IPD) features mag_normalization: Normalize using mean ('mean') or mean and variance ('mean_var') ipd_normalization: Normalize using mean ('mean') or mean and variance ('mean_var') rf attentiontransform_attend_concatenateconformer_encoderrel_posNabs_meanTrr num_blockschannel_reduction_positionchannel_reduction_typechannel_block_typetemporal_block_typetemporal_block_num_layerstemporal_block_num_headstemporal_block_dimension#temporal_block_self_attention_modeltemporal_block_att_context_sizer"r% mag_powerr&mag_normalizationipd_normalizationc s tt| |||||d|_||_td|j|dkr!|}||kr/td|d|||_td|jt j |_ t j |_ t|D]x}td|||kr|dkr]|jjn}td |||d krrt|d }n|d kr}t|d }ntd ||j |||jkrdkrnn|jjn}td||dkrt||dd}ntd|d|j |qItd||dkrt|_n|dkr|jdkr|jjn}t|d|_ntd|td|t j fddt|D|_t j |_dS)N)rr"r%rr&rrzTotal number of blocks: %drfzChannel reduction position z exceeds the number of blocks z1Channel reduction will be applied before block %dzPrepare block %drzDSetup channel block %s with %d input features and %d output featurestransform_average_concatenater'rzUnknown channel layer type: zSetup temporal block %srre)feat_inn_layersd_modelsubsampling_factorself_attention_modelatt_context_sizen_headszUnknown temporal block .zSetup channel reduction %saverager)r(z Unknown channel reduction type: zSetup %d output layersc s$g|]}tdddqS)re)rrrfeat_outrrrrrr3rrrrrr7r8r9as z6MaskEstimatorFlexChannels.__init__..)r:r;r r<rrdebugrBrr0r1rFchannel_blockstemporal_blocksrGrrr r{rr channel_reductionr output_layersrIrJ)rKrrrrrrrrrrrrr"r%rr&rrnchannel_in_features channel_blocktemporal_in_featurestemporal_blockchannel_reduction_in_featuresrLrr8r;s       $      z"MaskEstimatorFlexChannels.__init__rNcCrOrPr[r\r7r7r8r]sr^z%MaskEstimatorFlexChannels.input_typescCrOr_rcr\r7r7r8rd{r^z&MaskEstimatorFlexChannels.output_typesrYrZc Cs|j||d\}}|d|d}}t|jD]r}||jkrD|j||d}|d|d}} |d|| }|dkrC||}n ||jkrO|j|d}t |j |||d\}}Wdn1siwY||jkr|d} |||d| }|dkr|dd|}q|j|jkr|j|d}g} |j D](} t | ||d\} } Wdn1swY| | } | | qtj| dd } | | fS) z8Estimate `num_outputs` masks from the input spectrogram.rXrre)rYrf) audio_signallengthNdim)r<sizerGrrrreshaperepeat_interleaverrdisable_checksrrrJr{r0r|)rKrYrZrarbrSMrFrVr output_layerr mask_lengthr7r7r8rs@             z!MaskEstimatorFlexChannels.forward)rfrrrrrrrNNrNTNN)rrrrrrrrrrr;rrrr]rdrr0rrrrr7r7rLr8rsz)     0rc seZdZdZddejfdededejffdd Z d)d ej d ed ej fd dZ e e de de ddde diddej dej dej d ej fddZe de dide diddej d ej fddZe e de de dde de ddddej dej dej d eej ej ffd d!Zed eee ffd"d#Zed eee ffd$d%Ze d&ej dej d ej fd'd(ZZS)*MaskEstimatorGSSawEstimate masks using guided source separation with a complex angular Central Gaussian Mixture Model (cACGMM) [1]. This module corresponds to `GSS` in Fig. 2 in [2]. Notation is approximately following [1], where `gamma` denotes the time-frequency mask, `alpha` denotes the mixture weights, and `BM` denotes the shape matrix. Additionally, the provided source activity is denoted as `activity`. Args: num_iterations: Number of iterations for the EM algorithm eps: Small value for regularization dtype: Data type for internal computations (default `torch.cdouble`) References: [1] Ito et al., Complex Angular Central Gaussian Mixture Model for Directional Statistics in Mask-Based Microphone Array Signal Processing, 2016 [2] Boeddeker et al., Front-End Processing for the CHiME-5 Dinner Party Scenario, 2018 r:0yE>num_iterationsepsdtypecst|dkrtd|||_|dkrtd|||_|tjtjfvr1td|d||_t d|j j t d|jt d|jt d |jdS) Nrz+Number of iterations must be positive, got zeps must be positive, got Unsupported dtype z, expecting cfloat or cdoubleInitialized %s num_iterations: %sz eps: %g dtype: %s) r:r;rBrrr0cfloatcdoublerrrrMr)rKrrrrLr7r8r;s zMaskEstimatorGSS.__init__rexrrNcCs&tjj|d|dd}|||j}|S)a\Normalize input to have a unit L2-norm across `dim`. By default, normalizes across the input channels. Args: x: C-channel input signal, shape (B, C, F, T) dim: Dimension for normalization, defaults to -3 to normalize over channels Returns: Normalized signal, shape (B, C, F, T) r/T)ordrkeepdim)r0linalg vector_normr)rKrrnorm_xr7r7r8 normalizes zMaskEstimatorGSS.normalize)rSrTrUrSrTrVrRalphaactivitylog_pdfgamma)r]rdrrrcCs^|tj|dddd}t|}|d||ddddf}|tj|ddd|j}|S) atUpdate masks for the cACGMM. Args: alpha: component weights, shape (B, num_outputs, F) activity: temporal activity for the components, shape (B, num_outputs, T) log_pdf: logarithm of the PDF, shape (B, num_outputs, F, T) Returns: Masks for the components of the model, shape (B, num_outputs, F, T) Trirr.N.Nrr)r0maxexpsumr)rKrrr log_gammarr7r7r8 update_maskss  zMaskEstimatorGSS.update_maskscCstj|dd}|S)zUpdate weights for the individual components in the mixture model. Args: gamma: masks, shape (B, num_outputs, F, T) Returns: Component weights, shape (B, num_outputs, F) rfr)r0mean)rKrrr7r7r8update_weights#szMaskEstimatorGSS.update_weightszr zH_invBM_z)rrrrc Cs0|d}|||j}|td||j||}tj|dd}||d|j}||ddd}tj |\}} tj |j |jd}|tj |dd d d |j}||j}tjt|dd} td d || j| |}|dd}||j}| t|| d} | |fS)aUpdate PDF of the cACGMM. Args: z: directional statistics, shape (B, num_inputs, F, T) gamma: masks, shape (B, num_outputs, F, T) zH_invBM_z: energy weighted by shape matrices, shape (B, num_outputs, F, T) Returns: Logarithm of the PDF, shape (B, num_outputs, F, T), the energy term, shape (B, num_outputs, F, T) rzbmft,bift,bjft->bmfijrfr).NNrr/minTrrzbmfj,bmfkj,bkft->bmftjrer)rrr0einsumrvrconjrrzreighclamprealrlogsqrtabspow) rKrrr num_inputsscaleBMdenomLQ log_detBMrr7r7r8 update_pdf8s   $ zMaskEstimatorGSS.update_pdfcCstdttddS)rQrRr)rYr)rrr\r7r7r8r]s zMaskEstimatorGSS.input_typescCs dtdiS)rQrrR)rr\r7r7r8rdszMaskEstimatorGSS.output_typesrYc Cs|j\}}}}|d}|jj}|d|kr#td|jd|j|d|kr6td|jd|j|dkrAtd|tjj|dd l|j|j d }| s]Jd |j |j |d d } tj ||j d} | tj| ddd} | ddd|d} tj|||||j |jd} t|jD]} |j| d} |j| | | d\}} |j| ||d} qWdn1swYtt| rtd| | S)asApply GSS to estimate the time-frequency masks for each output source. Args: input: batched C-channel input signal, shape (B, num_inputs, F, T) activity: batched frame-wise activity for each output source, shape (B, num_outputs, T) Returns: Masks for the components of the model, shape (B, num_outputs, F, T) rerz#Batch dimension mismatch: activity z vs input rfz"Time dimension mismatch: activity z Expecting multiple outputs, got FenabledrExpecting complex input, got rrrrTrr/)rrw)rrrNzgamma contains NaNs: )rprrwtyperBr0ampautocastrvr is_complexrrrr unsqueezeexpandonesrGrrrranyisnan RuntimeError)rKrYrrSrrrVrrwrrritrrr7r7r8rs4  zMaskEstimatorGSS.forward)re)rrrrr0rrrrr;rrrrrrrrrrrr]rdrrr7r7rLr8rsX&$    @&rc seZdZdZddededeffdd Zed ee e ffd d Z ed ee e ffd d Z e dejdejdejd eejejffddZZS)MaskReferenceChannelaPA simple mask processor which applies mask on ref_channel of the input signal. Args: ref_channel: Index of the reference channel. mask_min_db: Threshold mask to a minimal value before applying it, defaults to -200dB mask_max_db: Threshold mask to a maximal value before applying it, defaults to 0dB r8 ref_channel mask_min_db mask_max_dbcsbt||_t||_t||_td|jj td|jtd|jtd|jdS)NzInitialized %s withz ref_channel: %dz mask_min: %fz mask_max: %f) r:r;rrmask_minmask_maxrrrMr)rKrrrrLr7r8r;s   zMaskReferenceChannel.__init__rNcCs$tdttdttdtdS)rQrRrWrYrZrrrrrr\r7r7r8r]s   z MaskReferenceChannel.input_typescCrOr_r[r\r7r7r8rdr^z!MaskReferenceChannel.output_typesrYrZrcCs>tj||j|jd}||dd|j|jddf}||fS)aApply mask on `ref_channel` of the input signal. This can be used to generate multi-channel output. If `mask` has `M` channels, the output will have `M` channels as well. Args: input: Input signal complex-valued spectrogram, shape (B, C, F, N) input_length: Length of valid entries along the time dimension, shape (B,) mask: Mask for M outputs, shape (B, M, F, N) Returns: M-channel output complex-valed spectrogram with shape (B, M, F, N) rrNre.)r0rrrr)rKrYrZrrar7r7r8rs"zMaskReferenceChannel.forward)rrr)rrrrrrr;rrrrr]rdrr0rrrrr7r7rLr8rs"  rcseZdZdZ             d(d ed ededeedeededededeedededededeedeffdd Z e de ee ffddZ e de ee ffd d!Ze  d)d"ejd#ejd$eejd%eejdejf d&d'ZZS)*MaskBasedBeamformeraMulti-channel processor using masks to estimate signal statistics. Args: filter_type: string denoting the type of the filter. Defaults to `mvdr` filter_beta: Parameter of the parameteric multichannel Wiener filter filter_rank: Parameter of the parametric multichannel Wiener filter filter_postfilter: Optional, postprocessing of the filter ref_channel: Optional, reference channel. If None, it will be estimated automatically ref_hard: If true, hard (one-hot) reference. If false, a soft reference ref_hard_use_grad: If true, use straight-through gradient when using the hard reference ref_subband_weighting: If true, use subband weighting when estimating reference channel num_subbands: Optional, used to determine the parameter size for reference estimation mask_min_db: Threshold mask to a minimal value before applying it, defaults to -200dB mask_max_db: Threshold mask to a maximal value before applying it, defaults to 0dB diag_reg: Optional, diagonal regularization for the multichannel filter eps: Small regularization constant to avoid division by zero mvdr_soudenrooneNrTFrư>r filter_type filter_beta filter_rankfilter_postfilterrref_hardref_hard_use_gradref_subband_weightingrrrpostmask_min_dbpostmask_max_dbdiag_regrc s.t|dvrtd|||_|jdkr)|dkr)td|j||d}d}t|||||||| ||d |_| | krGtd | d | d t| |_ t| |_ | | kr`td | d | d t| |_ t| |_ t d|jjt d|jt d|j t d|j t d|j t d|j dS)N)pmwfrzUnknown filter type rrzqUsing filter type %s: beta will be automatically set to zero (current beta %f) and rank to one (current rank %s).ror) betarank postfilterrr%r&r'rr*rzLower bound for the mask z(dB must be smaller than the upper bound dBzLower bound for the postmask z/dB must be smaller or equal to the upper bound rz filter_type: %sz mask_min: %ez mask_max: %ez postmask_min: %ez postmask_max: %e)r:r;rBr!rwarningr filterrrr postmask_min postmask_maxrrMr)rKr!r"r#r$rr%r&r'rrrr(r)r*rrLr7r8r;sV      zMaskBasedBeamformer.__init__rNcCs6tdttdttdtddtdtdddS)rQrRToptionalrW)rYrmask_undesiredrZ)rrrrr\r7r7r8r]`s  zMaskBasedBeamformer.input_typescCtdttdtdddSrQrRrWTr4r`r[r\r7r7r8rdj z MaskBasedBeamformer.output_typesrYrr6rZc Csl|durt||ddddfddd}g|d}}t|D]z}|dd|df} |dur8|dd|df} n|dkrAd| } n tj|dd| } tj| |j|jd } tj| |j|jd } |durn| |d } | |d } |j || | d } |j |j krtj|dd|df|j |j d } | | d} | | qtj|dd }|dur||ddddfd }||fS) aApply a mask-based beamformer to the input spectrogram. This can be used to generate multi-channel output. If `mask` has multiple channels, a multichannel filter is created for each mask, and the output is concatenation of individual outputs along the channel dimension. The total number of outputs is `num_masks * M`, where `M` is the number of channels at the filter output. Args: input: Input signal complex-valued spectrogram, shape (B, C, F, N) mask: Mask for M output signals, shape (B, num_masks, F, N) input_length: Length of valid entries along the time dimension, shape (B,) Returns: Multichannel output signal complex-valued spectrogram, shape (B, num_masks * M, F, N) Nr.rfFrjrerrro)rYmask_smask_nrh)rrrGr0rrrrr}r1r2r3r r{ concatenate) rKrYrr6rZrra num_masksmmask_dmask_uoutput_m postmask_mr7r7r8rrs4     " zMaskBasedBeamformer.forward)rrorNrTFFNrrrrr rNN)rrrrrrrrrr;rrrr]rdrr0rrrr7r7rLr8r s     C rcseZdZdZdddddejfdeded ed ed ed eed edej ffdd Z e de e effddZe de e effddZe ddejdeejdeejdejfddZZS)MaskBasedDereverbWPEaxMulti-channel linear prediction-based dereverberation using weighted prediction error for filter estimation. An optional mask to estimate the signal power can be provided. If a time-frequency mask is not provided, the algorithm corresponds to the conventional WPE algorithm. Args: filter_length: Length of the convolutional filter for each channel in frames. prediction_delay: Delay of the input signal for multi-channel linear prediction in frames. num_iterations: Number of iterations for reweighting mask_min_db: Threshold mask to a minimal value before applying it, defaults to -200dB mask_max_db: Threshold mask to a minimal value before applying it, defaults to 0dB diag_reg: Diagonal regularization for WPE eps: Small regularization constant dtype: Data type for internal computations References: - Kinoshita et al, Neural network-based spectrum estimation for online WPE dereverberation, 2017 - Yoshioka and Nakatani, Generalization of Multi-Channel Linear Prediction Methods for Blind MIMO Impulse Response Shortening, 2012 rerrr r filter_lengthprediction_delayrrrr*rrc stt||||d|_||_t||_t||_|tj tj fvr+t d|d||_ t d|jjt d|jt d|jt d|jt d|j dS) N)rErFr*rrz), expecting torch.cfloat or torch.cdoublerrz mask_min: %gz mask_max: %gr)r:r;rr1rrrrr0rrrBrrrrMr) rKrErFrrrr*rrrLr7r8r;s   zMaskBasedDereverbWPE.__init__rNcCs,tdttdtddtdtdddS)rQrRrWTr4rrr\r7r7r8r]s z MaskBasedDereverbWPE.input_typescCr7r8r[r\r7r7r8rdr9z!MaskBasedDereverbWPE.output_typesNrYrZrc Cs|j}|jj}tjj|ddL|j|jd}|s#td|jt |j D]+}t |}|dkrE|durEtj ||j |jd}||}|d} |j||| d \}} q(Wdn1s^wY||| fS) aGiven an input signal `input`, apply the WPE dereverberation algoritm. Args: input: C-channel complex-valued spectrogram, shape (B, C, F, T) input_length: Optional length for each signal in the batch, shape (B,) mask: Optional mask, shape (B, 1, F, N) or (B, C, F, T) Returns: Processed tensor with the same number of channels as the input, shape (B, C, F, T). FrrrrNrr/)rYrZpower)rrwrr0r r rvr rrGrrrrrr1) rKrYrZrio_dtyperwrai magnituderGrbr7r7r8rs  zMaskBasedDereverbWPE.forwardrC)rrrrr0rrrrrr;rrrrr]rdrrrrr7r7rLr8rDsP rD)%typingrrrrr0.nemo.collections.asr.modules.conformer_encoderr1nemo.collections.asr.parts.preprocessing.featuresr'nemo.collections.audio.modules.featuresr 4nemo.collections.audio.parts.submodules.multichannelr r r r rr(nemo.collections.audio.parts.utils.audiornemo.core.classesrrnemo.core.neural_typesrrrr nemo.utilsrrrrrrrDr7r7r7r8s*      'zB3