o i-@sddlmZddlmZmZmZmZmZddlZddl m Z ddl m Z ddlmZddlmZmZddlmZdd lmZe eje d kZGd d d eZdS) ) OrderedDict)DictListOptionalTupleUnionN)parse) STFTDecoder) STFTEncoder)tik_reg to_double) TCNDenseUNet) AbsSeparatorz1.9.0c seZdZdZ            d!fdd ZeddZeddZeddZ d"de e j de j de e deee e j e j effddZedd ZZS)#iNeuBeaiNeuBe, iterative neural/beamforming enhancement Reference: Lu, Y. J., Cornell, S., Chang, X., Zhang, W., Li, C., Ni, Z., ... & Watanabe, S. Towards Low-Distortion Multi-Channel Speech Enhancement: The ESPNET-Se Submission to the L3DAS22 Challenge. ICASSP 2022 p. 9201-9205. NOTES: As outlined in the Reference, this model works best when coupled with the MultiResL1SpecLoss defined in criterions/time_domain.py. The model is trained with variance normalized mixture input and target. e.g. with mixture of shape [batch, microphones, samples] you normalize it by dividing with torch.std(mixture, (1, 2)). You must do the same for the target signal. In the Reference, the variance normalization was performed offline (we normalized by the std computed on the entire training set and not for each input separately). However we found out that also normalizing each input and target separately works well. Args: n_spk: number of output sources/speakers. n_fft: stft window size. stride: stft stride. window: stft window type choose between 'hamming', 'hanning' or None. mic_channels: number of microphones channels (only fixed-array geometry supported). hid_chans: number of channels in the subsampling/upsampling conv layers. hid_chans_dense: number of channels in the densenet layers (reduce this to reduce VRAM requirements). ksz_dense: kernel size in the densenet layers thorough iNeuBe. ksz_tcn: kernel size in the TCN submodule. tcn_repeats: number of repetitions of blocks in the TCN submodule. tcn_blocks: number of blocks in the TCN submodule. tcn_channels: number of channels in the TCN submodule. activation: activation function to use in the whole iNeuBe model, you can use any torch supported activation e.g. 'relu' or 'elu'. output_from: output the estimate from 'dnn1', 'mfmcwf' or 'dnn2'. n_chunks: number of future and past frames to consider for mfMCWF computation. freeze_dnn1: whether or not freezing dnn1 parameters during training of dnn2. tik_eps: diagonal loading in the mfMCWF computation. hann rreludnn1F:0yE>c st||_||_||_||_||_|dd}ts Jdt||||d|_ t ||||d|_ t ||||||| | | | | d |_ t d||d|||| | | | | d |_dS)NrzHiNeuBe model requires torch>=1.9.0, please install latest torch version.)window) activation)super__init__n_spk output_fromn_chunks freeze_dnn1tik_epsis_torch_1_9_plusr encr decr rdnn2)selfr"n_fftstrider mic_channels hid_chanshid_chans_dense ksz_denseksz_tcn tcn_repeats tcn_blocks tcn_channelsrr#r$r%r&fft_c_channels __class__Z/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/enh/separator/ineube_separator.pyr!<sL   ziNeuBe.__init__cCs|j\}}}|dkr |Stjjjtjjj|||fdddd|ddfddd }|jd}|||d|d|}|dd}|S) a7unfolding STFT representation to add context in the mics channel. Args: mixture (torch.Tensor): 3D tensor (monaural complex STFT) of shape [B, T, F] batch, frames, microphones, frequencies. n_chunks (int): number of past and future to consider. Returns: est_unfolded (torch.Tensor): complex 3D tensor STFT with context channel. shape now is [B, T, C, F] batch, frames, context, frequencies. Basically same shape as a multi-channel STFT with C microphones. rconstant)moderr)rr)rr) kernel_sizepaddingr-) shapetorchnn functionalunfoldpad unsqueezereshape transpose)tf_rep chunk_sizebszfreq_ est_unfoldedr$r9r9r:rD{s    z iNeuBe.unfoldc Cs|dddd}|dd}|j\}}}}t|||d||||d|dd|}t|}t|} td|| } td|| } t t | dddd|dddd} td | | } td | |}| |}|ddS) amulti-frame multi-channel wiener filter. Args: mixture (torch.Tensor): multi-channel STFT complex mixture tensor, of shape [B, T, C, F] batch, frames, microphones, frequencies. estimate (torch.Tensor): monaural STFT complex estimate of target source [B, T, F] batch, frames, frequencies. n_chunks (int): number of past and future mfMCWF frames. If 0 then standard MCWF. tik_eps (float): diagonal loading for matrix inversion in MCWF computation. Returns: beamformed (torch.Tensor): monaural STFT complex estimate of target source after MFMCWF [B, T, F] batch, frames, frequencies. rrrrr=zbmft, bft->bmfzbmft, bnft->bmnfzbmnf, bnf->bmfz...mf,...mft->...ft) permuterHr@rrDrGr rAeinsumconjinverser to)mixtureestimater$r&rKmicsrMframes mix_unfolded estimate1zetascm_mix inv_scm_mix bf_vector beamformedr9r9r:mfmcwfs*    z iNeuBe.mfmcwfcCs"tjj|d||jdf}|S)Nrr=)rArBrCrEr@) input_tensor target_lenr9r9r:pad2sz iNeuBe.pad2Ninputilens additionalreturnc s&|j\}}}|||d}||}|jr|}|j\} } } } ||||j| | |d|||jd|fddt jdDt } |j dkrX|| fS|j dvr| d<t ||||j| | |j|j||j| | } || ||j| | |d|||jd||j dkrfd dt jdD|| fS|j d kr| d<| d <|t||jddd|||j| | d | ||j| | d fd }||d d df|d|||jd|fddt jdD|| fStt)aForward. Args: input (torch.Tensor): batched multi-channel audio tensor with C audio channels and T samples [B, T, C] ilens (torch.Tensor): input lengths [Batch] additional (Dict or None): other data, currently unused in this model. Returns: enhanced (List[Union(torch.Tensor)]): [(B, T), ...] list of len n_spk of mono audio tensors with T samples. ilens (torch.Tensor): (B,) others predicted data, e.g. masks: OrderedDict[ 'mask_spk1': torch.Tensor(Batch, Frames, Freq), 'mask_spk2': torch.Tensor(Batch, Frames, Freq), ... 'mask_spkn': torch.Tensor(Batch, Frames, Freq), ] additional (Dict or None): other data, currently unused in this model, we return it also in output. rr=cg|] }dd|fqSNr9.0src) output_dnn1r9r: z"iNeuBe.forward..rr)r`r*r`crhrir9rj) output_mfmcwfr9r:rnror*beamrNcrhrir9rj) output_dnn2r9r:rn"ro)r@r(rr%detachr)rGnum_spkrcrangerr#rr`r"r$r&r*rAcatrepeatrFNotImplementedError)r+rdrerfrK mixture_lenrWmix_stftest_dnn1rMrXrLothers est_mfmcwfest_dnn2r9)rmrrrpr:forwards          ziNeuBe.forwardcCs|jSri)r")r+r9r9r:rt+sziNeuBe.num_spk)rrrrrrrrrrrrrrrFrri)__name__ __module__ __qualname____doc__r! staticmethodrDr`rcrrATensorrrrrrrpropertyrt __classcell__r9r9r7r:rsL-?  '   ar) collectionsrtypingrrrrrrApackaging.versionrV espnet2.enh.decoder.stft_decoderr espnet2.enh.encoder.stft_encoderr espnet2.enh.layers.beamformerr r espnet2.enh.layers.tcndenseunetr #espnet2.enh.separator.abs_separatorr __version__r'rr9r9r9r:s