o oix_ @stddlmZmZddlZddlmmZddlmZmZd;dede de de d ef d d Z Gd d d ej Z dede d efddZ deded efddZdeded efddZGddde ZGddde ZGddde ZGddde ZGdd d e Zdd5d6Zd?d7d8Zd@d9d:ZdS)A)FinalListN)Tensornnx window_lengthstepdimreturncCst|j}t|}t|||||||<|dkr0||d|||d||n|dkrA|||||n||||||||||||<|||S)aReturns a tensor with chunks of overlapping windows of the first dim of x. Args: x (Tensor): Input of shape [B, T, ...] window_length (int): Length of each window step (int): Step/hop of each window w.r.t. the original signal x dim (int): Dimension on to apply the windowing Returns: windowed tensor (Tensor): Output tensor with shape (if dim==1) [B, (N - window_length + step) // step, window_length, ...] rr)listshapestrideintinsertappend as_strided)rrr r rrrK/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/df/multiframe.py as_windoweds    rc seZdZUdZeeed<eeed<eeed<eeed<ddedededeffd d Zd e fd d Z d e fddZ e dde de de fddZe d e de de fddZZS)MultiFrameModuleaMulti-frame speech enhancement modules. Signal model and notation: Noisy: `x = s + n` Enhanced: `y = f(x)` Objective: `min ||s - y||` PSD: Power spectral density, notated eg. as `Rxx` for noisy PSD. IFC: Inter-frame correlation vector: PSD*u, u: selection vector. Notated as `rxx` RTF: Relative transfere function, also called steering vector. num_freqs frame_size need_unfoldrealrF lookaheadcsvt||_||_||_|r"tdddd|d||fd|_ntdd|d||fd|_|dk|_ ||_ dS)adMulti-Frame filtering module. Args: num_freqs (int): Number of frequency bins used for filtering. frame_size (int): Frame size in FD domain. lookahead (int): Lookahead, may be used to select the output time step. Note: This module does not add additional padding according to lookahead! rrN) super__init__rrrr ConstantPad3dpad ConstantPad2drr)selfrrrr __class__rrr:s $  zMultiFrameModule.__init__speccCs:|jr||d|jd}|ddddddS|dS) Nrrr )rr!unfoldrpermute unsqueezer#r&rrrspec_unfold_realNs z!MultiFrameModule.spec_unfold_realcCs&|jr||d|jdS|dS)aPads and unfolds the spectrogram according to frame_size. Args: spec (complex Tensor): Spectrogram of shape [B, C, T, F] Returns: spec (Tensor): Unfolded spectrogram of shape [B, C, T, F, N], where N: frame_size. r)rr )rr!r,rr.r/rrr spec_unfoldUs zMultiFrameModule.spec_unfold:0yE>Hz>diag_epsepsr c Cstdtt||||S)N...nm,...m->...n)torcheinsuminverse_tik_reg)Rxxrssr4r5rrrsolveaszMultiFrameModule.solvecoefscCtd||S)N...n,...n->...r7r8r&r>rrr apply_coefsgszMultiFrameModule.apply_coefsrF)r2r3)__name__ __module__ __qualname____doc__rr__annotations__boolrrr0r1 staticmethodfloatr=rC __classcell__rrr$rr(s       rncCs4t|dd|ddfd|d}td||S)a^Compute the PSD correlation matrix Rxx for a spectrogram. That is, `X*conj(X)`, where `*` is the outer product. Args: x (complex Tensor): Spectrogram of shape [B, C, T, F]. Will be unfolded with `n` steps over the time axis. Returns: Rxx (complex Tensor): Correlation matrix of shape [B, C, T, F, N, N] rr...n,...m->...mnFr!r,r7r8conj)rrNrrrpsdns" rTr&r>cCr?)a8Deep filter implementation using `torch.einsum`. Requires unfolded spectrogram. Args: spec (complex Tensor): Spectrogram of shape [B, C, T, F, N] coefs (complex Tensor): Coefficients of shape [B, C, N, T, F] Returns: spec (complex Tensor): Spectrogram of shape [B, C, T, F] z...tfn,...ntf->...tfrArBrrrdf~s rUcCs|j\}}}}}}tj||||df|j|jd}|d|djdd|d<|d|d|djdd8<|d|djdd|d<|d|d|djdd7<|S)abDeep filter implementation for real valued input Tensors. Requires unfolded spectrograms. Args: spec (real-valued Tensor): Spectrogram of shape [B, C, N, T, F, 2]. coefs (real-valued Tensor): Coefficients of shape [B, C, N, T, F, 2]. Returns: spec (real-valued Tensor): Filtered Spectrogram of shape [B, C, T, F, 2] r)dtypedevice).rr ).r)rr7emptyrWrXsum)r&r>bc_tfoutrrrdf_reals $$rbc TeZdZUdZeeed<ddedededeffdd Zd e d e fd d Z Z S)DFDeep Filtering.rSrFrrrcst|||||_dS)NrrrSr#rrrrSr$rrrs z DF.__init__r&r>cCs|t|}t|}|dd|j}|j|jdd|jg|jddR}|jr0|}t ||}|j r<| }t ||dd|jddf<|S)NrOrr r).) r1r7view_as_complexnarrowrviewrrrSrUtrainingclone view_as_real)r#r&r>spec_uspec_frrrforwards ( z DF.forwardrD rErFrGrHrrJrIrrrrprMrrr$rrd   rdc rc)DFrealrerSrFrrrcstj|||dd||_dS)NT)rrfrgr$rrrs zDFreal.__init__r&r>cCs|||}|dd|j}|jdd|jgt|jdd}||}|jr+|}t||}||dd|jddf<|S)zPads and unfolds the spectrogram and applies deep filtering using only real valued types. Args: spec (Tensor): Spectrogram of shape [B, C, T, F, 2] coefs (Tensor): Spectrogram of shape [B, C, T, F, 2] rOrr r)N.) r0rirrrr rjrSrb)r#r&r>rnro new_shaperrrrps $  zDFreal.forwardrDrqrrr$rrsrrrscsBeZdZdZd dededeffdd Zd ed efd d ZZS)CRMzComplex ratio mask.rrrrrcs.|dkr|dksJ||ft|ddS)Nrr)rr)r#rrrr$rrrsz CRM.__init__r&r>cCs|d|S)Nr )squeezemul)r#r&r>rrr forward_implszCRM.forward_impl)rr) rErFrGrHrrrrxrMrrr$rrusruc eZdZUdZeeed<eeed<eeed<eeed<eeed<   dd ed ededededef fdd Z ddZ de de de de fddZ Z S)MfWfz&Multi-frame Wiener filter base module.cholesky_decompr9enforce_constraintsr5dloadrFTr2r3rrrc vtj|||d||_||_||_t|j|jd|_t |j|_ |jd|j d<|jd|j d<||_ ||_ dS)aMulti-frame Wiener Filter via an estimate of the inverse Args: num_freqs (int): Number of frequency bins to apply MVDR filtering to. frame_size (int): Frame size of the MF MVDR filter. lookahead (int): Lookahead of the frame. cholesky_decomp (bool): Whether the input is a cholesky decomposition of the correlation matrix. Defauls to `False`. inverse (bool): Whether the input is a normal or inverse correlation matrix. Defaults to `True`. enforce_constraints (bool): Enforce hermetian matrix for non-inverse input and a triangular matrix for cholesky decomposition inpiut. rrrN rrr{r9r|r7 triu_indicesr triu_idcs empty_like tril_idcsr5r} r#rrrr{r9r|r5r}r$rrr z MfWf.__init__cCH|jr|jrdS|jr|jsdS|js|jrdS|js |js"dSdSdS)z3Return an factor f such that Rxx/f in range [-1, 1]g@@g8|Ag-C6*?gh㈵>Nr9r{r#rrr get_r_factor    zMfWf.get_r_factorr&ifciRxxr cCs|t|}t|d|j|jdf}|jr<|jr1d|dddddd|jd|jdf<|| dd }|jrz|j sz|jszdtj |dd d _ |dddddd|jd|jdf }||dddddd|jd|jdf<t|d|jdf}|d d|j}|j st||j|j}tj||d}n td ||d}|||}|jr|}t||d d|jddf<|S) aMulti-frame Wiener filter based on Rxx**-1 and speech IFC vector. Args: spec (complex Tensor): Spectrogram of shape [B, 1, T, F] ifc (complex Tensor): Inter-frame speech correlation vector [B, T, F, N*2] iRxx (complex Tensor): (Inverse) noisy covariance matrix Rxx**-1 [B, T, F, (N**2)*2] OR cholesky_decomp Rxx=L*L^H of the same shape. Returns: spec (complex Tensor): Filtered spectrogram of shape [B, C, T, F] r*r)rNrrr+r rOdim1dim2r6.)r1r7rh unflattenrr{r|rmatmul transposerSr9diagonalimagrrirr:r}r5linalgr=r.r8rCrkrlrm)r#r&rrrn tril_conjrowrrrrps* *.* z MfWf.forwardrFTTr2r3rErFrGrHrrJrIrLrrrrrprMrrr$rrz8       " rzc ry)MfMvdrz^Multi-frame minimum variance distortionless beamformer based on Rnn**-1 and speech IFC vector.r{r9r|r5r}rFTr2r3rrrc r~)aMulti-frame minimum variance distortionless beamformer. Args: num_freqs (int): Number of frequency bins to apply MVDR filtering to. frame_size (int): Frame size of the MF MVDR filter. lookahead (int): Lookahead of the frame. cholesky_decomp (bool): Whether the input is a cholesky decomposition of the correlation matrix. Defauls to `False`. inverse (bool): Whether the input is a normal or inverse correlation matrix. Defaults to `True`. enforce_constraints (bool): Enforce hermetian matrix for non-inverse input and a triangular matrix for cholesky decomposition inpiut. rrrNrrr$rrrFrzMfMvdr.__init__cCr)z3Return an factor f such that Rnn/f in range [-1, 1]g@gAg-C6 ?gư>NrrrrrrfrzMfMvdr.get_r_factorr&riRnnr c Cs|t|}t|d|j|jdf}|jr<|jr1d|dddddd|jd|jdf<|| dd }|jrz|j sz|jszdtj |dd d _ |dddddd|jd|jdf }||dddddd|jd|jdf<t|d|jdf}|d d|j}|j st||j|j}tj||}ntd ||}td | |}|d  } || |jd|jd} ||| }|jr|}t||dd|jddf<|S)aMulti-frame MVDR filter based on Rnn**-1 and speech IFC vector. Args: spec (complex Tensor): Spectrogram of shape [B, C, T, F] ifc (complex Tensor): Inter-frame speech correlation vector [B, C*N*2, T, F] iRnn (complex Tensor): (Inverse) noise covariance matrix Rnn**-1 [B, T, F (N**2)*2] OR cholesky_decomp Rnn=L*L^H of the same shape. Returns: spec (complex Tensor): Filtered spectrogram of shape [B, C, T, F] r*r)rNrrr+r rOrr6r@).r N.)r1r7rhrrr{r|rrrrSr9rrrrirr:r}r5rr=r8rr.rCrkrlrm) r#r&rrrnrro numerator denumeratorscalerrrrrpqs0 *.*   zMfMvdr.forwardrrrrr$rr=rrrOr inputrrcCsL|jdks Jd|j||j|ksJdtj|d||d}|jddS)aCompute the trace of a Tensor along ``dim1`` and ``dim2`` dimensions. Args: input (torch.Tensor): Tensor of dimension `(..., channel, channel)` dim1 (int, optional): the first dimension of the diagonal matrix (Default: -1) dim2 (int, optional): the second dimension of the diagonal matrix (Default: -2) Returns: Tensor: trace of the input Tensor r)z/The dimension of the tensor must be at least 2.z3The size of ``dim1`` and ``dim2`` must be the same.rrr rY)ndimrr7rr[)rrrrrr_compute_mat_traces   rr3r2matregr5cCsZ|d}tj||j|jd}t|jd|}||}|||dddddf}|S)aPerform Tikhonov regularization (only modifying real part). Args: mat (torch.Tensor): input matrix (..., channel, channel) reg (float, optional): regularization factor (Default: 1e-8) eps (float, optional): a value to avoid the correlation matrix is all-zero (Default: ``1e-8``) Returns: Tensor: regularized matrix (..., channel, channel) r rV).NN.N)sizer7eyerWrXrr)rrr5Crepsilonrrrr:s r:XNcCs8t|dd|ddfd|d}td||}|S)NrrrPrQ)rrXwr;rrr compute_corrs"rTFc!sddlm}m}ddl}ddlm}ddlm} m} ddl m } t j dddd } d } |r4|r4d } d } |d |_ |jd d| }d|_d|_d|_|jdd}|j|j|j|j|jd| d|jd|jddjdd d}| d|jd|jddjdd d}||}| d||jt||||d}fdd|||fD\}}}t|ddddfdd}fdd|||fD\}}}|d}t|| | }|r||}nt|| | }|}|rt j|}|rt j|\}}tdt |dkdd|| |rt !d ||}t !d!||}nt "|t #|$dt #|%d"t #|%d"&d}'|(} | d#)t*|t*|| |jdS)$Nricinstallconfig load_audio save_audio ModelParams linewidthr(r3r2-C6?h㈵>T allow_reload`]r)rsrfft_sizehop_sizenb_bands assets/clean_freesound_33711.wav num_frameskeepdimassets/noise_freesound_2530.wav out/noisy.wavr{r9r|c g|] }t|qSrr7 from_numpyanalysisnumpy.0rrUrr  z$compute_ideal_wf..cg|]}t|qSrr)rAORDERrrr.r (Number of errors during cholesky_decomp:r6...fn,...fn->...fr*zout/ideal_mfwf_c{}_i{}.wav)+icecreamrrlibdf df.configrdf.iorrdf.modelrr7set_printoptionsincludeContext use_defaultsrrrrdnb_erbmeanrzrRr!r,r:rinv cholesky_exprintwherer[absr8rhrmr.flattenrv synthesisrformatr)!rxx_via_rssrnnr{r9r|manualrrrrrrrDLOADEPSpn_freqssrNrwfrSrrRssRnnr;rrinforYyrrrUrcompute_ideal_wfs    "    "rc$sddlm}m}ddl}ddlm}ddlm}m} ddl m } d|_ t j ddd d } d } |r7|r7d } d } ||jdd| } d| _d| _d| _| jdd}|j| j| j| j| jd|d| jd | jddjddd}|d| jd | jddjddd}||}| d|| jt||||d}fdd|||fD\}}}t|ddddfdd}fdd||fD\}}|d}t j|\}}|d}t|| | }|rt j|}|rt j|\}}tdt |dkdd ||!|rM|d}||"d | }|r!|#|$%d d!}|r,t &d"||}nt j'||}t &d#|$|} || "d | }!t &d$||!}"nt (|t )|"dt )|*d%t )|*d%+d}",|"-}#| d&.t/|t/||#| jdS)'NrrrrrTxrr(r3r2rrrrrrr)rrrrrrrrcrrrrrrrrKrz&compute_ideal_mvdr..crrrrrrrrMrrrr rOr6r@rr*zout/ideal_mfmvdr_c{}_i{}.wav)0rrrrrrrrrrrrr7rrrrrrdrrrrRr!r,reighr:rrrrr[rr.rrSrr8r=rhrmrrvrrrr)$r{r9r|rrrrrrrrrrrrrrNrmvdrrrrrrrrr^vrrifc0numdenumrrrrrrcompute_ideal_mvdrs    "  "r cCs4ttfD]}dD]}dD] }||||dq qqdS)N)TFr)rr )r|mr]irrrcompute_all_mfws r)rr)rOr )r3r2)TFTTF)FTTF)T)typingrrr7torch.nn.functionalr functionalrRrrrModulerrTrUrbrdrsrurzrrrLr:rrr rrrrrs*  F  ` d  QY