o oiN4@sddlZddlmZddlZddlmZmZddlZddlm Z m Z m Z m Z m Z mZmZddlmZdZGdddejZd d Zd d ZdS) N)nn)unfoldpad)FSN_FRAMENUMBANDMINFREQNBETA DYN_RANGE)thirdoctg:0yE>c s|eZdZdZ   ddedededeffdd Zd ejd ejd ejfd dZ e ddZ e dddZ e ddZ ZS) NegSTOILossaNegated Short Term Objective Intelligibility (STOI) metric, to be used as a loss function. Inspired from [1, 2, 3] but not exactly the same due to a different resampling technique. Use pystoi when evaluating your system. Args: sample_rate (int): sample rate of audio input use_vad (bool): Whether to use simple VAD (see Notes) extended (bool): Whether to compute extended version [3]. do_resample (bool): Whether to resample audio input to `FS` Shapes: (time,) --> (1, ) (batch, time) --> (batch, ) (batch, n_src, time) --> (batch, n_src) Returns: torch.Tensor of shape (batch, *, ), only the time dimension has been reduced. Warnings: This function does not exactly match the "real" STOI metric due to a different resampling technique. Use pystoi when evaluating your system. Notes: `use_vad` can be set to `False` to skip the VAD for efficiency. However results can become substantially different compared to the "real" STOI. When `True` (default), results are very close but still slightly different due to a different resampling technique. Compared against mpariente/pystoi@84b1bd8. References [1] C.H.Taal, R.C.Hendriks, R.Heusdens, J.Jensen 'A Short-Time Objective Intelligibility Measure for Time-Frequency Weighted Noisy Speech', ICASSP 2010, Texas, Dallas. [2] C.H.Taal, R.C.Hendriks, R.Heusdens, J.Jensen 'An Algorithm for Intelligibility Prediction of Time-Frequency Weighted Noisy Speech', IEEE Transactions on Audio, Speech, and Language Processing, 2011. [3] Jesper Jensen and Cees H. Taal, 'An Algorithm for Predicting the Intelligibility of Speech Masked by Modulated Noise Maskers', IEEE Transactions on Audio, Speech and Language Processing, 2016. TF sample_rateuse_vadextended do_resamplecst||_||_||_t|_t|_t |_ ||_ |j r*t }t jj|jt dd|_t|t |_d|j|_tt|jddd}tj|dd|_t||jttd}tjt|dd|_dS) Nsinc_interpolation) orig_freqnew_freqresampling_methodF) requires_gradr) super__init__rrrr intel_framesr betar dyn_rangerr torchaudio transformsResampleresamplerwin_lennffttorch from_numpynphanningfloatr Parameterwinr rrOBM)selfrrrrr+obm_mat __class__M/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/torch_stoi/stoi.pyr9s*  "zNegSTOILoss.__init__ est_targetstargetsreturnc CsR|j|jkrtd|j|j|jdkr!||d|ddS|jdkr=|j^}}||d||d||S|jd}|jrT|jtkrT| |}| |}|j r{| |||j |j |j|jd\}}}|jddd\}}|d ddf}|j||j |jdd }|j||j |jdd } |j r||d9}| |d9} t|j|dt} t|j| dt} t| dd|jfd d || jd|jd} t| dd|jfd d || jd|jd} |j r|d |jddf}| |dd9} | |dd9} |jr,|| }|| }||}|d}n[| jdddd | jdddd t}| |}d|j d}t!|| d|}||j"ddd}| | j"ddd} ||jdddd t}| | jdddd t} || }|d}|"d}|j r|d|dt} | S|"d} | S)aCompute negative (E)STOI loss. Args: est_targets (torch.Tensor): Tensor containing target estimates. targets (torch.Tensor): Tensor containing clean targets. Shapes: (time,) --> (1, ) (batch, time) --> (batch, ) (batch, n_src, time) --> (batch, n_src) Returns: torch.Tensor, the batch of negative STOI loss zCtargets and est_targets should have the same shape, found {} and {}rNrrrT) descending.)overlap)rr kernel_sizestridepdimkeepdim r>)#shape RuntimeErrorformatndimforwardviewrrrr"rremove_silent_framesrr+r#sortstftr$ unsqueezer%matmulr,abspowEPSsqrtrrr rowcol_normsumnormrminmean)r-r3r4innerwav_len batch_sizemask_x_specy_specx_toby_tobx_segy_segx_ny_n corr_comp norm_const y_seg_normedclip_valy_primoutputr1r1r2rFYs                     zNegSTOILoss.forwardc Cs,t|ddddddfd|fd|fd}t|ddddddfd|fd|fd}||dddf9}||dddf9}dttj|dddt}|jddd||d k} | d} |d dd}|d dd}t|| }t|| }t ||} t ||} |d dd}|d dd}| | | fS) aDetects silent frames on input tensor. A frame is excluded if its energy is lower than max(energy) - dyn_range Args: x (torch.Tensor): batch of original speech wav file (batch, time) dyn_range : Energy range to determine which frame is silent framelen : Window size for energy evaluation hop : Hop size for energy evaluation Returns: torch.BoolTensor, framewise mask. Nrr8r@T)r=r>rrAr) rr%log10rSrOamaxsqueezepermute _mask_audio_overlap_and_addlong) xyrwindowframelenhopx_framesy_frames x_energiesrYx_sily_silr1r1r2rHs(""     z NegSTOILoss.remove_silent_framescCsn|jd}t||}t|ddddddfd|fd|fddddf}tjj||dddf|ddS)a*We can't use torch.stft: - It's buggy with center=False as it discards the last frame - It pads the frame left and right before taking the fft instead of padding right Instead we unfold and take rfft. This gives the same result as pystoi.utils.stft. rNrr8.r)nr=)rBintrr%fftrfft)rpr+fft_sizer7r#rtframesr1r1r2rJs ( "zNegSTOILoss.stftcCs8dD]}||j|dd}||jd|ddt}q|S)z4Mean/variance normalize axis 2 and 1 of input vector)rrTrArr;)rUrSrO)rpr=r1r1r2rQ$szNegSTOILoss.rowcol_norm)TFT)rz)__name__ __module__ __qualname____doc__r|boolrr%TensorrF staticmethodrHrJrQ __classcell__r1r1r/r2r s8.   * r cCs|j\}}}| | }t|d|||d|f}||||||f}|dddd}||d|f}|ddd| f}|||||d|f}|jdd}|d||}||dfd|}|S)Nrrrr)axis)rBrreshaperlrR)rurtrX num_framesrssegmentssignalendr1r1r2rn-s   rncCstddt||D}|S)Nc Ss2g|]\}}t||dddt||fqS)r)rlenrR).0ximir1r1r2 Ks2z_mask_audio..)r%stackzip)rprY masked_audior1r1r2rmIsrm)r%rnumpyr'torch.nn.functionalrrr pystoi.stoirrrrr r r pystoi.utilsr rOModuler rnrmr1r1r1r2s $ "