o Si1D @sVddlZddlZddlZddlmZmZmZmZmZm Z m Z ddl Z ddl Z ddlmZmZddlmZddlmZgdZGddde jjZGd d d e jjZGd d d e jjZd e jdedededede jf ddZd e jdede jfddZe dZ deeee fdede fddZ!d$de jdede jfd d!Z"Gd"d#d#e jjZ#dS)%N)AnyDictOptionalSequenceTupleTypeVarUnion)CutSetFeatureExtractor)dereverb_wpe_torch)Pathlike) GlobalMVN SpecAugmentRandomizedSmoothing DereverbWPEc seZdZdZdeffdd Ze  ddedeedee d dfd d Z ed e d dfd dZ d e fddZ ddejdeejd ejfddZdejd ejfddZZS)r z,Apply global mean and variance normalization feature_dimcs8t||_|dt||dt|dS)N norm_means norm_stds)super__init__rregister_buffertorchzerosones)selfr __class__T/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/dataset/signal_transforms.pyrs zGlobalMVN.__init__Ncutsmax_cuts extractorreturncCsB|j||d}dd|D}|dj\}||}|||S)N)r r!cSsi|] \}}|t|qSr)r as_tensor).0namevaluerrr #sz'GlobalMVN.from_cuts..r)compute_global_feature_statsitemsshapeload_state_dict)clsrr r!statsr global_mvnrrr from_cutss  zGlobalMVN.from_cuts stats_filecCs,t|}|dj\}||}|||S)Nr)rloadr*r+)r,r0r-rr.rrr from_file)s   zGlobalMVN.from_filecCst||dSN)rsave state_dict)rr0rrrto_file1szGlobalMVN.to_filefeaturessupervision_segmentscCs||j|jSr3)rr)rr7r8rrrforward4szGlobalMVN.forwardcCs||j|jSr3)rr)rr7rrrinverse;szGlobalMVN.inverse)NNr3)__name__ __module__ __qualname____doc__intr classmethodr rr r/r r2r6rTensor IntTensorr9r: __classcell__rrrrr s6  r cs`eZdZdZ   ddeeeeeeffde deffdd Z d e j d e j fd d Z ZS)rz Randomized smoothing - gaussian noise added to an input waveform, or a batch of waveforms. The summed audio is clipped to ``[-1.0, 1.0]`` before returning. 皙?T333333?sigma sample_sigmapcs&t||_||_||_d|_dS)a RandomizedSmoothing's constructor. :param sigma: standard deviation of the gaussian noise. Either a constant float, or a schedule, i.e. a list of tuples that specify which value to use from which step. For example, ``[(0, 0.01), (1000, 0.1)]`` means that from steps 0-999, the sigma value will be 0.01, and from step 1000 onwards, it will be 0.1. :param sample_sigma: when ``False``, then sigma is used as the standard deviation in each forward step. When ``True``, the standard deviation is sampled from a uniform distribution of ``[-sigma, sigma]`` for each forward step. :param p: the probability of applying this transform. rN)rrrFrGrHstep)rrFrGrHrrrrEs  zRandomizedSmoothing.__init__audior"cOst|jtr |j}nt|j|j}|jd7_|jr:|jdftdd|jddD}|dt |d}|t |}t |d|j d}||}tj ||ddd S) Nrcs|]}dVqdSrKNrr$_rrr kz.RandomizedSmoothing.forward..g?)rHg)minmax) isinstancerFfloatschedule_value_for_steprIrGr*tuplerrand randn_likerandom_mask_along_batch_axisrHclip)rrJargskwargsrF mask_shapenoise noise_maskrrrr9]s (zRandomizedSmoothing.forward)rDTrE)r;r<r=r>rrVrrr?boolrrrAr9rCrrrrr?src seZdZdZ       d!d eed ed ed ed edef fdd Z d"dej deej dej fddZ d#dej de de dej fddZ deeeffddZdeeeffdd ZZS)$ra SpecAugment performs three augmentations: - time warping of the feature matrix - masking of ranges of features (frequency bands) - masking of ranges of frames (time) The current implementation works with batches, but processes each example separately in a loop rather than simultaneously to achieve different augmentation parameters for each example. PrR d333333??time_warp_factornum_feature_masksfeatures_mask_sizenum_frame_masksframes_mask_sizemax_frames_mask_fractioncstd|krdksJJ|dksJ|dksJ|dks%J|dks+J||_||_||_||_||_||_||_dS)a SpecAugment's constructor. :param time_warp_factor: parameter for the time warping; larger values mean more warping. Set to ``None``, or less than ``1``, to disable. :param num_feature_masks: how many feature masks should be applied. Set to ``0`` to disable. :param features_mask_size: the width of the feature mask (expressed in the number of masked feature bins). This is the ``F`` parameter from the SpecAugment paper. :param num_frame_masks: the number of masking regions for utterances. Set to ``0`` to disable. :param frames_mask_size: the width of the frame (temporal) masks (expressed in the number of masked frames). This is the ``T`` parameter from the SpecAugment paper. :param max_frames_mask_fraction: limits the size of the frame (temporal) mask to this value times the length of the utterance (or supervision segment). This is the parameter denoted by ``p`` in the SpecAugment paper. :param p: the probability of applying this transform. It is different from ``p`` in the SpecAugment paper! rrKN) rrrirjrkrlrmrnrH)rrirjrkrlrmrnrHrrrrs      zSpecAugment.__init__Nr7r8r"c Ost|jdks Jd|}|dur(t|dD] }|||||<q|S|D]\}}}||}|j||||fddd||||f<q*t|dD]}|j||ddd||<qO|S)a Computes SpecAugment for a batch of feature matrices. Since the batch will usually already be padded, the user can optionally provide a ``supervision_segments`` tensor that will be used to apply SpecAugment only to selected areas of the input. The format of this input is described below. :param features: a batch of feature matrices with shape ``(B, T, F)``. :param supervision_segments: an int tensor of shape ``(S, 3)``. ``S`` is the number of supervision segments that exist in ``features`` -- there may be either less or more than the batch size. The second dimension encoder three kinds of information: the sequence index of the corresponding feature matrix in `features`, the start frame index, and the number of frames for each segment. :return: an augmented tensor of shape ``(B, T, F)``. zESpecAugment only supports batches of single-channel feature matrices.NrTF)warpmask)lenr*clonerangesize_forward_single) rr7r8r]r^ sequence_idx start_frame num_frames end_framerrrr9s$  zSpecAugment.forwardTrprqcCst|jkr |S|r|jdur|jdkrt||jd}|rR|}t||j|j|dd}|j| d}t |j t ||j}t |j||}t||||dd}|S)zO Apply SpecAugment to a single feature matrix of shape (T, F). NrK)factorrR) mask_size mask_times mask_valueaxisr)randomrHri time_warpmeanmask_along_axis_optimizedrkrjrnrurSrlmathceilrm)rr7rprqrmax_tot_mask_framesrlmax_mask_framesrrrrvs< zSpecAugment._forward_singlec Ks$t|j|j|j|j|j|j|jdS)N)rirjrkrlrmrnrH)dictrirjrkrlrmrnrH)rr^rrrr5szSpecAugment.state_dictr5cCst|d|j|_|d|j|_|d|j|_|d|j|_|d|j|_|d|j|_|d|j|_dS)NrirjrkrlrmrnrH)getrirjrkrlrmrnrH)rr5rrrr+s"zSpecAugment.load_state_dict)rcrRrdrerfrgrhr3)TT)r;r<r=r>rr?rVrrrArBr9rbrvrstrrr5r+rCrrrrrysT , 0 ) rr7r|r}r~rr"c CsN|dvrtd|d}|dgt|dd}ttdt|d|f}td||||}| }| |  }|dkrs|dkr^||dd||f<| dSt ||D]\} } ||dd| | f<qcn-|dkr||dddd||f<| dSt ||D]\} } ||dddd| | f<q| d}|S)a Apply Frequency and Time masking along axis. Frequency and Time masking as described in the SpecAugment paper. :param features: input tensor of shape ``(T, F)`` :mask_size: the width size for masking. :mask_times: the number of masking regions. :mask_value: Value to assign to the masked regions. :axis: Axis to apply masking on (1 -> time, 2 -> frequency) )rKrRz.Only Frequency and Time masking are supported!rNrK) ValueError unsqueezereshapelistrurrandintr?rYlongsqueezezip) r7r|r}r~rvalues min_values mask_starts mask_ends mask_startmask_endrrrr%s,      rr{cCs|d}|||dkr|Stj|d||}tj||||d}||kr-|S|dd}tjjj|ddddd|ddf||dfddd}tjjj|dddd|dddf|||dfddd}tj ||fdd  d dS) aT Time warping as described in the SpecAugment paper. Implementation based on Espresso: https://github.com/freewym/espresso/blob/master/espresso/tools/specaug_interpolate.py#L51 :param features: input tensor of shape ``(T, F)`` :param factor: time warping parameter. :return: a warped tensor of shape ``(T, F)`` rrKNrobicubicF)rumode align_cornersrRdim) runprrrrnn functional interpolatecatr)r7r{tcenterwarpedleftrightrrrrRs*  rTschedulerIcCsPt|\}}|d|ksJd|d|d|ddt||d}||S)Nrz.Cannot determine the scheduled value for step z with schedule: zG. Did you forget to add the first part of the schedule for steps below ?rK)rbisect bisect_right)rrI milestonesridxrrrrWvs rW?tensorrHcCsB|jdftdd|jddD}t||ktj}|S)a For a given tensor with shape (N, d1, d2, d3, ...), returns a mask with shape (N, 1, 1, 1, ...), that randomly masks the samples in a batch. E.g. for a 2D input matrix it looks like: >>> [[0., 0., 0., ...], ... [1., 1., 1., ...], ... [0., 0., 0., ...]] :param tensor: the input tensor. :param p: the probability of masking an element. rcsrLrMrrNrrrrPrQz/random_mask_along_batch_axis..rKN)r*rXrrYtofloat32)rrHr_rqrrrr[s(r[csBeZdZdZd dedeffdd Zdejd ejfd d ZZ S) ra@ Dereverberation with Weighted Prediction Error (WPE). The implementation and default values are borrowed from `nara_wpe` package: https://github.com/fgnt/nara_wpe The method and library are described in the following paper: https://groups.uni-paderborn.de/nt/pubs/2018/ITG_2018_Drude_Paper.pdf n_fft hop_lengthcst||_||_dSr3)rrrr)rrrrrrrs  zDereverbWPE.__init__rJr"csP|jdkrtjfdd|DddS|jdksJtjfdd|DddS)a Expects audio to be 2D or 3D tensor. 2D means a batch of single-channel audio, shape (B, T). 3D means a batch of multi-channel audio, shape (B, D, T). B => batch size; D => number of channels; T => number of audio samples. rRcs$g|]}t|djjdqS)rrr)r rrrr$arrr s z'DereverbWPE.forward..rrrocsg|] }t|jjdqS)r)r rrrrrrrs)ndimrrstack)rrJr]r^rrrr9s    zDereverbWPE.forward)rr) r;r<r=r>r?rrrAr9rCrrrrrs r)r)$rrrtypingrrrrrrrnumpyrrlhotser r lhotse.augmentationr lhotse.utilsr __all__rModuler rrrAr?rVrrrrWr[rrrrrs@$  /:- -!"