o }oiP!@sddlZddlZddlZddlZddlZddlZddlmZddlm Z ddlm Z m Z ddl m Z ddlmZmZmZmZmZddlmZddlZddlmZddlZddlZddlZddl Z ddl!m"Z"m#Z#ddl$m%Z%m&Z&dd l'm(Z(dd l)m*Z*dd l+m,Z,dd l-m-Z-dd l.m/Z/m0Z0ddl1m2Z2ddl3m4Z4 eGdddZ5dde6de5fddZ7de8de6fddZ9ddZ:de8de8de;fddZe6ffd#d$Z? dd%e6d&e6d'e@d(e@d)e@d*eAd+e6de6fd,d-ZBd.d/ZCejDjEd0ej>d1ee6e@fd&e6dej>fd2d3ZFd4e6d1e8de6fd5d6ZGejDjEd7ej>dej>fd8d9ZHejDjEd7ej>d:e@dej>fd;d<ZIdej>d=eAde@fd>d?ZJ dd@e6dAe@dBe@dCej>dee@e@ff dDdEZKejDjEdCej>d1ee6e@fdej>fdFdGZLejDjEdHej>dIej>dej>fdJdKZMejDjEd7ej>dej>fdLdMZNejDjEdNej>d1ee6e@fdej>fdOdPZOdCej>d1e8dee6e8ffdQdRZPejDjEdCej>d1ee6e@fdej>fdSdTZQdUe6d1e8de6fdVdWZR  XddYe6dZe8d[e@d*eAd+e6d\eSde6fd]d^ZTd_d`ZUdae6dbe6dee%e%ffdcddZVdee8de;fdfdgZW h 0 i j kddee8dle6dme6dne6doe6dpe6d[e@d*eAdee8e8ffdqdrZXdee8deSfdsdtZYddle6dme6doe6deeZe8e8ffdudvZ[    w    j x yddze6d{ee6d|ee6d}ee6d~eAdBe@de@d:e@d1e8de@deAdeAdej\fddZ] jddNej>de@d)e@dej^fddZ_de6de;de;fddZ` Xdd(e@d)e@de6d+e6deSde6f ddZade6fddZbde6fddZc N dde6de6de6de6fddZd dde6de6de6de6fddZed"e6dejffddZgdeee@deee@fddZhde6deee@fddZide6deeee@eee@ffddZj ddNeee@dee@deee@fddZk dd7eee@de@dBe@de@deSde6f ddZl  jdde6de6dee@de6deSdBe@de@fddZmdddZndde6dee6de%fddZoddee@d[e@fddZp jddee6ee@fdee6ee@fd[e@dee%e%ffddńZqde"fddȄZrde6de8de8de8d[e@f dd΄Zs Xddej>de#deAdeSfddՄZt X ddeej>dee6ee6ee@eAfffde#deAdeSdeAdeee@fddۄZudS)N) dataclass)repeat)ceilfloor)Path)DictListOptionalTupleUnion) DictConfig OmegaConf) AnnotationSegment) detection) roc_auc_score) ParameterGrid)tqdm)EncDecClassificationModelEncDecFrameClassificationModel) get_full_path)loggingc@sZeZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<dZ eed <d S) PostProcessingParamsaH Postprocessing parameters for end-to-end speaker diarization models. These parameters can significantly affect DER performance depending on the evaluation style and the dataset. It is recommended to tune these parameters based on the evaluation style and the dataset to achieve the desired DER performance. ?onsetoffset pad_onset pad_offsetmin_duration_onmin_duration_offN) __name__ __module__ __qualname____doc__rfloat__annotations__rrrrr r'r'^/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/utils/vad_utils.pyr1s      rpostprocessing_yamlreturncCstt}|durtd|St|d-}t|d}td|d|D]\}}t ||r:t |||q+Wd|S1sFwY|S)a% Load postprocessing parameters from a YAML file. Args: postprocessing_yaml (str): Path to a YAML file for postprocessing configurations. Returns: postprocessing_params (dataclass): Postprocessing parameters loaded from the YAML file. NzeNo postprocessing YAML file has been provided. Default postprocessing configurations will be applied.r parameterszPostprocessing YAML file 'z' has been loaded.) r structuredrrinfoopenyaml safe_loaditemshasattrsetattr)r)postprocessing_paramsfile yaml_paramskeyvaluer'r'r(load_postprocessing_from_yamlBs$     r:configc sd|vr |dr |d}nd}d|vrtj|d|n|}t|dtkrSg}t|dddd}|D] }|t | q6Wdn1sMwYnt|dt kr`|d}nt d d |d |d t|dtkrzt |djnd d|ddur|ddkrtj|dd}t|t}t t|t|t|ddd}Wdn1swYnfddt|dddD}tj|rtdt|t|ddd"} |D]} | D]} t| | | d| qqWd|S1swY|S)z Perform VAD on long audio snippet might cause CUDA out of memory issue. Automatically split manifest entry by split_duration to avoid the potential memory issue. prepared_manifest_vad_inputzmanifest_vad_input.jsonout_dirinputr+utf-8encodingNzThe input for manifest preparation would either be a string of the filepath to manifest or a list of {'audio_filepath': i, 'offset': 0, 'duration': null}.infersplit_durationwindow_length_in_sec)labelrCrD manifest_dir num_workers processeszsplitting manifestTtotaldescleavecsg|]}t|qSr'write_vad_infer_manifest).0input_el args_funcr'r( sz$prepare_manifest..rNrOz/The prepared manifest file exists. Overwriting!a )ospathjointypestrr/ readlinesappendjsonloadsstriplist ValueErrorrparentgetmultiprocessingPoolziprrimapwrite_vad_infer_manifest_starlenexistsrr.removedumpwriteflush) r;manifest_vad_input default_path input_listmanifestlinepinputsresultsfoutresr+r'rTr(prepare_manifest`sj             r}cCt|Sz? A workaround for tqdm with starmap of multiprocessing rPargsr'r'r(rlrlr6rUc Csg}|d}|d}|d}|d}|dd}|dd}t|s6t|d |} | r6| }z{d } tj|| ||d \} } tj| | d } | }|}d }|dkr||krq|d krb|}d}n d}||}||8}|}d}n&|dksy|dkr|d}nd}|dkr|}|}n ||}||8}||}||8}|||d|d}||||7}|dksUW|SW|St y}z2d}t |ddd}| |dt |Wdn1swYWYd}~|SWYd}~|Sd}~ww)a! Used by prepare_manifest. Given a list of files, split them with maximum split_duration and write them to the manifest. Args: files (dict) : file to be processed args_func: label (str): label for audio snippet.y split_duration (float): max duration of each audio clip (each line in json) window_length_in_sec (float) : length of window for generating the frame. Used for taking care of joint. Returns: res (list) : list of generated metadata line of json for file rFrCrDaudio_filepathdurationNrrrG>)srrryrsingleendstartnext_)rrrFtextrz error.logwr?r@:) rgris_fileabsoluteas_posixlibrosaload get_durationr` Exceptionr/rqr^)r6rUr|rFrCrDfilepath in_duration in_offset new_filepathrx_srrleftcurrent_offsetstatuswrite_duration offset_incmetadataeerr_filer{r'r'r(rQsx       **& rQdatacCs0t|dkr dgSdgt|}tt|D]}|dkr-||||dkr(dnd||<q|t|dkrF||||dkrAdnd||<q||||dkr_||||dkr_d||<q||||dkrx||||dkrxd||<q||||dkr||||dkrd||<qd||<q|S)a/ Generate a list of status for each snippet in manifest. A snippet should be in single, start, next or end status. Used for concatenating to full audio file. Args: data (list): list of filepath of audio snippet Returns: status (list): list of status of each snippet. rIrNrrrr)rmrange)rrir'r'r(get_vad_stream_statuss ""( ( (  rrcCsfg}t|ddd}|D] }|t|qWdn1s"wYt|j}t||fS)z2 Load torch.Tensor and the name from file r+r?r@N)r/r_r`r%rstemtorchtensor)rframefrwnamer'r'r(load_tensor_from_files  rframe_pred_dirsmoothing_methodoverlaprDshift_length_in_secrHr=cCst|d}|r |}ntj|d|dt|}tj|s&t||||||d} |durf|dkrftj|d } t |t | } t t | t| t|dd d } Wd|S1s_wY|St |dd d D]} t| | qm|S) a Generate predictions with overlapping input windows/segments. Then a smoothing filter is applied to decide the label for a frame spanned by multiple windows. Two common smoothing filters are supported: majority vote (median) and average (mean). This function uses multiprocessing to speed up. Args: frame_pred_dir (str): Directory of frame prediction file to be processed. smoothing_method (str): median or mean smoothing filter. overlap (float): amounts of overlap of adjacent windows. window_length_in_sec (float): length of window for generating the frame. shift_length_in_sec (float): amount of shift of window for generating the frame. out_dir (str): directory of generated predictions. num_workers(float): number of process for multiprocessing Returns: overlap_out_dir(str): directory of the generated predictions. z/*.frameoverlap_smoothing_output_r)rrDrr=rNrIrJzgenerating predsTrLFrW)globrZr[r\r^rnmkdirrhrirjrrdrrk&generate_overlap_vad_seq_per_file_starrm!generate_overlap_vad_seq_per_file)rrrrDrrHr=frame_filepathlistoverlap_out_dirper_argsrxryrzframe_filepathr'r'r(generate_overlap_vad_seq s@     rcCr~r)rrr'r'r(r_rrrrcCs|d}|d}|d}|dd}t||}t||d}t|d|} t| |} | dkr:td| dtt||} |d krt| } t| } t|D]+\}}|| d kr_qT||}||}| |||| ||<| ||d| ||<qT| | } | | d kd }|| | d k<| S|d krd dt| D} t|D]0\}}|| d krq||}||}t||D]}|| dkrt| || d fd | |<qqt dd| D} t | }| |d }|| |<| Std)a Use generated frame prediction (generated by shifting window of shift_length_in_sec (10ms)) to generate prediction with overlapping input window/segments. See description in generate_overlap_vad_seq. Use this for single instance pipeline. rrDr frame_len{Gz?rIzrNote we jump over frame sequence to generate overlapping input segments. Your input makes jump_on_frame=a < 1 which is invalid because it cannot jump and will stuck. Please try different window_length_in_sec, shift_length_in_sec and overlap choices. jump_on_target = int(seg * (1 - overlap)) jump_on_frame = int(jump_on_frame/shift) meanrmediancSsg|]}tdqS)r)remptyrRrr'r'r(rVsz7generate_overlap_vad_seq_per_tensor..cSsg|] }tj|ddqS)r)q)r nanquantile)rRlr'r'r(rVsz0smoothing_method should be either mean or median) rgintrermrzeros enumeratercat unsqueezestackisnan)rrrrrDrrshiftsegjump_on_target jump_on_frame target_lenpreds pred_countrog_predrrlast_non_zero_predjnan_idxlast_non_nan_predr'r'r(#generate_overlap_vad_seq_per_tensorfs^           rrc Cs|d}|d}t|\}}i}|D]}t||tks$t||tkr*||||<qt|||}tj||d|} t| ddd} |D] } | | ddqFWd | S1s]wY| S) z< A wrapper for generate_overlap_vad_seq_per_tensor. r=r.rr?r@.4frYN) rr]r%rrrZr[r\r/rq) rrr=rrrper_args_floatrroverlap_filepathrpredr'r'r(rs$     rsegmentscCs|jtdgks|jtddgks|jtddgkr|S||dddfd}|dddf|dddfk}tjjj|ddgddd}||df}tjjj|ddgddd}||df}tj||fdd }|S) z Merged the given overlapped segments. For example: torch.Tensor([[0, 1.5], [1, 3.5]]) -> torch.Tensor([0, 3.5]) rrINrconstantr)moder9dim)shaperSizesortnn functionalpadr)rmerge_boundary head_paddedhead tail_paddedtailmergedr'r'r(merge_overlap_segments r thresholdcCs(||dddf|dddf|kS)z Remove segments which duration is smaller than a threshold. For example, torch.Tensor([[0, 1.5], [1, 3.5], [4, 7]]) and threshold = 2.0 -> torch.Tensor([[1, 3.5], [4, 7]]) NrIrr')rrr'r'r(filter_short_segmentss( rperccCs.t|}tt|tt||ddS)z) Calculate percentile given data drI)rmr%sortedrmathr)rrsizer'r'r( percentiles&rscalerrsequencecCsx|dkr d}d}n|dkrt|}t|}n|dkr$t|d}t|d}||||}||||}t|t|fS)zE Calculate onset and offset threshold given different scale. rrrIrelativerc)minmaxrr%)rrrrminimaxir'r'r(cal_vad_onset_offsets   rc Cs2|dd}|dd}|dd}|dd}|dd}d }d}d } td } td t|D]E} |ri|| |krh| ||td ||krbttd ||| ||gd } t| | fd } | |}d }q0|| |kru| |}d }q0|rttd ||| ||gd } t| | fd } t | } | S) a Binarize predictions to speech and non-speech Reference Paper: Gregory Gelly and Jean-Luc Gauvain. "Minimum Word Error Training of RNN-based Voice Activity Detection", InterSpeech 2015. Implementation: https://github.com/pyannote/pyannote-audio/blob/master/pyannote/audio/utils/signal.py Args: sequence (torch.Tensor) : A tensor of frame level predictions. per_args: onset (float): onset threshold for detecting the beginning and end of a speech offset (float): offset threshold for detecting the end of a speech. pad_onset (float): adding durations before each speech segment pad_offset (float): adding durations after each speech segment; frame_length_in_sec (float): length of frame. Returns: speech_segments(torch.Tensor): A tensor of speech segment in the form of: `torch.Tensor([[start1, end1], [start2, end2]])`. frame_length_in_secrrrrrrrFrT) rgrrrrmrrrrr) rrrrrrrspeechrrspeech_segmentsnew_segr'r'r( binarizations>        &r original_segmentsto_be_removed_segmentscCs(|D]}|||jdd}q|S)aH Remove speech segments list in to_be_removed_segments from original_segments. (Example) Remove torch.Tensor([[start2, end2],[start4, end4]]) from torch.Tensor([[start1, end1],[start2, end2],[start3, end3], [start4, end4]]), -> torch.Tensor([[start1, end1],[start3, end3]]) rIr)eqall logical_not)r r rr'r'r(remove_segmentsJs rcCsB||dddfd}t|dddf|dddffS)z Get the gap segments. For example, torch.Tensor([[start1, end1], [start2, end2], [start3, end3]]) -> torch.Tensor([[end1, start2], [end2, start3]]) NrrIr)rr column_stack)rr'r'r(get_gap_segmentsXs&rrcCs|jtdgkr |S|dd}|dd}|dd}|dkrH|dkr*t||}|dkrFt|}t|t||}t||fd}t|}|S|dkrdt|}t|t||}t||fd}t|}|dkrmt||}|S)a Filter out short non-speech and speech segments. Reference: Paper: Gregory Gelly and Jean-Luc Gauvain. "Minimum Word Error Training of RNN-based Voice Activity Detection", InterSpeech 2015. Implementation: https://github.com/pyannote/pyannote-audio/blob/master/pyannote/audio/utils/signal.py Args: speech_segments (torch.Tensor): A tensor of speech segments in the format torch.Tensor([[start1, end1], [start2, end2]]). per_args: min_duration_on (float): Threshold for short speech segment deletion. min_duration_off (float): Threshold for small non-speech deletion. filter_speech_first (float): Whether to perform short speech segment deletion first. Use 1.0 to represent True. Returns: speech_segments (torch.Tensor): A tensor of filtered speech segments in the format torch.Tensor([[start1, end1], [start2, end2]]). rrrr filter_speech_first?) rrrrgrrrrr)rrrr rnon_speech_segmentsshort_non_speech_segmentsr'r'r( filteringcs4       rcCs|dd}t|dd|d|d|\|d<|d<d|vr,|dr(d|d<nd |d<i}|D]}t||tksBt||tkrH||||<q0||fS) z1 Preparing for generating segment table. r=Nrrrrrrr)rgrr]r%r)rrr=rrr'r'r(prepare_gen_segment_tables    rcCs|d}t||}t||}|jtdgkr|St|d\}}|ddddf|ddddf|}t||f}|S)za See description in generate_overlap_vad_seq. Use this for single instance pipeline. rrNrIr)r rrrrrr)rrUNIT_FRAME_LENrrdurr'r'r(%generate_vad_segment_table_per_tensors  ,r pred_filepathc CsZt|\}}t||\}}t||}|ddrdnd}||}tj||} |jddkrat| ddd"} |ddrB| d n | d Wd | SWd | S1sZwY| St| ddd;} |D]/} |ddr| d |d | ddd| dddqk| | ddd| dddqkWd | S1swY| S)z= A wrapper for generate_vad_segment_table_per_tensor use_rttmF.rttm.txtrrr?r@z.SPEAKER 1 0 0 speech z 0 0 speech NzSPEAKER z 1 r rz speech z speech ) rrrrgrZr[r\rr/rq) rrrrr=rrext save_name save_pathfprr'r'r(#generate_vad_segment_table_per_files8           ,$ r%F vad_pred_dirr5rrc sdfddtD}|s-d}|D]}|dt|t||}qtj|}tj|s8t||||d} i| |} d}|dur}|dkr}t| } t |t | } t t | t| t|d d d Wd|S1svwY|St |d d d D]} t| | q|S) a Convert frame level prediction to speech segment in start and end times format. And save to csv file in rttm-like format 0, 10, speech 17,18, speech Args: vad_pred_dir (str): directory of prediction files to be processed. postprocessing_params (dict): dictionary of thresholds for prediction score. See details in binarization and filtering. frame_length_in_sec (float): frame length. out_dir (str): output dir of generated table/csv file. num_workers(float): number of process for multiprocessing Returns: out_dir(str): directory of the generated table. )rrrcs$g|]}|rtj|qSr')endswithrZr[r\rRrsuffixesr&r'r(rV$z.generate_vad_segment_table.. seg_output-)rr=rNrIzcreating speech segmentsTrLrW)rZlistdirr^r[r\rnrrhrirjrrdrrk(generate_vad_segment_table_per_file_starrmr%) r&r5rrHr=rvad_pred_filepath_list out_dir_namer8rrxryvad_pred_filepathr'r)r(generate_vad_segment_tablesB       r3cCr~r)r%rr'r'r(r/5rr/vad_table_filepathgroundtruth_RTTM_filec Cstj|ddd}tj|dddd}|jddddd }t}|D]\}}|d|t|d|d|d<q"t}|D]\}}d |tt|d t|d t|d <q?||fS) a6 Construct a Pyannote object for evaluation. Args: vad_table_filepath(str) : path of vad rttm-like table. groundtruth_RTTM_file(str): path of groundtruth rttm file. Returns: reference(pyannote.Annotation): groundtruth hypothesis(pyannote.Annotation): prediction r Nsepheaderr7 delimiterr8rrspeakercolumnsSpeechrrI)pdread_csvrenameriterrowsrr%)r4r5rrF referenceindexrow hypothesisr'r'r(&vad_construct_pyannote_object_per_file<s $,rKparamscCsHd}d|vr|d}d}|dtt|}|r"|D]}||d<q|S)zB Get the parameter grid given a dictionary of parameters. FrT)poprdr)rLhas_filter_speech_firstr params_gridrr'r'r(get_parameter_gridYs   rPr|DetERrvad_predgroundtruth_RTTM result_filevad_pred_method focus_metricc! Csd}i} zt|Wntdt|||\} } } t} t|}|D]$}|D]}t||tjks>t||tj krFt ||||<q*zt ||||d}| D]}| |}t j ||d}t||\}}| ||qRtj|dd| jdd}|jd gd }|jd gd }|jd gd }|d ks|dks|dksJd|||d| t|<td|d| t|| t||d}~| t|dddd}||d| t|dWdn1swY||kr|}| t|}|}td||Wq%ty,}ztd|d|WYd}~q%d}~wtjjyJ} ztd|d| WYd} ~ q%d} ~ ww||fS)a Tune thresholds on dev set. Return best thresholds which gives the lowest detection error rate (DetER) in thresholds. Args: params (dict): dictionary of parameters to be tuned on. vad_pred_method (str): suffix of prediction file. Use to locate file. Should be either in "frame", "mean" or "median". groundtruth_RTTM_dir (str): Directory of ground-truth rttm files or a file contains the paths of them. focus_metric (str): Metrics we care most when tuning threshold. Should be either in "DetER", "FA", "MISS" frame_length_in_sec (float): Frame length. num_workers (int): Number of workers. Returns: best_threshold (float): Threshold that gives lowest DetER. rz(Please check if the parameters are valid)rrHrT) ignore_errorsFdisplayr)zdetection error rate%)z false alarmr[)missr[rQFAMISSz>Metric we care most should be only in 'DetER', 'FA' or 'MISS'!)z DetER (%)zFA (%)zMISS (%)z parameter z, z (%)rXr?r@rYNz Current bestzPass z , with error ) check_if_param_validre pred_rttm_maprDetectionErrorRaterPr]npfloat64int64r%r3rZr[r\rKshutilrmtreereportilocitemr^rr.resetr/rqprint RuntimeErrorrCerrorsEmptyDataError)!rLrSrTrUrVrWrrH min_scoreall_perfpaired_filenamesgroundtruth_RTTM_dict vad_pred_dictmetricrOparamr vad_table_dirfilenamer5r4rGrJrgrQr]r^scorer$best_thresholdoptimal_scoresre1r'r'r(vad_tune_threshold_on_devksl  $       r|cCs|D],}|dkrt|dtkstdq|dkrq|dkrq||D] }|dks-tdq#qtdd|d DrEtdd|d DsItd d S) z, Check if the parameters are valid. rzCInvalid inputs! filter_speech_first should be either True or False!rrrz]Invalid inputs! All float parameters except pad_onset and pad_offset should be larger than 0!css|]}|dkVqdS)rINr'rRrr'r'r( sz'check_if_param_valid..rrzJInvalid inputs! The onset and offset thresholds should be in range [0, 1]!T)r]boolrer)rLrrr'r'r(r_s& ,r_c Cs^i}tj|r&t|ddd}|}Wdn1s wYntj|r7ttj|d}nt d|D]}tj | ddd }|||<q=i}tj|rvt|ddd}|} Wdn1spwYntj|rttj|d |} nt d | D]}tj | ddd }|||<q| | @} | ||fS) z< Find paired files in vad_pred and groundtruth_RTTM r+r?r@Nz*.rttmzcgroundtruth_RTTM should either be a directory contains rttm files or a file contains paths to them!rrIrz*.zavad_pred should either be a directory containing vad pred files or a file contains paths to them!) rZr[isfiler/read splitlinesisdirrr\rebasenamersplitkeys) rSrTrVrrr$groundtruth_RTTM_filesrrwrsvad_pred_filesrqr'r'r(r`s<       r`rrIpath2audio_filepath2_vad_predpath2groundtruth_rttmgroundtruth_labels sample_raterunit_frame_len label_repeat xticks_stepc Cstjddgdtj||d||d\} }tj| |d} t||| | }t| | d}d}|rJt|\}}|t|| t|| | }t |}t }| t| j || d | d t| dg|jd d d |d|ddg|}|r|rtd|s|std|r|durt||kdd }n*|r|durt||\}}t||}t||}|t|| t|| | }nd}|r|drt||}n&|rdd|D}| dkrt|| }|t|| t|| | }nd}|dur |j t|| |ddd|dur |j t|| |d dd|dur3|j t|| |ddd|jd dd |jddd|d|ddg|td t| d| tj| |d S)!a Plot Audio and/or VAD output and/or groundtruth labels for visualization Args: path2audio_file (str): path to audio file. path2_vad_pred (str): path to vad prediction file, path2groundtruth_rttm(str): path to groundtruth RTTM file. ground_truth_labels(str): a list of groundtruth label. sample_rate (int): sample rate of audio file. offset (float): offset in seconds. duration (float): duration in seconds. threshold (float): threshold for prediction score (from 0 to 1). per_args(dict): a dict that stores the thresholds for postprocessing. unit_frame_len (float): unit frame length in seconds for VAD predictions. label_repeat (int): repeat the label for this number of times to match different frame lengths in preds and labels. xticks_step (int): step size for xticks. rRrfigsizeTr[rmonorrrrINgrayrrbaxis labelcolorSignalrz3threshold and per_args cannot be used at same time!z?One and only one of threshold and per_args must have been used!rcSg|]}t|qSr'r%r(r'r'r(rVOzplot..r+rFrFrzg--z speech prob lower rightlocshadowzPreds and Probas皙皙?rate)pltfigurerrrrbarangerrrmsubplotplotrset_xlim tick_params set_ylabelset_ylimtwinxrewhererrgen_pred_from_speech_segmentsr'extract_labelsrlegend set_xticksipdAudio)rrrrrrrrrrrraudiortimelen_pred frame_snippetrrax1ax2 pred_snippetrrrrFr'r'r(rsj         "  "    rprobcCsdt|j}dd|D}|jddd|D]}t|d|}t|d|}d|||<q|S)zX Generate prediction arrays like 000111000... from speech segments {[0,1][2,4]} cSrr'rdr}r'r'r(rVlrz1gen_pred_from_speech_segments..cS|dSNrr'rr'r'r(mz/gen_pred_from_speech_segments..r8rrI)rbrrrr)rrrrrrrr'r'r(res rpath2ground_truth_labelrcCs~tj|dddd}|jddddd}g}|D]%}||d|k|d|d|k@}t|d kr7|d q|d q|S) z Extract ground-truth label for given time period. path2ground_truth_label (str): path of groundtruth RTTM file time (list) : a list of array representing time period. \s+Nr9rrr;r<r@rIr)rCrDrErmr`)rrrlabelsposrwr'r'r(rvs$   rrsuse_featc st||}t|d}||}d} g} t|ddd#} | D]} t| ddd} | | d dqWd n1sAwYtd t| d t | }t t t d D]\}}fdd|D}t jjj|r|d|dd}n |d|dd}t j|dd}t|jdkr|jddkr|d}|d d df}|dkr|}n+||dkr|d | }n||dkr||| }n||dkr||d }n|}|}| t|7} tj|| |d}t|ddd}tt|D] } |d|| q Wd n 1s"wYWd n 1s2wY~||dksF||dkrVtd| |d| dd} qc|S)zB Generate VAD frame level prediction and write to out_dir rrr+r?r@r/rz.wavNz Inference on z audio files/json lines!)rMcsg|]}|jqSr')todevicer( vad_modelr'r(rVsz+generate_vad_frame_pred..rI)processed_signalprocessed_signal_length) input_signalinput_signal_lengthrr=rrrz.framerXz {0:0.4f} rz Overall length of prediction of z is !)rr/rarbsplitr`rr.rmrrrtest_dataloaderrampautocastrr]softmaxrsqueezecputolistrZr[r\rrqformatdebug)rrDrrsr=r time_unittrunctrunc_lall_lenrrrwr6rr test_batch log_probsprobsrto_saveoutpathr{r'rr(generate_vad_frame_preds` $      r model_pathcCd|drtd|tj|d}|S|dr"tj|d}|Std|tj|d}|Sz, Initiate VAD model with model path z.nemozUsing local VAD model from ) restore_pathz.ckpt)checkpoint_pathzUsing NGC cloud VAD model ) model_name)r'rr.r restore_fromload_from_checkpointfrom_pretrainedrrr'r'r(init_vad_model    rcCrr)r'rr.rrrrrr'r'r(init_frame_vad_modelrr!asr_stitched_output_manifest.jsonsegmented_output_manifestspeech_segments_tensor_dirstitched_output_manifestcCstj|s t|g}t|ddd}|D] }t|}||qWdn1s.wYt|ddd#}t }d} t |dkrt dt |D]} || dd|| dd|| dd } } t | | g d } t|| fd }|| dd }| |7} || dd d dddd }|| dd || d krtj||d}t|||| dd || d}t|||d|t }d} qO| d7} qOnd} || d|| d|| d } } t | | g d } t|| fd }|| d }| |7} || d d dddd }tj||d}t|||| d || d}t|||d|td|d||WdS1sbwYdS)z3 Stitch the prediction of speech segments. r+r?r@NrrErIrrr pred_textrrrrz.pt)rspeech_segments_filepathrrYr z&Finish stitch segmented ASR output to z8, the speech segments info has been stored in directory )rZr[rnrr/rarbr`rTensorrmrrrrrrr\saverprqrrrr.)rrrsegmented_outputrrwr6r{r all_pred_textrrrr rrspeech_segments_tensor_pathmetar'r'r(stitch_segmented_asr_outputsv     &    & "    &rvad_asr_out.jsoninput_manifestaligned_vad_asr_output_manifestc CsBt}t|ddd}|D] }t|}|||d<q Wdn1s%wYg}t|ddd8}|D]-}t|}|d}||vrV||d|d<||d|d<nd|d<d|d<||q6Wdn1snwYt|d dd} |D]} t| | | d | q}Wd|S1swY|S) z Generate aligned manifest for evaluation. Because some pure noise samples might not appear in stitched_output_manifest. r+r?r@rNrrrErrY)dictr/rarbr`rprqrr) rrrstitched_outputrrwr6outsampler{rr'r'r(construct_manifest_eval2s<        r cCst|s td|tj|dddd}|jddddd }|dt|d<|dt|d<|d|d|d <|jdgd }t t |d|d |d <|S) z4 Load rttm file and extract speech segments zFile not found: rNr9rrr;r<r@r)bysegment) rrnrerCrDrEastyper% sort_valuesrdrj)rrr'r'r(load_rttm_fileVs r intervalscCsd|jdddg}|D]#}|r|dd|dkr ||q t|dd|d|dd<q |S)z= Merge speech segments into non-overlapping segments cSrrr'rr'r'r(rmrz!merge_intervals..rrrIr)rr`r)rrintervalr'r'r(merge_intervalsis  r rttm_filecCs*tt|d}dd|D}t|}|S)zm load speech segments from rttm file, where each segment is represented as [start, end] interval rcSrr'rr(r'r'r(rVrz2load_speech_segments_from_rttm..)rdrr)rrr'r'r(load_speech_segments_from_rttm{srcCstt|d}dd|D}|jdddg}g}|D]5}|r+|dd|d kr1||q||d t|dd|dgt|dd|d|dd<q||fS) a' Load speech segments from RTTM file, merge and extract possible overlaps Args: rttm_file (str): Path to RTTM file Returns: merged (List[List[float]]): merged speech intervals without overlaps overlaps (List[List[float]]): intervals with overlap speech rcSrr'rr(r'r'r(rVrz:load_speech_overlap_segments_from_rttm..cSrrr'rr'r'r(rrz8load_speech_overlap_segments_from_rttm..rrrIr)rdrrr`rr)rrroverlapsrr'r'r(&load_speech_overlap_segments_from_rttms  $ r max_durationcCsRg}d}|D]}|d}|||g|d}q|dur'||kr'|||g|S)a Get non-speech segments from given speech segments and maximum duration Args: speech_segments (List[List[float]]): speech segment intervals loaded by load_speech_segments() max_duration (Optional[float]): maximum duration of the audio, used to calculate the last silence segment Returns: nonspeech_segments (List[List[float]]): intervals of non-speech segments rrrIN)r`)rrnonspeech_segmentsrsp_segrr'r'r(get_nonspeech_segmentss  rT frame_lengthas_strc Csg}tt||}d}t|D]S}|||} |t|dkr=||d| kr=|d7}|t|dkr=||d| ks)||ddkr_||d| krW||dkr_nn|dq|dq|rqddd|DSdd|DS)a~ Generate frame-level binary labels for audio, '0' for non-speech and '1' for speech Args: segments (List[List[float]]): speech segments loaded by load_speech_segments_from_rttm frame_length (float): frame length in seconds, e.g. 0.01 for 10ms frames offset (float): Offset of the audio clip duration (float): duration of the audio clip rrIr cSrr')r^r(r'r'r(rVrz$get_frame_labels..cSrr'rr(r'r'r(rVr)rrbrrrmr`r\) rrrrrrn_framessidrtr'r'r(get_frame_labelss     8  r"rEr audio_filer#showcCsNtjddgdtj|dd||d\}}tj||d} t|} t| ||| } dd | D} t| } t } | || t |j||d | d t| d g| jd dd| d| dd g| }|j t | || ddd|jd dd|jddd|d|ddg|rt|rt|tj|ddS)zA Plot audio signal and frame-level labels from RTTM file rRrrrTrrcSrr'rr(r'r'r(rVrz)plot_sample_from_rttm..rrrIrrrrrr+rFrrrLabelsrrr)rrrrrrr"rrmr set_titlerrbrrrrrrrrrr$savefigrr)r#rrr#r$rrrrrrrlengthrrr'r'r(plot_sample_from_rttms2     r)皙?cCst|}t|}t|}t|}||kr||}||}t|||krf|}t|t|dkrI|dgt|t|t|7}t|}|dt|d}t ||S|dkrq|d| }|dt |d}|S||kr||}||}t|||kr|j t|dd}|d|}|S|j t |dd}|dkr||| d7}|S|S)a Aligns labels to frames when the frame length (e.g., 10ms) is different from the label length (e.g., 20ms). The threshold 0.2 is not critical, as the actual ratio will always be close to an integer unless using frame/label lengths that are not multiples of each other (e.g., 15ms frame length and 20ms label length), which is not valid. The value 0.2 is chosen for easier unit testing. Args: probs (List[float]): List of probabilities. labels (List[int]): List of labels. threshold (float): Threshold for rounding the ratio to an integer. Returns: labels (List[int]): List of labels aligned to frames. rrrINr) rmrrr%longrrviewamaxalign_labels_to_framesrrepeat_interleave)rrr frames_len labels_lenratior|r'r'r(r.s<"   r.speaker_overridecCst}tj|dddd}|jddddd}|D])\}}|dur2||t|d|d|d<q|d|t|d|d|d<q|S) a Read rttm file and construct a Pyannote object. Args: rttm_file(str) : path of rttm file. speaker_override(str) : if not None, all speakers will be replaced by this value. Returns: annotation(pyannote.Annotation): annotation object rNr9rrr;r<r@)rrCrDrErFr)rr3 annotationrrHrIr'r'r(read_rttm_as_pyannote_objectCs  $r5rcCs|g}d}t|D]"\}}|dkr|dkr||}q|dkr*|||d|gd}q|dkr<||t|d|g|S)a Convert a list of labels to a list of speech segments. Args: labels (List[float]): list of labels frame_length_in_sec (float): frame length in seconds Returns: segments (List[Tuple[float, float]]): list of speech segments rrI)rr`rm)rrrrrrFr'r'r(!convert_labels_to_speech_segmentsWs r6 prediction groundtruthc Cs:t}t|tr|drt|dd}nPt|trE|drEtj|ddd}|D]\}}d|tt |dt |dt |d <q*nt|t r`t ||}|D] }d|t|d|d <qQnt d t} t|tr{|dr{t|dd} | |fSt|t rt ||}|D] }d| t|d|d <q| |fSt d ) a} Construct a Pyannote object for evaluation. Args: prediction (str) : path of VAD predictions stored as RTTM or CSV-like txt. groundtruth (str): path of groundtruth rttm file. frame_length_in_sec(float): frame length in seconds Returns: reference(pyannote.Annotation): groundtruth hypothesis(pyannote.Annotation): prediction rr)r3rr Nr6rrIzAprediction must be a path to rttm file or a list of frame labels.zBgroundtruth must be a path to rttm file or a list of frame labels.) r isinstancer^r'r5rCrDrFrr%rdr6re) r7r8rrJrrHrIrrrGr'r'r(,frame_vad_construct_pyannote_object_per_fileos0,    r:cfgcCsnt}i}i}g}t|j}t|d}|D]}t| }t |d|d} t | |d<t| j } | |vr@t d|| |||jrd|vrSdnd} || d} | rt | |d} t| } t| |jjj|d|d d }|| || <d d |D|| <q|d ddurdd |d D|| <qt dqWdn1swY|||fS)aE Load manifest file and prepare label/rttm mapping Args: cfg: DictConfig object Returns: manifest_orig (List[Dict]): original manifest data key_labels_map (Dict): mapping from unique_audio_name to its labels key_rttm_map (Dict): mapping from unique_audio_name to its rttm file r+r)r# manifest_filez=Please make sure each line is with different audio_filepath! rttm_filepathrNrr)rrrrcSrr'rr(r'r'r(rVrz1frame_vad_infer_load_manifest..rFcSrr'rr(r'r'r(rVrzJMust have either `label` or `rttm_filepath` in manifest when evaluate=True)setrrrrr/r_rarbrcrr^rreaddr`evaluatergrr"vadr,rr)r;unique_audio_nameskey_labels_map key_rttm_map manifest_origr<finrwentryruniq_audio_namerttm_keyrr label_strr'r'r(frame_vad_infer_load_manifestsH           "rKpred_dirrCrDkey_pred_rttm_mapc Cs.g}g}t}i}tt|d} t| dt| dD]j} g} | d} | D]} | } | s3q*| t | q*Wdn1sEwY| j }| ||<t | ||d||<||||||||vrr||}n||}t||||d\}}|||qt||d}|jd d }||fS) an Perform evaluation on frame-VAD results Args: pred_dir: directory of frame-VAD prediction files with in `.frame` format key_labels_map: dictionary of mapping each to its labels key_rttm_map: dictionary of mapping each to its GROUNDTRUTH rttm file key_pred_rttm_map: dictionary of mapping each to its PREDICTED rttm file frame_length_in_sec: frame length in seconds, e.g. 0.02s Returns: auroc: AUROC score in 0~100% report: Pyannote detection.DetectionErrorRate() report z*.framezEvaluating VAD results)rNrMr+N)rr)r7r8r)y_truey_scoreFrY)rrardrrrrmr/r_rcr`r%rr.extendr:rrg)rLrCrDrMr all_probs all_labelsrt key_probs_mappredictions_list frame_pred pred_probsrFrwr8r8rGrJaurocrgr'r'r(frame_vad_eval_detection_errors@      rXts_vad_binary_veccfg_vad_paramsunit_10ms_frame_countbypass_postprocessingcCsNt||}|st||}t||}|Sd|_d|_d|_d|_t||}|S)a Post-processing on diarization results using VAD style post-processing methods. These post-processing methods are inspired by the following paper: Medennikov, Ivan, et al. "Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario." (2020). Args: ts_vad_binary_vec (Tensor): Sigmoid values of each frame and each speaker. Dimension: (num_frames,) cfg_vad_params (OmegaConf): Configuration (omega config) of VAD parameters. unit_10ms_frame_count (int, optional): an integer indicating the number of 10ms frames in a unit. For example, if unit_10ms_frame_count is 8, then each frame is 0.08 seconds. bypass_postprocessing (bool, optional): If True, diarization post-processing will be bypassed. Returns: speech_segments (Tensor): start and end of each speech segment. Dimension: (num_segments, 2) Example: tensor([[ 0.0000, 3.0400], [ 6.0000, 6.0800], ... [587.3600, 591.0400], [591.1200, 597.7600]]) rr)rr/r rrrrr)rZr[r\r]ts_vad_binary_framesrr'r'r(ts_vad_post_processings $   r_rbatch_preds_listaudio_rttm_map_dict precisioncsg}|rdnd}tt|t||dD]S\}\} } | d} ||jdd} ddt| jd D} t| jd D](}t| d d |f|||d }|| }|}fd d|D}| | |q:| | q|S) aT Converts floating point number tensor diarization results to timestamps using VAD style post-processing methods. Args: batch_preds_list (List[Tensor]): Tensor diarization results for each sample. Dimension: [(num_frames, num_speakers), ...] audio_rttm_map_dict (Dict[str, Dict[str, Union[float, int]]]): Dictionary mapping unique audio file names to their rttm file entries. cfg_vad_params (OmegaConf): Configuration (omega config) of VAD parameters. unit_10ms_frame_count (int): an integer indicating the number of 10ms frames in a unit. For example, if unit_10ms_frame_count is 8, then each frame is 0.08 seconds. bypass_postprocessing (bool, optional): If True, diarization post-processing will be bypassed. precision (int, optional): The number of decimal places to round the timestamps. Defaults to 2. Returns: total_speaker_timestamps (List[List[List[float]]]): A list of lists of timestamp tensors for each session (utterance) Levels: - Session-level (uniq_id) [session1_list, session2_list,...] - Segment-level: [[start1, end1], [start2, end2],...]] - List of start and end timestamp [start, end] BinarizationzPost-processing)rMrNrrrcSsg|]}gqSr'r'rr'r'r(rVZsz*predlist_to_timestamps..rN)r[r\r]cs$g|]\}}t|t|gqSr')round)rRsttrrbr'r(rVdr+) rrr2rmrrrr_rrPr`)r`rar[r\r]rbtotal_speaker_timestamps pp_message sample_idxuniq_idaudio_rttm_valuesrspeaker_assign_matspeaker_timestampsspk_idts_matts_seg_raw_list ts_seg_listr'rfr(predlist_to_timestamps/s*$  rr)N)NF)r|rrQrrR)r) NNNrrNNNrrIr)r)F)rr)r)T)NrETrr)r*)rYF)Fr)vrrarrhrZre dataclassesr itertoolsrrrpathlibrtypingrrr r r IPython.displayrZrrmatplotlib.pyplotpyplotrnumpyrbpandasrCrr0 omegaconfr r pyannote.corerrpyannote.metricsrsklearn.metricsrsklearn.model_selectionrrnemo.collections.asr.modelsrr4nemo.collections.common.parts.preprocessing.manifestr nemo.utilsrrr^r:rr}rlrdrQrrrr%rrrjitscriptrrrrrrr rrrrrr%rr3r/rKrPr|r_r>r`rrarrayrrrrrrr  DataFramerrrrrr"r)r.r5r6r:rKrXr_rrr'r'r'r(s           >P ?  G    $B  $ F$" >   \&&     b  > N $"*       +?  (4 6 6