o }oiM5@s8ddlZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z ddl Z ddlZddlZddlmZddlmZmZmZddlmZddlmZddlmZdd lmZmZdd lm Z d d Z!d e"de"fddZ#dddZ$dddZ%ddZ&ddZ'ddZ(ddZ)dd Z*dd"d#Z+d$d%Z,d&d'Z-dd(d)Z.d*d+Z/d,d-Z0d.e e"d/e e1fd0d1Z2 2dd3e3fd4d5Z4d6d7Z5dd9d:Z6dd;d<Z7d=d>Z8d?d@Z9dAe e:dBe e:de3fdCdDZ;dAe e:dBe e:fdEdFZddJe1dKe1de:fdNdOZ?ddQe e e:dKe1dRe1de e e:fdSdTZ@dUe e:dVe e e:de e e:fdWdXZA 8ddYe"dZe"d[e3dKe1de"f d\d]ZB  ^ _ ` ddae"dbe"dce:dde:dee:d[e3f dfdgZC h P  i jddke:dce:dde:dle:dee:dKe1dme3dne1doe1de e e:fdpdqZDdke:dce:dde:dle:de e e:f drdsZEdte:due:dve1dne1dejFf dwdxZGdydzZHd{ejFde e e:fd|d}ZId~e e e:de1de e"fddZJde:de:dejFdejFde:de ejFejFff ddZKde:de e e:de e:e1ffddZLdejFde:de:de e e:de1dce:dne1de e1e ejFe e e:e e1ffddZMde:de:dne1dejFdejFde e1dce:dde:de e ejFe e e:e e1ffddZNdee"eOfdee1ejFffddZPde e"de e"fddZQde1de:de1fddZRde e e:e1fde ejFde e e"e e"ffddZSdZe"fddZTde e"de ejFfddZUddZVdZe"fddZWde"dee"e e e:e1ffde ejFfddZXd~e e e:e:fde"dee"e"fde e e"efde e e"efde e e"efde"dBfddZYde e e:de"fddZZddd„Z[GddĄdăZ\dS)N)deepcopy)DictListTupleUnion) ListConfig) AnnotationSegmentTimeline)tqdm) repeat_signal)LongFormSpeakerClustering)get_argmin_matsplit_input_data)loggingcCs0t|turtjtj|d}|Std)z1 Return base name from provided filepath rzinput must be filepath string)typestrospathsplitextbasename TypeError)filepathuniq_idrb/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/utils/speaker_utils.pyget_uniqname_from_filepath$s rlinereturncCst|}t|d}|S)zJ Retrieve `uniq_id` from the `audio_filepath` in a manifest line. audio_filepath)jsonloadsstripr)rdicrrrrget_uniq_id_from_manifest_line/s r$cCst|d}|ddur|ddur|S|dr)ttt|d|td|}nd}|drDttt|d|d|td|}nd}|d|d|}|S) z9 Return basename with offset and end time labels rttm_filepathoffsetNduration rNULL_)rrintroundpow)metadecimals bare_uniq_idr'endtimerrrrget_uniq_id_with_dur8s "*r3Fc Csi}t|dy}|}tdt||D]_}|}t|}|d| dd| dd| dd| dd| d d| d d| d dd }|rSt |}nd | vr^|d }nt |dd}||vrn|||<qt d|ddWd|S1swY|S)a This function creates AUDIO_RTTM_MAP which is used by all diarization components to extract embeddings, cluster and unify time stamps Args: manifest (str): Path to the manifest file attach_dur (bool, optional): If True, attach duration information to the unique name. Defaults to False. Returns: AUDIO_RTTM_MAP (dict) : Dictionary with unique names as keys and corresponding metadata as values. rzNumber of files to diarize: {}rr&Nr'r(text num_speakers uem_filepath ctm_filepath)rr&r'r(r5r6r7r8r)rzfile z^ is already part of AUDIO_RTTM_MAP, it might be duplicated, Note: file basename must be unique)open readlinesrinfoformatlenr"r r!getr3keysrKeyError) manifest attach_durAUDIO_RTTM_MAPinp_filelinesrr#r/uniqnamerrraudio_rttm_mapLs>              !!rGc Cs~dd||fD}dd|||fD}t|st|rt|r+|g|gdg}}}n|||}}}ttt|t|t|gdkoIt|dk}t|t|dddkoct|t|dddk}t|dkrwtd dt||D} n|d|dk} d d i} t||| grt|dkrd d tt||D| d<n d|d|dfi| d<|| d<| Stdt|rtddS)a Check whether multiscale parameters are provided correctly. window_lengths_in_sec, shift_lengfhs_in_sec and multiscale_weights should be all provided in omegaconf.listconfig.ListConfig type. In addition, the scales should be provided in descending order, from the longest scale to the base scale (the shortest). Example: Single-scale setting: parameters.window_length_in_sec=1.5 parameters.shift_length_in_sec=0.75 parameters.multiscale_weights=null Multiscale setting (base scale - window_length 0.5 s and shift_length 0.25): parameters.window_length_in_sec=[1.5,1.0,0.5] parameters.shift_length_in_sec=[0.75,0.5,0.25] parameters.multiscale_weights=[1,1,1] In addition, you can also specify session-by-session multiscale weight. In this case, each dictionary key points to different weights. cSsg|]}t|tqSr) isinstancefloat.0varrrr z'parse_scale_configs..cSsg|] }t|tttfqSr)rHrlisttuplerJrrrrMsg?rNcSsg|]\}}||kqSrr)rKwsrrrrMsuse_single_scale_clusteringFcSsi|] \}\}}|||fqSrr)rKkrSrTrrr sz'parse_scale_configs.. scale_dictmultiscale_weightsz-Multiscale parameters are not properly setup.z^You must provide a list config for all three parameters: window, shift and multiscale weights.) allr=setrOsortedzip enumerate ValueErrorany) window_lengths_in_secshift_lengths_in_secrYcheck_float_configcheck_list_configwindow_lengths shift_lengths length_checkscale_order_checkshift_length_checkmultiscale_args_dictrrrparse_scale_configs~sJ  " 2    rkc Cs`dd|ddD}|dr*t|}d|ddi|d<|ddd|d<n|}|d\}}|D]w}ggg}}} t|dD]5} || \}}t||t||kr`td t||} |||| ||jd|| qHt|d d ||d<tj |dd ||d <tj |dd ||d <t| ||d <q6|S)a The embeddings and timestamps in multiscale_embeddings_and_timestamps dictionary are indexed by scale index. This function rearranges the extracted speaker embedding and timestamps by unique ID to make the further processing more convenient. Args: multiscale_embeddings_and_timestamps (dict): Dictionary of embeddings and timestamps for each scale. multiscale_args_dict (dict): Dictionary of scale information: window, shift and multiscale weights. Returns: embs_and_timestamps (dict) A dictionary containing embeddings and timestamps of each scale, indexed by unique ID. cSsi|]}|iqSrrrKrrrrrWz+get_embs_and_timestamps..rrUrXrYNrQz;Mismatch of counts between embedding vectors and timestampsdim embeddings timestampsmultiscale_segment_counts) r?rr\r=r_torchtensorappendshape unsqueezerIcat) $multiscale_embeddings_and_timestampsrjembs_and_timestamps_multiscale_args_dictrprqrembeddings_listtime_stamps_listsegment_index_list scale_idxtime_stamps_tensorrrrget_embs_and_timestampss.     rcCs^dd|dD}t|dD]}||}|D]}d||i||d|<qq|S)aU The timestamps in `multiscale_timestamps` dictionary are indexed by scale index. This function rearranges the extracted speaker embedding and timestamps by unique ID to make the further processing more convenient. Args: multiscale_timestamps (dict): Dictionary of timestamps for each scale. multiscale_args_dict (dict): Dictionary of scale information: window, shift and multiscale weights. Returns: timestamps_dict (dict) A dictionary containing embeddings and timestamps of each scale, indexed by unique ID. cSsi|]}|diiqS)rXrrlrrrrWrNz"get_timestamps..rrX time_stamps)r?r\)multiscale_timestampsrjtimestamps_dictrrrrrrget_timestampss rc Cst|}g}tt|dD]Q}||\}}}||d\}}} t|t|krRtt|t|d} d| || g||d<||d| d|q||d|d|q|d\}}}||d|d||S)z' Return contiguous time stamps rQg@ rR)rranger=splitrIrjoinru) stampsrEcontiguous_stampsistartendspeaker next_startnext_end next_speakeravgrrrget_contiguous_stampssrc Cst|}g}tt|dD]<}||\}}}||d\}}} t|t|kr=|| kr=d||| g||d<q||d|d|q|d\}}}||d|d||S)z0 Merge time stamps of the same speaker. rQrrR)rrr=rrIrru) rEroverlap_stampsrrrrrrrrrr merge_stamps'srcCsJt|d}|D]}|\}}}t|t|}}||t||<q|S)z\ Convert the given labels to pyannote object to calculate DER and for visualization uri)rr"rrIr )labels uniq_name annotationlabelrrrrrrlabels_to_pyannote_object;s rc Cstj||d}t|d2}|D]&}|}|\}}}t|t|} t|}d||| |} || qWd|S1sCwY|S)zU Write rttm file with uniq_id name in out_rttm_dir with timestamps in labels .rttmrSz6SPEAKER {} 1 {:.3f} {:.3f} {} N) rrrr9r"rrIr<write) rr out_rttm_dirfilenamefrrrrr(logrrrlabels_to_rttmfileHs     rcCstt||S)z8 Convert string to float then round the number. )r-rI)x round_digitsrrrstring_to_floatYsrcCsH|}t|d|}t|d|t|d|}|d}|||fS)a Convert a line in RTTM file to speaker label, start and end timestamps. Args: rttm_line (str): A line in RTTM formatted file containing offset and duration of each segment. round_digits (int): Number of digits to be rounded. Returns: start (float) Start timestamp in floating point number. end (float): End timestamp in floating point number. speaker (str): speaker string in RTTM lines. r%)r"rr) rttm_linerrttmrrrrrrconvert_rttm_line`s  rc Cshg}t|d#}|D]}t|dd\}}}|d|||q Wd|S1s-wY|S)z7 Prepare time stamps label list from rttm file r4r%)rz{} {} {}N)r9r:rrur<) rttm_filenamerrrrrrrrrrttm_to_labelsys   rcCs\tj|dd|d}t|d}|D]}||qWddS1s'wYdS)aj Write cluster labels that are generated from clustering into a file. Args: base_scale_idx (int): The base scale index which is the highest scale index. lines_cluster_labels (list): The start and end time-stamps of each segment with the predicted cluster label. out_rttm_dir (str): The path where output rttm files are saved. z../speaker_outputssubsegments_scalez_cluster.labelrSN)rrrr9r)base_scale_idxlines_cluster_labelsrout_label_namerclus_label_linerrrwrite_cluster_labelss  "rsegment_rangescluster_labelsc Cs`g}t|D]\}}dt|}||\}}||d|d|qt|}t|} | |fS)a Generate cluster (speaker labels) from the segment_range list and cluster label list. Args: segment_ranges (list): List containing intervals (start and end timestapms, ranges) of each segment cluster_labels (list): List containing a cluster label sequence Returns: diar_hyp (list): List containing merged speaker-turn-level timestamps and labels in string format Example: >>> diar_hyp = ['0.0 4.375 speaker_1', '4.375 5.125 speaker_0', ...] lines (list) List containing raw segment-level timestamps and labels in raw digits >>> diar_hyp = ['0.0 0.25 speaker_1', '0.25 0.5 speaker_1', ..., '4.125 4.375 speaker_1'] speaker_r)r^rrurr) rrrEidxrtagsttr cont_linesdiar_hyprrrgenerate_cluster_labelss  rTverbosecsg}g}d}g} d} |jdkrtdd} t| d} |ddr.tj| } tj| dt | dd| d D]\} |} |j rS| d d }|d urRt d nd }| dj dd}| j| d| d| d| dt|t|jt|jt|j|dd |dd d }~ | rtjnt| j|}|}t||j dkrt dt||\}}|rt||| fdd|Dt|d}| |g| dd }|d urt!j"#|r|st$|}t|d}| |gq9d}g}q9|rt%|| |||fS)a Performs spectral clustering on embeddings with time stamps generated from VAD output Args: embs_and_timestamps (dict): This dictionary contains the following items indexed by unique IDs. 'embeddings' : Tensor containing embeddings. Dimensions:(# of embs) x (emb. dimension) 'timestamps' : Tensor containing ime stamps list for each audio recording 'multiscale_segment_counts' : Tensor containing the number of segments for each scale AUDIO_RTTM_MAP (dict): AUDIO_RTTM_MAP for mapping unique id with audio file path and rttm path out_rttm_dir (str): Path to write predicted rttms clustering_params (dict): Clustering parameters provided through config that contains max_num_speakers (int), oracle_num_speakers (bool), max_rp_threshold(float), sparse_search_volume(int) and enhance_count_threshold (int). use_torch_script (bool): Boolean that determines whether to use torch.jit.script for speaker clustering device (torch.device): Device we are running on ('cpu', 'cuda'). verbose (bool): Enable TQDM progress bar. Returns: all_reference (list[uniq_name,Annotation]): reference annotations for score calculation all_hypothesis (list[uniq_name,Annotation]): hypothesis annotations for score calculation FTcudaz[cuda=False, using CPU for eigen decomposition. This might slow down the clustering process.)rexport_script_modulezspeaker_clustering_script.pt clustering)descleavedisabler6NzNProvided option as oracle num of speakers but num_speakers in manifest is nullrRrrrrQrprqrYchunk_cluster_countembeddings_per_chunk) embeddings_in_scalestimestamps_in_scalesrrrYoracle_num_speakersmax_num_speakersmax_rp_thresholdsparse_search_volumerrz9Mismatch of length between cluster_labels and timestamps.csg|] }d|dqS)r r)rKseg_linerrrrM sz&perform_clustering..rr&)&rrwarningr r>rsjitscriptsaver itemsrr_rv forward_inferr,rrIrrr empty_cachegccollectrcpunumpyr=rrextendrrurrexistsrr)rzrCrclustering_paramsdevicerall_hypothesis all_reference no_referencesrrspeaker_clusteringaudio_rttm_valuesuniq_embs_and_timestampsr6rrrqrrE hypothesis rttm_file ref_labels referencerrrperform_clusteringsr                 rcCsp|}t|dkr"t|dt|d|d}}}||fSt|dt|d|d}}}||fS)z: Extract VAD timestamp from the given RTTM lines. r%rrrrQ)r"rr=rI)rvad_outrdurr+rrrget_vad_out_from_rttm_lines  $$rcCsj||d}||ddr$t||d|}t||d|}||fSt|}|j|j}d}||fS)aG Extract offset and duration information from AUDIO_RTTM_MAP dictionary. If duration information is not specified, a duration value is extracted from the audio file directly. Args: AUDIO_RTTM_MAP (dict): Dictionary containing RTTM file information, which is indexed by unique file id. uniq_id (str): Unique file id Returns: offset (float): The offset value that determines the beginning of the audio stream. duration (float): The length of audio stream that is expected to be used. rr(Nr')r>r-sf SoundFileframes samplerate)rCrr0 audio_pathr(r'soundrrrget_offset_and_duration)s  rc CsT||d}|D]\}}|t||t|||d|d}t|||dqdS)a Write the json dictionary into the specified manifest file. Args: outfile: File pointer that indicates output file path. AUDIO_RTTM_MAP (dict): Dictionary containing the input manifest information uniq_id (str): Unique file id overlap_range_list (list): List containing overlapping ranges between target and source. decimals (int): Number of decimals to round the offset and duration values. rUNKrr'r(rrrN)r-r dumpr) outfilerCroverlap_range_listr0rrrr/rrrwrite_overlap_segmentsDs     rcCsV|r$tj|r$t|d }|}Wd|S1swY|Std|)z Read rttm files and return the rttm information lines. Args: rttm_file_path (str): An absolute path to an RTTM file Returns: lines (list): List containing the strings from the RTTM file. r4NzkRequested to construct manifest from rttm with oracle VAD option or from NeMo VAD but received filename as )rrrr9r:FileNotFoundError)rttm_file_pathrrErrrread_rttm_linesas   rc Cst}t|d#}|D]}|}t|}|ddkr#||dq Wdn1s.wYt|}||}|D] }||=t|dq?t |dkrWt ddS)a? This function will check the valid speech segments in the manifest file which is either generated from NeMo voice activity detection(VAD) or oracle VAD. If an audio file does not contain any valid speech segments, we ignore the audio file (indexed by uniq_id) for the rest of the processing steps. r4r(rrNzN is ignored since the file does not contain any speech signal to be processed.zCAll files present in manifest contains silence, aborting next steps) r[r9r"r r!addr?rrr=r_) rC vad_manifest vad_uniq_idsvad_filerr#provided_uniq_ids silence_idsrrrrvalidate_vad_manifestxs$     rrangeArangeBcCs4|d|d}}|d|d}}||ko||kS)ad Check whether two ranges have overlap. Args: rangeA (list, tuple): List or tuple containing start and end value in float. rangeB (list, tuple): List or tuple containing start and end value in float. Returns: (bool): Boolean that indicates whether the input ranges have overlap. rrQr)r r start1end1start2end2rrr is_overlaps rcCsBt||sJd|d|t|d|dt|d|dgS)a| Calculate the overlapping range between rangeA and rangeB. Args: rangeA (list, tuple): List or tuple containing start and end value in float. rangeB (list, tuple): List or tuple containing start and end value in float. Returns: (list): List containing the overlapping range between rangeA and rangeB. z#There is no overlap between rangeA:z and rangeB:rrQ)rmaxmin)r r rrrget_overlap_ranges$r intervals_inc Cst|}|dkr gS|dkr|Sg}d}d}dd|D}t|}tj|dd\}}dd|D}|} | dd| dd} } td|D]0} | | d| | d}}| |krat|| } qHt| t| } } || | g|} t|| } qHt| t| } } || | g|S)a/ Interval merging algorithm which has `O(N*logN)` time complexity. (N is number of intervals) Merge the range pairs if there is overlap exists between the given ranges. This algorithm needs a sorted range list in terms of the start time. Note that neighboring numbers lead to a merged range. Example: input: [(1, 10), (11, 20)] output: [(1, 20)] Refer to the original code at https://stackoverflow.com/a/59378428 Args: intervals_in (list): List containing ranges. Example: >>> intervals_in [(102, 103), (104, 109), (107, 120)] Returns: merged_list (list): List containing the combined ranges. Example: >>> merged_list [(102, 120)] rrQcS$g|]}t|dt|dgqSrrQr,rKrrrrrM$z'merge_int_intervals..rncSrrrrrrrrMr) r=rsrtsortrrrr,ru) r num_intervals merged_liststt2rinterval_tensor_sortedr+ _sorted_int intervalsrrrrrrmerge_int_intervalss2   r!rr0cCs&tjt|d|gddS)z3 Convert floating point number to integer. r)rr0)rsr-rtr,itemrr0rrrfl2ints&r%cCs"tjt|d|g|dS)z3 Convert integer to floating point number. r)r")rsr-rtr#r$rrrint2fls"r&rrangesmarginc svg}g}|D]"}tt|dtt|d}}||kr(|||gqt|}g}fdd|D}|S)a Combine overlaps with floating point numbers. Since neighboring integers are considered as continuous range, we need to add margin to the starting range before merging then subtract margin from the result range. Args: ranges (list): List containing ranges. Example: [(10.2, 10.83), (10.42, 10.91), (10.45, 12.09)] decimals (int): Number of rounding decimals margin (int): margin for determining overlap of the two ranges when ranges are converted to integer ranges. Default is margin=2 which follows the python index convention. Examples: If margin is 0: [(1, 10), (10, 20)] -> [(1, 20)] [(1, 10), (11, 20)] -> [(1, 20)] If margin is 1: [(1, 10), (10, 20)] -> [(1, 20)] [(1, 10), (11, 20)] -> [(1, 10), (11, 20)] If margin is 2: [(1, 10), (10, 20)] -> [(1, 10), (10, 20)] [(1, 10), (11, 20)] -> [(1, 10), (11, 20)] Returns: merged_list (list): List containing the combined ranges. Example: [(10.2, 12.09)] rrQcs,g|]}t|dt|dgqSr)r&rr0r(rrrM%,z)merge_float_intervals..)r,r%rur!) r'r0r( ranges_intmerged_ranges_intrrrmerged_ranges_floatrr)rmerge_float_intervalss*r. target_rangesource_range_listcCs@t|dkrgSg}|D]}t||rt||}||q |S)a  Get the ranges that has overlaps with the target range from the source_range_list. Example: source range: |===--======---=====---====--| target range: |--------================----| out_range: |--------===---=====---==----| Args: target_range (list): A range (a start and end value pair) that defines the target range we want to select. target_range = [(start, end)] source_range_list (list): List containing the subranges that need to be selected. source_range = [(start0, end0), (start1, end1), ...] Returns: out_range (list): List containing the overlap between target_range and source_range_list. r)r=rrru)r/r0 out_ranges_range ovl_rangerrrget_sub_range_list)s    r4rC manifest_fileinclude_uniq_idcCst|dk}|D]_}||d}t|}t|||\}} g} |D]} t| \} } | | | | gq t| |}t|dkrGtd|dq| dkrUtd|dqt |||| gd}t |||||qWd|S1sswY|S) a Write manifest file based on rttm files (or vad table out files). This manifest file would be used by speaker diarizer to compute embeddings and cluster them. This function takes care of overlapping VAD timestamps and trimmed with the given offset and duration value. Args: AUDIO_RTTM_MAP (dict): Dictionary containing keys to unique names, that contains audio filepath and rttm_filepath as its contents, these are used to extract oracle vad timestamps. manifest (str): The path to the output manifest file. Returns: manifest (str): The path to the output manifest file. rSr&rz File ID: z6: The VAD label is not containing any speech segments.z/: The audio file has negative or zero duration.)r0r/N) r9rrrrur.r=rrr4r)rCr5r6r0rrr rttm_linesr'r(vad_start_end_list_rawrrrvad_start_end_listrrrrwrite_rttm2manifestLs.       r:??皙?segments_manifest_filesubsegments_manifest_filewindowshiftmin_subsegment_durationc Cs:|durt}tj|d}t|d}t|da}|} | D]R} | } t| } | d| d| d| df\} } }}t | |||d }|rQd | vrQ| d }nd}|D]}|\}}||krr| ||||d }t ||| d qUq!Wdn1s~wYWd|SWd|S1swY|S) a} Generate subsegments manifest from segments manifest file Args: segments_manifest file (str): path to segments manifest file, typically from VAD output subsegments_manifest_file (str): path to output subsegments manifest file (default (None) : writes to current working directory) window (float): window length for segments to subsegments length shift (float): hop length for subsegments shift min_subsegments_duration (float): exclude subsegments smaller than this duration value Returns: returns path to subsegment manifest file Nzsubsegments.jsonr4rSrr'r(rr'r@rAr(rrr) rgetcwdrrr9r:r"r r!get_subsegments_scriptablerr)r>r?r@rArBr6pwdsegments_manifestsubsegments_manifestsegmentssegmentr#audior'r(r subsegmentsr subsegmentrrr/rrr)segments_manifest_to_subsegments_manifestusL $    ( rN{Gz?>dr'r(use_asr_style_frame_count sample_rate feat_per_secc CsLg} |} | |} ||kr|krnnd} n9|durBtd||t||t} t| t||t} | || } ntd|||t} | dkrgt|||kre| | t||g| S| dkrt|| |d| }t| t|||t|}tj ||d}||k}tj ||||gdd}| } | S)ab Return subsegments from a segment of audio file. Example: (window, shift) = 1.5, 0.75 Segment: [12.05, 14.45] Subsegments: [[12.05, 13.55], [12.8, 14.3], [13.55, 14.45], [14.3, 14.45]] Args: offset (float): Start time of audio segment window (float): Window length for segments to subsegments length shift (float): Hop length for subsegments shift duration (float): Duration of segment min_subsegment_duration (float): Exclude subsegments smaller than this duration value decimals (int): Number of decimal places to round to use_asr_style_frame_count (bool): If True, use asr style frame count to generate subsegments. For example, if duration is 10 secs and frame_shift is 0.08 secs, it results in (10/0.08)+1 = 125 + 1 frames. Returns: subsegments (List[tuple[float, float]]): subsegments generated for the segments as list of tuple of start and duration of each subsegment rQTrNr"rn) npceilr,astyperrursarange ones_liker-stacktolist)r'r@rAr(rBr0rRrSrTrLr slice_endslicesnum_feat_frames start_col dur_col_rawdur_col valid_maskvalid_subsegmentsrrrget_subsegmentss0"$ rdc Cs~g}|}||}t|||}|dkrdn|d}t|D]} ||} | |kr+|} ||| |g|| d|}q|S)a This function returns subsegments from a segment of an audio file. Although this implementation is inefficient due to the use of a for-loop for segmentation, it is designed to be torch-jit-scriptable. Use `get_subsegments` for a more efficient implementation. Args: offset (float): start time of audio segment window (float): window length for segments to subsegments length shift (float): hop length for subsegments shift duration (float): duration of segment Returns: subsegments (List[tuple[float, float]]): subsegments generated for the segments as list of tuple of start and duration of each subsegment rrQ)mathrVrru) r'r@rAr(rLrr\baser]slice_idrrrrrEs rE start_secend_sec slice_lengthcCs2t||}tt||t||}|||S)a Extract time-series signal from the given audio buffer based on the start and end timestamps. Args: start_sec (float): Start of the targeted segments in second end_sec (float): Start of the targeted segments in second slice_length (int): Length of the entire audio segment that the samples are extracted from sample_rate (int): Sampling rate of the time-series audio signal Returns: (Tensor) Trimmed ime-series audio signal samples )r,r)sigrhrirjrS start_idxend_idxrrrget_target_sigs  rncCs8t|jdD]}||}|d|dkrtdqdS)a  Check whether the range list has any faulty timestamp order. Args: range_tensor (list): List containing the start and end time of the segments. Example: >>> range_tensor = [[0.5, 3.12], [3.51, 7.26], ... ] rrQzIRange start time should be preceding the end time but we got: {range_tup}T)rrvr_) range_tensorrV range_tuprrr check_ranges%s  rqrocsfddtjdDS)zL For online segmentation. Force the list elements to be float type. cs,g|]}t|dt|dgqSr)rI)rKrVrorrrM:r*z"tensor_to_list..r)rrvrrrrrrtensor_to_list6srsspeaker_timestamps model_spk_numc Cs\g}t|D]%}||}t|}|D]}||ddd|dddt|gqq|S)a Generate diarization output lines list from the speaker timestamps list by merging overlapping intervals. Args: speaker_timestamps (list): List containing the start and end time of the speech intervals for each speaker. Example: >>> speaker_timestamps = [[0.5, 3.12], [3.51, 7.26],... ] model_spk_num (int): Number of speakers in the model. Returns: speaker_lines_total (list): List containing the diarization output lines in the format: "start_time end_time speaker_id" Example: >>> speaker_lines_total = ["0.5 3.12 speaker_0", "3.51 7.26 speaker_1",...] rz.3frrQ speaker_)rr.rr,)rtruspeaker_lines_totalspk_idx ts_invervalsmerged_ts_intervals ts_intervalrrr!generate_diarization_output_lines=s .r| frame_start buffer_endvad_timestampscumulative_speech_labelscursor_for_old_segmentsc Csg}||krt|t|g}t|}t|}tt|t|g|d}t||d}t||dd}t||dd}t|}t|}||fS)ad Bring the new speech labels from the current buffer. Followingly: 1. Concatenate the old speech labels from self.cumulative_speech_labels for the overlapped region. - This goes to new_speech_labels. 2. Update the new 1 sec of speech label (speech_label_for_new_segments) to self.cumulative_speech_labels. 3. Return the speech label from cursor_for_old_segments to buffer end. Args: frame_start (float): Start of the middle audio chunk in the audio buffer buffer_end (float): End of the audio buffer vad_timestamps (Tensor): Tensor containing VAD intervals (start and end timestamps) cumulative_speech_labels (torch.Tensor): Cumulative speech/non-speech timestamps (equivalent to VAD timestamps) cursor_for_old_segments (float): Floating point number that indicates the point where new segments should replace the old segments Returns: speech_label_for_new_segments (Tensor): The intervals (start and end) timestamps where the new incoming speech segments should be collected from cumulative_speech_labels (Tensor): Cumulative speech/non-speech timestamps (equivalent to VAD timestamps) with newly added speech/non-speech timestamps from the `vad_timestamps` input )r/r0r)r()rIrsr4r.rsrt) r}r~rrrupdate_overlap_rangenew_incoming_speech_labelsupdate_overlap_speech_labelsspeech_label_for_new_segmentsrrrget_speech_labels_for_updateYs$$  rsegment_range_tscCsl|}t|}d}t|dkr,|d|d}||dkr%|d7}|d}nnt|dkst||}||fS)a Function for updating a cursor online speaker diarization. Remove the old segments that overlap with the new frame (self.frame_start) cursor_for_old_segments is set to the onset of the t_range popped lastly. Args: frame_start (float): Start of streaming pipeline frame segment_range_ts (float): Interval (start and end timestamps) of the targeted segments Returns: cursor_for_old_segments (float): Floating point number that indicates the point where new segments should replace the old segments cursor_index (int): The index of the first newly accepted segments rTrRrQ)r=)r}rr cursor_indexcountt_rangerrrget_new_cursor_for_updates    rrk buffer_startrL ind_offsetcCs"g}g}g} t||} d} |D]b} | d| d} }| |kr q|d7}||}t| |} | |kr9tt| |} t|| | | |}t|dkrKtdt|| krYt|t|| }|| }|| }| ||||g||qt| t|krt|kstdtd|| ||fS)a\ Create short speech segments from slices for online processing purpose. Args: sig (Tensor): Tensor containing the raw time-series signal buffer_start (float): Start point of the time-series signal buffer buffer_end (float): End point of the time-series signal buffer subsegments (list): List containing the interval information (start and duration) of each segment ind_offset (int): Offset for index that compensates the point of the current position in the streaming session window (float): Window length in second shift (float): Shift length in second Returns: sigs_list (list): list of sliced input signal audio_lengths (list): list of audio sample lengths rrrQz6len(signal) is zero. Signal length should not be zero.z)Signal information lists have a mismatch.)r,rIrrnr=r_r ru)rkrr~rLrr@rSsig_rangel_list sig_indexes sigs_listrjrisubsegrhr buffer_lensignal start_abs_sec end_abs_secrrrget_online_segments_from_slicess:"       rspeech_labels_for_update audio_buffersegment_indexesc Csg}g} g} t|dkr|d} nd} t|D]Q\} } t| d|t| d|g}td|d|dg}t|d|||d|dd}t|||||| |d\} }}}||| || |qt|t| krzt| ks}JJ|| | fS)a1 Generate subsegments for online processing from the given segment information. This function extracts subsegments (embedding vector level) time-series from the raw time-series buffer based on the segment interval (start and end timestamps) information. Args: buffer_start (float): Start point of the time-series signal buffer buffer_end (float): End point of the time-series signal buffer sample_rate (int): Sampling rate of the audio input speech_labels_for_update (Tensor): Tensor containing intervals (start and end timestamps) of the speech segments audio_buffer (Tensor): Tensor containing the raw time-series signal segment_indexes (list): List containing the unique indices of segments window (float): Window length in second shift (float): Shift length in second Returns: sigs_list (list): List containing the tensors of the old and the newly added time-series signals sig_rangel_list (list): List containing the old and the newly added intervals (timestamps) of the speech segments sig_indexes (list): List containing the old and the newly added unique indices of segments rrRrQrC)rkrr~rLr@rrS)r=r^rIr#rrErr)rr~rSrrrr@rArrrrr range_spl range_offsrange_trLsigsr'indsrrr"get_online_subsegments_from_buffer s:)  (  ( rrcCsPi}t|d|d|dd\}}t|}tt|D] }||}|||<q|S)a" Calculate cosine similarity values among speaker embeddings for each scale then apply multiscale weights to calculate the fused similarity matrix. Args: uniq_embs_and_timestamps: (dict) The dictionary containing embeddings, timestamps and multiscale weights. If uniq_embs_and_timestamps contains only one scale, single scale diarization is performed. Returns: scale_mapping_argmat (dict) Dictionary containing scale mapping information matrix for each scale. rprqrr)rrrr)rrrr=)rscale_mapping_argmatrrsession_scale_mapping_listrmapping_argmatrrrget_scale_mapping_argmatVs  r cont_stamps ovl_spk_idxc Csddtt|D}tt|D]'}t|D] \}}|\}}}|||vr7|||d|d|qqg} |D]} t| dkrL| t| q=| S)a Generate timestamps that include overlap speech. Overlap-including timestamps are created based on the segments that are created for clustering diarizer. Overlap speech is assigned to the existing speech segments in `cont_stamps`. Args: cont_stamps (list): Non-overlapping (single speaker per segment) diarization output in string format. Each line contains the start and end time of segments and corresponding speaker labels. ovl_spk_idx (list): List containing segment index of the estimated overlapped speech. The start and end of segments are based on the single-speaker (i.e., non-overlap-aware) RTTM generation. Returns: total_ovl_cont_list (list): Rendered diarization output in string format. Each line contains the start and end time of segments and corresponding speaker labels. This format is identical to `cont_stamps`. cSg|]}gqSrrrKr+rrrrMz&get_overlap_stamps..rrvr)rr=r^rrurr) rrovl_spk_cont_listrxr cont_a_linerrrtotal_ovl_cont_list ovl_cont_listrrrget_overlap_stampsrs  restimated_num_of_spks min_thresholdoverlap_infer_spk_limitcCs ||d|d|d}|S)a This function controls the magnitude of the sigmoid threshold based on the estimated number of speakers. As the number of speakers becomes larger, diarization error rate is very sensitive to overlap speech detection. This function linearly increases the threshold in proportion to the estimated number of speakers so more confident overlap speech results are reflected when the number of estimated speakers is relatively high. Args: estimated_num_of_spks (int): Estimated number of speakers from the clustering result. min_threshold (float): Sigmoid threshold value from the config file. This threshold value is the minimum threshold when `estimated_num_of_spks=2`. overlap_infer_spk_limit (int): If the `estimated_num_of_spks` is less than `overlap_infer_spk_limit`, overlap speech estimation is skipped. Returns: adaptive_threshold (float): Threshold value that is scaled based on the `estimated_num_of_spks`. rrQr)rrradaptive_thresholdrrrget_adaptive_thresholdsr clus_labels msdd_predscKs|d|jd}ddt|D}|t|dk}g}|dr,t||d|d}n|d}t|D]\}} |d|d|f|k} |d|f} |drbt| d } nt |d|fd d dd} t | d kr|rt | d d d} | d |d D]}|t| kr|| |q| | dd | d d| q4t |}t|}t||}||fS)a Generate speaker timestamps from the segmentation information. If `use_clus_as_main=True`, use clustering result for main speaker labels and use timestamps from the predicted sigmoid values. In this function, the main speaker labels in `maj_labels` exist for every subsegment step, while overlap speaker labels in `ovl_labels` only exist for segments where overlap speech occurs. Args: clus_labels (list): List containing integer-valued speaker clustering results. msdd_preds (list): List containing tensors of the predicted sigmoid values. Each tensor has shape of: (Session length, estimated number of speakers). params: Parameters for generating RTTM output and evaluation. Parameters include: infer_overlap (bool): If False, overlap speech will not be detected. use_clus_as_main (bool): Add overlap-speech detection from MSDD to clustering results. If False, only MSDD output is used for constructing output RTTM files. overlap_infer_spk_limit (int): Above this limit, overlap-speech detection is bypassed. use_adaptive_thres (bool): Boolean that determines whether to use adaptive thresholds depending on the estimated number of speakers. max_overlap_spks (int): Maximum number of overlap speakers detected. Default is 2. threshold (float): Sigmoid threshold for MSDD output. Returns: maj_labels (list): List containing string-formatted single-speaker speech segment timestamps and corresponding speaker labels. Example: [..., '551.685 552.77 speaker_1', '552.99 554.43 speaker_0', '554.97 558.19 speaker_0', ...] ovl_labels (list): List containing string-formatted additional overlapping speech segment timestamps and corresponding speaker labels. Note that `ovl_labels` includes only overlapping speech that is not included in `maj_labels`. Example: [..., '152.495 152.745 speaker_1', '372.71 373.085 speaker_0', '554.97 555.885 speaker_1', ...] rrRcSrrrrrrrrMrz/generate_speaker_timestamps..ruse_adaptive_thres thresholduse_clus_as_mainrNrQmax_overlap_spksrrv)squeezervrr,rr^rrr[rUargsortsumrurrr)rrparamsroverlap_speaker_list infer_overlapmain_speaker_linesrseg_idx cluster_label spk_for_seg sm_for_seg main_spk_idxidx_arrrr maj_labels ovl_labelsrrrgenerate_speaker_timestampss8 &   ( $ rcCsvg}t|ddd(}t|D]\}}|}t|}t|d}||qWd|S1s4wY|S)zRRetrieve `uniq_id` values from the given manifest_file and save the IDs to a list.r4utf-8encodingrN)r9r^r:r"r r!rru)r5 uniq_id_listrArrr#rrrrget_uniq_id_list_from_manifests    rr preds_listcCsFdd|D}t|D]\}}t|j}|||j||gq |S)a Create session-level dictionary containing data needed to construct RTTM diarization output. Args: uniq_id_list (list): List containing the `uniq_id` values. test_data_collection (collections.DiarizationLabelEntity): Class instance that is containing session information such as targeted speaker indices, audio filepath and RTTM filepath. preds_list (list): List containing tensors of predicted sigmoid values. Returns: session_dict (dict): Dictionary containing session-level target speakers data and predicted simoid values in tensor format. cSsi|]}|gqSrrrrrrrWrmz#get_id_tup_dict..)r^r audio_fileru target_spks)rtest_data_collectionr session_dictrrrrrrget_id_tup_dicts  rcCstj|d}|dkrtj|rt|t|t|dd}tj|d}t d|tj|s=t ||ddi}|d D]3\}\} } tj|d |d } tj| spt || | | dd t d |d | t | } | ||<qEt||} | S)a This function is needed for preparing diarization training data for multiscale diarization decoder (MSDD). Prepare multiscale timestamp data for training. Oracle VAD timestamps from RTTM files are used as VAD timestamps. In this function, timestamps for embedding extraction are extracted without extracting the embedding vectors. Args: manifest_filepath (str): Input manifest file for creating audio-to-RTTM mapping. _out_dir (str): Output directory where timestamp json files are saved. Returns: multiscale_args_dict (dict): - Dictionary containing two types of arguments: multi-scale weights and subsegment timestamps for each data sample. - Each data sample has two keys: `multiscale_weights` and `scale_dict`. - `multiscale_weights` key contains a list containing multiscale weights. - `scale_dict` is indexed by integer keys which are scale index. - Each data sample is indexed by using the following naming convention: `__` Example: `fe_03_00106_mixed_626310_642300` speaker_outputsrT)rBzoracle_vad_manifest.jsonz/Extracting oracle VAD timestamps and saving at )r6rXrz.json)r>r?r@rAr6z3Subsegmentation for timestamp extracted for: scale-z at )rrrrshutilrmtreemakedirsrGrr;r:rrNextract_timestampsr)manifest_filepath_out_dirrj global_rank speaker_dirsplit_audio_rttm_map_speaker_manifest_pathmultiscale_timestamps_by_scalerr@rAsubsegments_manifest_pathrmultiscale_timestamps_dictrrrprepare_split_datas8        rc Cstd|di}t|ddd<}t|D],\}}|}t|}|d}||vr2g||<|d}||d}||||gqWd |S1sQwY|S) a This method extracts timestamps from segments passed through manifest_file. Args: manifest_file (str): Manifest file containing segmentation information. Returns: time_stamps (dict): Dictionary containing lists of timestamps. zExtracting timestamps from z for multiscale subsegmentation.r4rrrr'r(N) rr;r9r^r:r"r r!ru) r5rrArrr#rrrrrrrXs"      rmanifest_file_pathclus_label_dictcKsTt|}g}gg}}d}t|ddd} t| D]x\} } t| } || } || }|t|t||| fi|\}}|drH||}n|}t|| d}|drdt |dd d }t || |d|| |g| d d }|d urt j |r|st|}t|| d}|| |gqd }g}qWd ||fS1swY||fS)a Create RTTM files that include detected overlap speech. Note that the effect of overlap detection is only notable when RTTM files are evaluated with `ignore_overlap=False` option. Args: manifest_file_path (str): Path to the input manifest file. clus_label_dict (dict): Dictionary containing subsegment timestamps in float type and cluster labels in integer type. Indexed by `uniq_id` string. msdd_preds (list): List containing tensors of the predicted sigmoid values. Each tensor has shape of: (Session length, estimated number of speakers). params: Parameters for generating RTTM output and evaluation. Parameters include: infer_overlap (bool): If False, overlap-speech will not be detected. See docstrings of `generate_speaker_timestamps` function for other variables in `params`. Returns: all_hypothesis (list): List containing Pyannote's `Annotation` objects that are created from hypothesis RTTM outputs. all_reference List containing Pyannote's `Annotation` objects that are created from ground-truth RTTM outputs Fr4rrrrrcSst|dS)Nr)rIr)rrrrrz(make_rttm_with_overlap..)keyr&NT)rGr9r^r:r$rur=rrr\rr>rrrr)rrrrrCmanifest_file_lengths_listrrrrArrr manifest_dicrrr hyp_labelsrrrrrrrmake_rttm_with_overlaprs>      rrrrrall_uemsrcCst|ddt|dd}}t|t|d} t| |d} |durItj|rIt|d|dd } | | Wdn1sDwY| || g|d d} | durtj| r|||gg} t | }|}t||d}t | |d }| || ||g|||fS) a Convert speaker timestamps to pyannote.core.Timeline object. Args: speaker_timestamps (List[Tuple[float, float]]): Timestamps of each speaker: start time and end time of each speaker. uniq_id (str): Unique ID of each speaker. audio_rttm_values (Dict[str, str]): Dictionary of manifest values. all_hypothesis (List[Tuple[str, pyannote.core.Timeline]]): List of hypothesis in pyannote.core.Timeline object. all_reference (List[Tuple[str, pyannote.core.Timeline]]): List of reference in pyannote.core.Timeline object. all_uems (List[Tuple[str, pyannote.core.Timeline]]): List of uems in pyannote.core.Timeline object. out_rttm_dir (str | None): Directory to save RTTMs Returns: all_hypothesis (List[Tuple[str, pyannote.core.Timeline]]): List of hypothesis in pyannote.core.Timeline object with an added Timeline object. all_reference (List[Tuple[str, pyannote.core.Timeline]]): List of reference in pyannote.core.Timeline object with an added Timeline object. all_uems (List[Tuple[str, pyannote.core.Timeline]]): List of uems in pyannote.core.Timeline object with an added Timeline object. r'Nr()rtrur/rrSr&r) rIr>r|r=rrrrr9 write_rttmrurget_uem_object)rtrrrrrrr'rrrrr uem_linesorg_ref_labelsrruem_objrrrtimestamps_to_pyannote_objects("$       rrcCs8t|d}|D]}|\}}|tt|t|q|S)a Generate pyannote timeline segments for uem file. file format UNIQ_SPEAKER_ID CHANNEL START_TIME END_TIME Args: uem_lines (list): list of session ID and start, end times. Example: [[0.0, 30.41], [60.04, 165.83]] uniq_id (str): Unique session ID. Returns: timeline (pyannote.core.Timeline): pyannote timeline object. r)r rr rI)rrtimeline uem_stt_end start_timeend_timerrrrs r绽|=cCsP||jdd}|r||jdd|}tjtjj|ddddd}||}|S)z Mean and l2 length normalize the input speaker embeddings Args: embs: embeddings of shape (Batch,emb_size) Returns: embs: normalized embeddings of shape (Batch,emb_size) r)axisrrR)ordrrQ)meanstdrU expand_dimslinalgnorm)embsuse_stdeps embs_l2_normrrrembedding_normalizes  rc@szeZdZdZdefddZdejdejdeejdeee d eed e d e d e eejeee eeffd dZ dS)OnlineSegmentora: Online Segmentor for online (streaming) diarizer. - The class instances created by this class takes time-series signal from the audio buffer and creates subsegments for embedding extraction. - Since online segmentation is based on a short audio buffer, the methods in this class extracts a few subsegments from the given intervals for the raw time-series signal. Attributes: frame_start (float): Start of the middle chunk buffer_start (float): Start of the entire buffer buffer_end (float): End of the entire buffer sample_rate (int): Sampling rate of the input time-series signal cumulative_speech_labels (Tensor): Torch tensor matrix containing culmulative VAD (speech activity) timestamps rScCs(d|_d|_d|_||_tg|_dS)Nr)r}rr~rSrsrtr)selfrSrrr__init__&s zOnlineSegmentor.__init__rrsegment_raw_audiorrr@rArc CsF|jdkrt|dkr%|jddkr%t|ddd|dd<|}||_n@t|j|\} } |d| }|d| }|d| }t|t|krRt|ksWtdtdt|j|j |j|| \}|_t |j|j |j |||||d\} } } | | | | | | t|t|krt|kstdtd|||fS)at Remove the old segments that overlap with the new frame (self.frame_start) cursor_for_old_segments is pointing at the onset of the t_range popped most recently. Frame is in the middle of the buffer. |___Buffer___[___________]____________| |____________[ Frame ]____________| | <- buffer start |____________| <- frame start Args: audio_buffer (Tensor): Tensor containing raw time-series signal vad_timestamps (Tensor): Tensor containing VAD intervals (start and end timestamps) segment_raw_audio (list): List containing the previously added tensors of the raw time-series signal segments segment_range_ts (list): List containing the previously added intervals (start and end timestamps) of each segment segment_indexes (list): List containing the previously added global integer indicies of the segments from start to current cursor window (float): Window length in second shift (float): Shift length in second Returns: segment_raw_audio (list): List containing the newly added tensors of the raw time-series signal segment_range_ts (list): List containing the newly added interval (start and end timestamps) of each segment segment_indexes (list): List containing the newly added global integer indicies of the segments from start to current cursor rrNz8Scale-wise segment information has a mismatch in length.)rr~rSrrrr@rAz-Segment information has a mismatch in length.) rr=rvrrrr}r_rr~rrSr)rrrrrrr@rArrrrrrrrrrun_online_segmentation-sL 1        z'OnlineSegmentor.run_online_segmentationN) __name__ __module__ __qualname____doc__r,rrsTensorrrIrrrrrrrs(  r)r%)F)r)T)r)rr)Fr)Nr;r<r=F)rOrFrPrQ)Fr)]rr rerrcopyrtypingrrrrrrU soundfilerrsomegaconf.listconfigr pyannote.corerr r r (nemo.collections.asr.data.audio_to_labelr 4nemo.collections.asr.parts.utils.longform_clusteringr 3nemo.collections.asr.parts.utils.offline_clusteringrr nemo.utilsrrrr$r3rGrkrrrrrrrrrrr,rboolrrrrrrrIrrr!r%r&r.r4r:rNrdrEr rnrqrsr|rrrrdictrrrrrrrrrrrrrrrrrs       2L.     j ";,*+$ + =  &< " C   %   H $M  G ? =  9