o }oiQ@sddlZddlZddlmZddlmZddlmZddlm Z m Z m Z ddl Z ddl ZddlmZmZmZmZmZmZddlmZddlmZdhd ed ed efddZ    didededededededededededed ed efdd Zd!ed"ed#ed efd$d%Zd&ed efd'd(Z d)ed*ed+ed,ed e ee!ff d-d.Z"d/ed e ee!ffd0d1Z#d2e ee!fd3e ee!fd4ed5ed,ef d6d7Z$d8ed9e e!d:efd;d<Z%d=ed e efd>d?Z&d=e ed e eeffd@dAZ'dBe edCe ed e eeffdDdEZ(djdFedCe edGed e eeffdHdIZ)d/ed4ed*ed+ed5ed,ef dJdKZ*     LdkdMedNedOedPedQedRedSe+fdTdUZ,dVe eefd e e!fdWdXZ-dldZe eefd[e e!d\e+fd]d^Z.dZed_e ee!ffd`daZ/dZed_e ee!ffdbdcZ0ddeeBdeed efdfdgZ1dS)mN)Counter) OrderedDict)Path)DictListUnion)audio_rttm_mapget_subsegments_scriptableget_uniqname_from_filepathrttm_to_labels)segments_manifest_to_subsegments_manifestwrite_rttm2manifest)logging)DataStoreObjectnumoutput_precisionreturncCs t|t||}|d|dS)a Get a string of a float number with rounded precision. Args: num (float): float number to round output_precision (int): precision of the output floating point number min_precision (int, optional): Minimum precision of the output floating point number. Defaults to 1. max_precision (int, optional): Maximum precision of the output floating point number. Defaults to 3. Returns: (str): Return a string of a float number with rounded precision. .f)minmax)rr min_precision max_precisionrc/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/utils/manifest_utils.pyget_rounded_str_float%s rNAunknown1sourcechannel start_timedurationtokenconf type_of_tokenspeakerNA_tokenUNKdefault_channelc CsXgd} t|tkr|dddrt|}nt|tkr'tdt|t|tkr;|dddr;t|}nt|tkrJtdt|t|tkr^|dddr^t|}n|dure|}nt|tkrttdt||durt|tkrt|}|durt|tkrd |krdksntd ||durt|tkrtd t|d ||dur|| vrtd | d|d ||durt|tkrtdt||dur| n|}|dur|n|}|dur|n|}|dur| n|}t|| }t|| }||krt|| n|}|d|d|d|d|d|d|d|dS)u Get a line in Conversation Time Mark (CTM) format. Following CTM format appeared in `Rich Transcription Meeting Eval Plan: RT09` document. CTM Format: Reference: https://web.archive.org/web/20170119114252/ http://www.itl.nist.gov/iad/mig/tests/rt/2009/docs/rt09-meeting-eval-plan-v2.pdf Args: source (str): is name of the source file, session name or utterance ID channel (int): is channel number defaults to 1 start_time (float): is the begin time of the word, which we refer to as `start_time` in NeMo. duration (float): is duration of the word token (str): Token or word for the current entry conf (float): is a floating point number between 0 (no confidence) and 1 (certainty). A value of “NA” is used (in CTM format data) when no confidence is computed and in the reference data. type_of_token (str): is the token type. The legal values of are “lex”, “frag”, “fp”, “un-lex”, “for-lex”, “non-lex”, “misc”, or “noscore” speaker (str): is a string identifier for the speaker who uttered the token. This should be “null” for non-speech tokens and “unknown” when the speaker has not been determined. NA_token (str, optional): A token for . Defaults to ''. output_precision (int, optional): The precision of the output floating point number. Defaults to 3. Returns: str: Return a line in CTM format filled with the given information. )lexfragfpzun-lexzfor-lexznon-lexmiscnoscorerrz>`start_time` must be a float or str containing float, but got z<`duration` must be a float or str containing float, but got Nz8`conf` must be a float or str containing float, but got rz(`conf` must be between 0 and 1, but got z!`type` must be a string, but got z type z`type` must be one of z , but got z$`speaker` must be a string, but got   )typestrreplaceisdigitfloat ValueErrorintr) r"r#r$r%r&r'r(r)r*r+r,rVALID_TOKEN_TYPESrrr get_ctm_line6sB-      *  4r=soldnewcCs||d}||S)z Replace end of string. Args: s (str): string to operate on old (str): ending of string to replace new (str): replacement for ending of string Returns: new.join(li) (string): new string with end replaced r)rsplitjoin)r>r?r@lirrrrreplaces rDpathcCs@tj|ddd}t|dkrd|}|S|d}|S)z Get uniq_id from path string with period in it. Args: path (str): path to audio file Returns: uniq_id (str): unique speaker ID rNrr)osrEbasenamesplitlenrB)rE split_pathuniq_idrrrget_uniq_id_with_periods  rMsubsegments_manifest_filewindowshiftdecicCsi}t|dp}|}|D]`}|}t|}|d|d|d} } } t| ||| d} |ddur:|d} nt| } | |vrIggd|| <| D]}|\}}qK|| d t||t|||g|| d |qWd|S1szwY|S) ar Get subsegment dictionary from manifest file. Args: subsegments_manifest_file (str): Path to subsegment manifest file window (float): Window length for segmentation shift (float): Shift length for segmentation deci (int): Rounding number of decimal places Returns: _subsegment_dict (dict): Subsegment dictionary raudio_filepathoffsetr%)rTrOrPr%rLN)tsjson_dicrUrV) open readlinesstripjsonloadsr rMappendround)rNrOrPrQ_subsegment_dictsubsegments_manifestsegmentssegmentdicaudiorTr% subsegmentsrL subsegmentstartdurrrrget_subsegment_dicts,     & rhinput_manifest_pathcCsli}t|d%}|}|D]}t|}d|d<t|d}|||<qWd|S1s/wY|S)z Get dictionary from manifest file. Args: input_manifest_path (str): Path to manifest file Returns: input_manifest_dict (dict): Dictionary from manifest file rR-textrSN)rWrXrZr[r )riinput_manifest_dictinput_manifest_fp json_lines json_linerbrLrrrget_input_manifest_dicts     rprlr^output_manifest_path step_countcCst|dt}|D]f\}}t|d}tj|dd} | jd|} t| dD]F} | dddf| |} | dddf| d|} || df}|| df}t|||}||}||d<||d<t ||| d q)q WddS1s|wYdS) a Write subsegments to manifest filepath. Args: input_manifest_dict (dict): Input manifest dictionary _subsegment_dict (dict): Input subsegment dictionary output_manifest_path (str): Path to output manifest file step_count (int): Number of the unit segments you want to create per utterance deci (int): Rounding number of decimal places wrUr)axisrNrTr%r4) rWitemsnparrayargsortshaperanger]rZdumpwrite)rlr^rqrrrQoutput_manifest_fprL subseg_dict subseg_arraysubseg_array_idxchunked_set_countidxchunk_index_sttchunk_index_end offset_secend_secrgmetarrrwrite_truncated_subsegmentss&     "rnamelinesrcCsXt|d}|D]}||}t|||dqWddS1s%wYdS)z Write json lines to file. Args: name (str): Output file path lines (list): List of json lines idx (int): Indices to dump to the file rsr4NrWrZr{r|)rrrfoutirbrrr write_files   "rpathlistcCs@t|d}|}Wdt|S1swYt|S)z Read list of lines from target file. Args: pathlist (str): Input file path Returns: sorted(pathlist) (list): List of lines rRN)rWrXsorted)rrrrr read_files  rcCs:t}t|}|D]}tj|dd}|||<q |S)z Read dictionaries from list of lines Args: pathlist (list): List of file paths Returns: path_dict (dict): Dictionary containing dictionaries read from files rr)odrrGrErHrI)r path_dict line_pathrLrrrget_dict_from_wavlists   r data_pathlistuniqidscCsHi}|D]}tj|dd}||vr|||<qtd|d|S)z Create dictionaries from list of lines Args: data_pathlist (list): List of file paths uniqids (list): List of file IDs Returns: path_dict (dict): Dictionary containing file paths rrzuniq id z is not in wav filelist)rGrErHrIr:)rrrrrLrrrget_dict_from_list0s  r data_pathlen_wavscCsP|durt|}|durt||ksJt||}|S|dur&dd|D}|S)aA Create dictionary from list of lines (using the get_dict_from_list function) Args: data_path (str): Path to file containing list of files uniqids (list): List of file IDs len_wavs (int): Length of file list Returns: data_pathdict (dict): Dictionary containing file paths NcSsi|]}|dqSNr).0rLrrr Usz!get_path_dict..)rrJr)rrrr data_pathdictrrr get_path_dictDs  rcCsd|vrtd|rd|vrtd|st|dd|d}t|}t|dd}t|dd}d} t|}t|} t| ||} |} t| | ||| t| |||} t|| |||t |t |d S) a Create segmented manifest file from base manifest file Args: input_manifest_path (str): Path to input manifest file output_manifest_path (str): Path to output manifest file window (float): Window length for segmentation shift (float): Shift length for segmentation step_count (int): Number of the unit segments you want to create per utterance deci (int): Rounding number of decimal places z.jsonz4input_manifest_path file should be .json file formatz5output_manifest_path file should be .json file format_zseg.jsonz _seg.jsonz _subseg.jsong?N) r:rDrpr;rr r rhrrGremove)rirqrOrPrrrQrlsegment_manifest_pathsubsegment_manifest_pathmin_subsegment_durationAUDIO_RTTM_MAPsegments_manifest_filerNsubsegments_dictrrrcreate_segment_manifestYs2     rFwav_pathmanifest_filepath text_path rttm_pathuem_pathctm_path add_durationc Cstj|r t|t|}t|}t|} t|} t || | } t || | } t || | } t || | }g}| D]}||| || || |||f\}}}}}| }|duro| }t |}t dd|D }nd}|dury| }|durt| }|d }Wdn1swYnd}|dur| }d}|rtj|dd\}}tj||d}|d|d|||||d g}||q9t||tt|dS) a Create base manifest file Args: wav_path (str): Path to list of wav files manifest_filepath (str): Path to output manifest file text_path (str): Path to list of text files rttm_path (str): Path to list of rttm files uem_path (str): Path to list of uem files ctm_path (str): Path to list of ctm files add_duration (bool): Whether to add durations to the manifest file NcSsg|]}|dqS)rF)rI)rlabelrrr sz#create_manifest..rrj)sr)yrinfer) rSrTr%rrk num_speakers rttm_filepath uem_filepath ctm_filepath)rGrEexistsrrrrJrkeysrrYr r__len__rWrXlibrosaload get_durationextendrrz)rrrrrrr wav_pathlist wav_pathdictrr text_pathdict rttm_pathdict uem_pathdict ctm_pathdictruidwavrkrttmuemctm audio_linelabelsrrr%rrrrrrcreate_manifestsf        rmanifestc Cstt|}g}z t|ddd}Wn td|g}|D]%}|}|s-q$zt|}Wntj yC| |Yq$w| |q$| |rqt t|d||D] }t d|dq^td||S) z Read manifest file Args: manifest (str or Path): Path to manifest file Returns: data (list): List of JSON items rRutf-8encodingz#Manifest file could not be opened: z1 Errors encountered while reading manifest file: z-- Failed to parse line: ``z0Errors encountered while reading manifest file: )rr6rWget ExceptionrXrYrZr[JSONDecodeErrorr\closererrorrJ RuntimeError)rdatarerrorslineitemrrrr read_manifests2    rT output_pathtarget_manifest ensure_asciicCsXt|ddd}|D]}tj|||d|dq WddS1s%wYdS)a Write to manifest file Args: output_path (str or Path): Path to output manifest file target_manifest (list): List of manifest file entries ensure_ascii (bool): default is True, meaning the output is guaranteed to have all incoming non-ASCII characters escaped. If ensure_ascii is false, these characters will be output as-is. rsrr)rr4Nr)rrroutfiletgtrrrwrite_manifests   "r target_ctmcCs\|jdddt|d}|D] }|d}||qWddS1s'wYdS)z Write ctm entries from diarization session to a .ctm file. Args: output_path (str): target file path target_ctm (dict): list of ctm entries cS|dSNrrrrrrzwrite_ctm..keyrsrN)sortrWr|)rrrpairrrrr write_ctms  "rcCsx|jdddt|d%}|D]}|d}|dd}||dq|dWd d S1s5wYd S) z Write text from diarization session to a .txt file Args: output_path (str): target file path target_ctm (dict): list of ctm entries cSrrrrrrrr&rzwrite_text..rrsrr3r4N)rrWrIr|)rrrrrwordrrr write_texts  "rfilepath base_pathcCs,t|}|s|s||}|S)a  Return absolute path to an audio file. Check if a file exists at `filepath`. If not, assume that the path is relative to `base_path`. Args: filepath (str or Path): path to file base_path (Path): base path to resolve relative path )r expanduseris_file is_absoluteabsolute)rrrrrfilepath_to_absolute/s  r)rr)rrr r!r)NNNNF)T)2rZrG collectionsrrrpathlibrtypingrrrrnumpyrv.nemo.collections.asr.parts.utils.speaker_utilsrr r r r r nemo.utilsrnemo.utils.data_utilsrr9r;r6rr=rDrMdictrhrprrrrrrrboolrrrrrrrrrrs           U&!   $&( . U$%