o Si#@sdZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z ddl Z ddlmZddlmZmZmZmZmZddlmZdd lmZmZdd lmZid d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5Zed6Zed7Z ed8Z!ed9Z"ed:Z#  ;dHde$d?ee%ee%e eeffffd@dAZ&dBe%d?e%fdCdDZ'dEe ed?e efdFdGZ(dS)Iz BABEL is a collection of corpora created during the IARPA BABEL program: https://www.iarpa.gov/index.php/research-programs/babel It has about 25 languages with 40h - 160h of training recordings and ~20h of development set recordings. N) defaultdict)Path)DictIterableListOptionalUnion)sliding_window) Recording RecordingSetSupervisionSegmentSupervisionSet$validate_recordings_and_supervisions)combine)*remove_missing_recordings_and_supervisionstrim_supervisions_to_recordings)Pathlike101 Cantonese102Assamese103Bengali104Pashto105Turkish106Tagalog107 Vietnamese201Haitian202Swahili203Lao204Tamil205Kurmanji206Zulu207z Tok-Pisin301Cebuano302Kazakh303Telugu LithuanianGuaraniIgboAmharic MongolianJavaneseDholuoGeorgian)304305306307401402403404z-(\(\(\)\)||||)z"<(limspack|lipsmack|breath|cough)>z<(click|ring|dtmf|int|sta)>z z"<(male-to-female|female-to-male)> F corpus_dir output_dir no_eval_okreturncCsvtt}|}t|}dd|dD}|std|dt|dkr2td|d|d d |d j}d D]|}|d |d }t dd| dD}t dd| dD}t ||} t| d krx|dkrp|rpq9td|g} |d |d} t | dD]} | jd^} }}}}}}}ddd|d}| dg}ddtd|D}|dg7}tdt|ddd|dddD]q\\}}\}}zJt|dd}t|dd}| t|d|d|d|d|dtd |d! | j|t||d"d#d t|t||d|d|d$Wqty>}ztd%t|td&| d'd}~wwqt| } t| d krStd(| t | } |dkret| d kren t!| | \} } t"| | } t#| | | | d)||<|durt|}|j$d*d*d+t|}|d,krd-n|}| %|d.|d/|d0| %|d.|d1|d0q9t|S)2a Prepares manifests using a single BABEL LDC package. This function works like the following: - first, it will scan `corpus_dir` for a directory named `conversational`; if there is more than once, it picks the first one (and emits a warning) - then, it will try to find `dev`, `eval`, and `training` splits inside (if any of them is not present, it will skip it with a warning) - finally, it scans the selected location for SPHERE audio files and transcripts. :param corpus_dir: Path to the root of the LDC package with a BABEL language. :param output_dir: Path where the manifests are stored.json :param no_eval_ok: When set to True, this function won't emit a warning that the eval set was not found. :return: cSsg|]}|r|qS)is_dir).0drHrHH/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/babel.py ]sz1prepare_single_babel_language..conversationalz;Could not find 'conversational' directory anywhere inside 'z' - please check your path.z=It seems there are multiple 'conversational' directories in 'z)' - we are selecting the first one only (rzq). Please ensure that you provided the path to a single language's dir, and the root dir for all BABEL languages.)devevaltrainingzconversational/z/audiocs|]}t|VqdSNr from_filerJprHrHrL n  z0prepare_single_babel_language..z*.sphcsrSrTrUrWrHrHrLrYqrZz*.wavrQz No SPHERE or WAV files found in z/transcription*_AB)inLineoutLinecSs(g|]\}}|dr|ds|qS)[) startswith)rJprev_llrHrHrLrMsNd06)ndigits)id recording_idstartdurationchanneltextlanguagespeakerz&Error while parsing segment. Message: z/Too many errors while parsing segments (file: 'z5'). Please check your data or increase the threshold.zNo supervisions found in ) recordings supervisionsT)parentsexist_okrRtrainzbabel- _recordings_z .jsonl.gz_supervisions_)&rdictrrglob ValueErrorlenloggingwarningparentr from_recordingsglobrtqdmstemsplitget read_text splitlinesr zipfloatappendr introundnormalize_textBABELCODE2LANG Exceptionstrdeduplicate_supervisionsr from_segmentsrrrmkdirto_file)rDrErF manifestsorig_corpus_dirr audio_dirsph_recordingswav_recordingsrtrutext_dirrXp0p1 lang_codersdatehourrpr\lines timestamprqnext_timestamprnenderr save_splitrHrHrLprepare_single_babel_languageBs           .      rrqcCs@td|}td|}td|}td|}td|}|S)Nzz zz ra) OOV_PATTERNsubSPK_NOISE_PATTERN NOISE_PATTERN SIL_PATTERNREMOVE_PATTERN)rqrHrHrLrs     rrucCs|ddlm}|ddt|ddd}g}|D]"\}}t|dkr4tdt|d|djd ||dq|S) Nr)groupbycSs|jSrT)rl)srHrHrLsz*deduplicate_supervisions..)keyrOzFound z$ supervisions with conflicting IDs (z) - keeping only the first one.) cytoolzrsorteditemsr~rrrlr)rur duplicatesfilteredkvrHrHrLrs  r)NF))__doc__rre collectionsrpathlibrtypingrrrrrrrr lhotser r r r rlhotse.manipulationr lhotse.qarr lhotse.utilsrrcompilerrrrrboolrrrrrHrHrHrLs