o 2wi4@sddlZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z m Z mZmZddlmZddlmZmZddlmZmZddlmZmZdd lmZmZmZdd lm Z m!Z!m"Z"m#Z# d(de de ee$e e$fde%defddZ&d d d d de'fde de ee$e e$fde%de%de$de$defddZ(     d)de de e dee$e e$fde e d e$d!e)de e$e e$eeeffffd"d#Z*d$ed%e$de e$e efde eeeffd&d'Z+dS)*N)ThreadPoolExecutor)Path)DictListOptionalSequenceTupleUnion)tqdm) fix_manifests$validate_recordings_and_supervisions) Recording RecordingSet)manifests_existread_manifests_if_cached) AlignmentItemSupervisionSegmentSupervisionSet)Pathlikeis_module_availableresumable_download safe_extract.mini_librispeechF target_dir dataset_partsforce_downloadreturnc CstdsJdddlm}|d}t|}|d}|jddd|d kr(t}n|d kr/t}nt|tr7|g}t |d d D]{}t d ||tvrQt d}n|tvr[t d}n t d|q=||}|d} | r}t d|d| dq=|d} || } t|d| | |dtj|ddt| } t| |dWdn1swY| q=tr|d} | r|s|StdsJdddl} t|d}| jt|dt|}|j|d| Wd|S1swY|S) a Download the GLOBE dataset from HuggingFace Hub. :param target_dir: Pathlike, the path of the dir to storage the dataset. :param dataset_parts: "librispeech", "mini_librispeech", or a list of splits (e.g. "dev-clean") to download. :param force_download: Bool, if True, download the tars no matter if the tars exist. :return: the path to downloaded and extracted directory with data. datasetsz(Please run `pip install datasets` first.r) load_datasetz MushanW/GLOBE LibriSpeechTparentsexist_ok librispeechrDownloading LibriSpeech partsdescProcessing split: /12/31Invalid dataset part name: .completed Skipping because  exists..tar.gz/filenamer ignore_errorspathN.ali_completedgdownFTo download LibriSpeech alignments, please install "pip install gdown"LibriSpeech-Alignments.zipoutput)rrrrmkdir LIBRISPEECHMINI_LIBRISPEECH isinstancestrr logginginfobase_urlwarningis_filershutilrmtreetarfileopenrtouch alignmentsr9downloadalignments_urlzipfileZipFile extractall)rrrrds corpus_dirparturlpart_dircompleted_detectortar_nametar_pathtarr9 ali_zip_pathfr^Q/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/lhotse/recipes/globe.pydownload_globesl             r`z http://www.openslr.org/resourcesrMrErOc Cst|}|d}|jddd|dkrt}n|dkrt}nt|tr%|g}t|ddD]{}td||tvr?|d }n|tvrI|d }n t d |q+||} | d } | rktd |d| dq+|d} || } t |d| | |dt j | ddt| } t| |dWdn1swY| q+|r|d} | r|s|StdsJdddl}t|d}|j||dt|}|j|d| Wd|S1swY|S)a Download and untar the dataset, supporting both LibriSpeech and MiniLibrispeech :param target_dir: Pathlike, the path of the dir to storage the dataset. :param dataset_parts: "librispeech", "mini_librispeech", or a list of splits (e.g. "dev-clean") to download. :param force_download: Bool, if True, download the tars no matter if the tars exist. :param alignments: should we download the alignments. The original source is: https://github.com/CorentinJ/librispeech-alignments :param base_url: str, the url of the OpenSLR resources. :param alignments_url: str, the url of LibriSpeech word alignments :return: the path to downloaded and extracted directory with data. r Tr!r$rr%r&r(r)r*r+r,r-r.r/r0r1r2r4r6Nr8r9r:rr;r<)rr>r?r@rArBr rCrDrFrGrrHrIrJrKrrLrr9rNrPrQrR)rrrrMrErOrTrUrVrWrXrYrZr[r9r\r]r^r^r_download_librispeechasf            raautononerTalignments_dir output_dirnormalize_textnum_jobsc st|}|dur t|n|}|sJd||dkr-ttdd|dD}n(|dkrMtttdd|dD}|sLtd |nt |t rU|g}i}|durlt|}|j d d d t ||d }t |}t|d dD]}td|t||ddrtd|dqwg} g} ||} g} t| ddddD]B} i}|| j|| jddd}|rt|}t| }|D] }| |t| ||qWdn1swYqt| dddD]}|}|durq|\}}| || |qt| }t !| }|dkr)ddt !fdd |D}t"||\}}t#|||durP|$|d!|d"|$|d#|d"||d$||<qwWd|S1sdwY|S)%a Returns the manifests which consist of the Recordings and Supervisions. When all the manifests are available in the ``output_dir``, it will simply read and return them. :param corpus_dir: Pathlike, the path of the data dir. :param alignments_dir: Pathlike, the path of the alignments dir. By default, it is the same as ``corpus_dir``. :param dataset_parts: string or sequence of strings representing dataset part names, e.g. 'train-clean-100', 'train-clean-5', 'dev-clean'. By default we will infer which parts are available in ``corpus_dir``. :param output_dir: Pathlike, the path where to write the manifests. :param normalize_text: str, "none" or "lower", for "lower" the transcripts are converted to lower-case. :param num_jobs: int, number of parallel threads used for 'parse_utterance' calls. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'audio' and 'supervisions'. NzNo such directory: rcs|]}|jVqdSNname.0r7r^r^r_ s z&prepare_librispeech..*rbcsrirjrkrmr^r^r_roszACould not find any of librispeech or mini_librispeech splits in: Tr!)rrfz Dataset partsr&zProcessing LibriSpeech subset: r$)rUrfprefixzLibriSpeech subset: z already prepared - skipping.z *.trans.txtzDistributing tasksF)r'leaverrz.alignment.txt ProcessinglowercSs|Srj)rt)textr^r^r_sz%prepare_librispeech..csg|]}|qSr^)transform_text)rnsto_lowerr^r_ sz'prepare_librispeech..librispeech_supervisions_z .jsonl.gzlibrispeech_recordings_) recordings supervisions)%ris_dirsetr@ intersectionglobr?union ValueErrorrArBr>rrr rCrDrrglobparent relative_tostemsplitexistsparse_alignmentsrKappendsubmitparse_utteranceresultrfrom_recordingsr from_segmentsr r to_file)rTrerrfrgrh manifestsexrUr~r part_pathfutures trans_pathrMali_pathr]linefuturer recordingsegment recording_setsupervision_setr^ryr_prepare_librispeechs                FFrdataset_split_pathrc Cs|jdd\}}|t|ddj|d}|s(td|dStj ||d}t ||d|j d d t d d |j|||vrKd ||indd }||fS)Nrd)maxsplit-r1z.flaczNo such file: ) recording_idgrEnglishz-.*word) idrstartdurationchannellanguagespeakerru alignment)striprrreplacerrGrCrFr from_filerrresubr)rrrMrru audio_pathrrr^r^r_r.s2  r)rrF)NrbNrcrd),rCrrHrJrPconcurrent.futures.threadrpathlibrtypingrrrrrr tqdm.autor lhotser r lhotse.audior rlhotse.recipes.utilsrrlhotse.supervisionrrr lhotse.utilsrrrrrBboolr`LIBRISPEECH_ALIGNMENTS_URLraintrrr^r^r^r_s     O P