o SiS,@sddlZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z m Z mZmZddlmZddlmZmZddlmZmZddlmZmZdd lmZmZmZdd lm Z m!Z!m"Z"m#Z#d Z$d Z%d Z&ddddde&fde de ee'e e'fde(de(de'de'defddZ)     d,de de e dee'e e'fd e e d!e'd"e*de e'e e'eeeffffd#d$Z+d%ed&e'de e'e efde eeeffd'd(Z,d)e de e'e effd*d+Z-dS)-N)ThreadPoolExecutor)Path)DictListOptionalSequenceTupleUnion)tqdm) fix_manifests$validate_recordings_and_supervisions) Recording RecordingSet)manifests_existread_manifests_if_cached) AlignmentItemSupervisionSegmentSupervisionSet)Pathlikeis_module_availableresumable_download safe_extract)z dev-cleanz dev-otherz test-cleanz test-otherztrain-clean-100ztrain-clean-360ztrain-other-500)z dev-clean-2z train-clean-5z@https://drive.google.com/uc?id=1WYfgr31T-PPwMcxuAq09XZfHQO5Mw8fE.mini_librispeechFz http://www.openslr.org/resources target_dir dataset_partsforce_download alignmentsbase_urlalignments_urlreturnc Cst|}|d}|jddd|dkrt}n|dkrt}nt|tr%|g}t|ddD]{}td||tvr?|d }n|tvrI|d }n t d |q+||} | d } | rktd |d| dq+|d} || } t |d| | |dt j | ddt| } t| |dWdn1swY| q+|r|d} | r|s|StdsJdddl}t|d}|j||dt|}|j|d| Wd|S1swY|S)a Download and untar the dataset, supporting both LibriSpeech and MiniLibrispeech :param target_dir: Pathlike, the path of the dir to storage the dataset. :param dataset_parts: "librispeech", "mini_librispeech", or a list of splits (e.g. "dev-clean") to download. :param force_download: Bool, if True, download the tars no matter if the tars exist. :param alignments: should we download the alignments. The original source is: https://github.com/CorentinJ/librispeech-alignments :param base_url: str, the url of the OpenSLR resources. :param alignments_url: str, the url of LibriSpeech word alignments :return: the path to downloaded and extracted directory with data. LibriSpeechTparentsexist_ok librispeechrzDownloading LibriSpeech partsdesczProcessing split: z/12z/31zInvalid dataset part name: z .completedz Skipping z because z exists.z.tar.gz/)filenamer) ignore_errors)pathNz.ali_completedgdownzFTo download LibriSpeech alignments, please install "pip install gdown"rzLibriSpeech-Alignments.zip)output)rmkdir LIBRISPEECHMINI_LIBRISPEECH isinstancestrr logginginfowarningis_filershutilrmtreetarfileopenrtouchrr,downloadzipfileZipFile extractall)rrrrrr corpus_dirparturlpart_dircompleted_detectortar_nametar_pathtarr, ali_zip_pathfrJN/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/librispeech.pydownload_librispeech"sf            rLautononer@alignments_dir output_dirnormalize_textnum_jobsc st|}|dur t|n|}|sJd||dkr-ttdd|dD}n(|dkrMtttdd|dD}|sLtd |nt |t rU|g}i}|durlt|}|j d d d t ||d }t |}t|d dD]}td|t||ddrtd|dqwg} g} ||} g} t| ddddD]B} i}|| j|| jddd}|rt|}t| }|D] }| |t| ||qWdn1swYqt| dddD]}|}|durq|\}}| || |qt| }t !| }|dkr)ddt !fdd |D}t"||\}}t#|||durP|$|d!|d"|$|d#|d"||d$||<qwWd|S1sdwY|S)%a Returns the manifests which consist of the Recordings and Supervisions. When all the manifests are available in the ``output_dir``, it will simply read and return them. :param corpus_dir: Pathlike, the path of the data dir. :param alignments_dir: Pathlike, the path of the alignments dir. By default, it is the same as ``corpus_dir``. :param dataset_parts: string or sequence of strings representing dataset part names, e.g. 'train-clean-100', 'train-clean-5', 'dev-clean'. By default we will infer which parts are available in ``corpus_dir``. :param output_dir: Pathlike, the path where to write the manifests. :param normalize_text: str, "none" or "lower", for "lower" the transcripts are converted to lower-case. :param num_jobs: int, number of parallel threads used for 'parse_utterance' calls. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'audio' and 'supervisions'. NzNo such directory: rcs|]}|jVqdSNname.0r+rJrJrK s z&prepare_librispeech..*rMcsrTrUrVrXrJrJrKrZszACould not find any of librispeech or mini_librispeech splits in: Tr")rrQz Dataset partsr&zProcessing LibriSpeech subset: r%)rArQprefixzLibriSpeech subset: z already prepared - skipping.z *.trans.txtzDistributing tasksF)r'leaverrz.alignment.txt ProcessinglowercSs|SrU)r_)textrJrJrKsz%prepare_librispeech..csg|]}|qSrJ)transform_text)rYsto_lowerrJrK sz'prepare_librispeech..librispeech_supervisions_z .jsonl.gzlibrispeech_recordings_) recordings supervisions)%ris_dirsetr0 intersectionglobr/union ValueErrorr1r2r.rrr r3r4rrglobparent relative_tostemsplitexistsparse_alignmentsr:appendsubmitparse_utteranceresultrfrom_recordingsr from_segmentsr r to_file)r@rPrrQrRrS manifestsexrArirj part_pathfutures trans_pathrali_pathrIlinefuturer{ recordingsegment recording_setsupervision_setrJrdrKprepare_librispeechps                FFrdataset_split_pathrc Cs|jdd\}}|t|ddj|d}|s(td|dStj ||d}t ||d|j d d t d d |j|||vrKd ||indd }||fS)NrO)maxsplit-r(z.flaczNo such file: ) recording_idrEnglishz-.*word) idrstartdurationchannellanguagespeakerr` alignment)striprurreplacerrr6r3r5r from_filerrresubr)rrrrr` audio_pathrrrJrJrKrzs2  rzrc Csi}t|D]4}|\}}}|ddd}dgttt|ddd}ddt|||ddD||<q |S)N"r,rc Ss,g|]\}}}t||t||dddqS))ndigits)symbolrr)rround)rYrrendrJrJrKrfs z$parse_alignments..rO) r read_text splitlinesrurlistmapfloatzip)rrrutt_idwords timestampsrJrJrKrws" rw)NrMNrNrO).r3rr7r9r=concurrent.futures.threadrpathlibrtypingrrrrrr tqdm.autor lhotser r lhotse.audior rlhotse.recipes.utilsrrlhotse.supervisionrrr lhotse.utilsrrrrr/r0LIBRISPEECH_ALIGNMENTS_URLr2boolrLintrrzrwrJrJrJrKs      P  "!