o Sio @s"dZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z ddlmZmZddlmZmZddlmZmZdd lmZmZmZd Zd d efd ede ede edefddZde ede e ee effddZ ddede ede ee ee eeffffddZ dS)aR About the yes no dataset: This dataset was created for the Kaldi project (see kaldi.sf.net), by a contributor who prefers to remain anonymous. The main point of the dataset is to provide an easy and fast way to test out the Kaldi scripts for free. The archive "waves_yesno.tar.gz" contains 60 .wav files, sampled at 8 kHz. All were recorded by the same male speaker, in Hebrew. In each file, the individual says 8 words; each word is either the Hebrew for "yes" or "no", so each file is a random sequence of 8 yes-es or noes. There is no separate transcription provided; the sequence is encoded in the filename, with 1 for yes and 0 for no, for instance: # tar -xvzf waves_yesno.tar.gz waves_yesno/1_0_1_1_1_0_1_0.wav waves_yesno/0_1_1_0_0_1_1_0.wav ... The dataset can be downloaded from the following address: https://www.openslr.org/1/ N) defaultdict)Path)DictListOptionalTupleUnion) fix_manifests$validate_recordings_and_supervisions) Recording RecordingSet)SupervisionSegmentSupervisionSet)Pathlikeresumable_download safe_extractz5http://www.openslr.org/resources/1/waves_yesno.tar.gz.F target_dirforce_downloadurlreturncCst|}|jddd|d}|d}|d}|r&td|d|St|||dtj|dd t |}t ||d Wd n1sJwY| |S) aDownload and untar the dataset. :param target_dir: Pathlike, the path of the dir to store the dataset. The extracted files are saved to target_dir/waves_yesno/*.wav :param force_download: Bool, if True, download the tar file no matter whether it exists or not. :param url: str, the url to download the dataset. :return: the path to downloaded and extracted directory with data. Tparentsexist_ok waves_yesnozwaves_yesno.tar.gzz .completedz Skipping - z exists.)filenamer) ignore_errors)pathN) rmkdiris_filelogginginforshutilrmtreetarfileopenrtouch)rrr extracted_dirtar_pathcompleted_detectortarr+H/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/yesno.pydownload_yesno)s  r-datasetc sdddg}g}|D]Q}|jd}t|dksJt|ddhddhks/Jd|fd d |D}d |}t|}| |t |j|jd |j d d|d}| |q ||fS)zBuild a list of Recording and SupervisionSegment from a list of sound filenames. :param dataset: List[Pathlike], a list of sound filenames :return: a tuple containing a list of Recording and a list of SupervisionSegment NOYES)01_r1r2z words is: csg|]}|qSr+r+).0wword_mapr+r, `sz$_prepare_dataset.. grHebrew)id recording_idstartdurationchannellanguagetext) stemsplitlensetunionjoinr from_fileabsoluteappendr r?)r. recordings supervisions audio_pathwordsrB recordingsegmentr+r7r,_prepare_datasetMs,  (   rR corpus_dir output_dirc Cs@t|}|sJd||durt|}|jdddt|d}t|dks-J||ddd}|ddd}t|d ksGJt|d ksOJtt}t d d g||gD]A\}}t |\}} t |} t | } t| | \} } t| | |dur| |d |d | |d|d | | d||<q\|S)aT Returns the manifests which consist of the Recordings and Supervisions. When all the manifests are available in the ``output_dir``, it will simply read and return them. :param corpus_dir: Pathlike, the path of the data dir. It's expected to contain wave files with the pattern x_x_x_x_x_x_x_x.wav, where there are 8 x's and each x is either 1 or 0. :param output_dir: Pathlike, the path where to write the manifests. :return: a Dict whose key is either "train" or "test", and the value is Dicts with the keys 'recordings' and 'supervisions'. zNo such directory: NTrz*.wav<traintestyesno_supervisions_z .jsonl.gzyesno_recordings_)rLrM)ris_dirrlistglobrEsortrdictziprRr from_recordingsr from_segmentsr r to_file) rSrT wave_files train_settest_set manifestsnamer.rLrM recording_setsupervision_setr+r+r, prepare_yesnots4     rm)N)!__doc__r r"r$ collectionsrpathlibrtypingrrrrrlhotser r lhotse.audior r lhotse.supervisionr r lhotse.utilsrrr _DEFAULT_URLboolstrr-rRrmr+r+r+r,sJ   $ (