o 2wi:@sdZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z m Z mZmZddlmZddlmZmZddlmZmZdd lmZmZdd lmZmZdd lmZm Z m!Z!m"Z"d Z#d Z$dZ%dZ&gdZ'gdZ(dZ)dZ*dZ+   d,de,dede-defddZ.de,dedee/e/ffdd Z0de,d!e/dedeeeffd"d#Z1de,d$e/dedeeeffd%d&Z2de,dedeeeffd'd(Z3 d-de,ded)e ede e,e e,eeeffffd*d+Z4dS).ak About the speech_commands corpus This is a set of one-second .wav audio files, each containing a single spoken English word or background noise. These words are from a small set of commands, and are spoken by a variety of different speakers. This data set is designed to help train simple machine learning models. It is covered in more detail at https://arxiv.org/abs/1804.03209. Version 0.01 of the data set (configuration "v0.01") was released on August 3rd 2017 and contains 64,727 audio files. Version 0.02 of the data set (configuration "v0.02") was released on April 11th 2018 and contains 105,829 audio files. N) defaultdict)Path)DictListOptionalSequenceTupleUnion)tqdm) fix_manifests$validate_recordings_and_supervisions) Recording RecordingSet)manifests_existread_manifests_if_cached)SupervisionSegmentSupervisionSet)Pathlikeis_module_availableresumable_download safe_extractz@http://download.tensorflow.org/data/speech_commands_v0.01.tar.gzzIhttp://download.tensorflow.org/data/speech_commands_test_set_v0.01.tar.gzz@http://download.tensorflow.org/data/speech_commands_v0.02.tar.gzzIhttp://download.tensorflow.org/data/speech_commands_test_set_v0.02.tar.gz)trainvalidtest) downgoleftnooffonrightstopupyes _silence_ _unknown__background_noise_2.Fspeechcommands_version target_dirforce_downloadreturnc Cst|}|d|}|jdddd|d|f}t|d|dD][}td|||}|d }|rHtd |d |d q%|d }||} td|| |dtj|ddt |  } | |Wdn1swwY| q%|S)a[ Download and unzip Speech Commands dataset :param speechcommands_version: str, dataset version. :param target_dir: Pathlike, the path of the dir to store the dataset. :param force_download: bool, if True, download the archive even if it already exists. :return: the path to downloaded and extracted directory with data. SpeechCommandsTparentsexist_okspeech_commands_v0.0speech_commands_test_set_v0.0z Downloading Speech Commands v0.0desczProcessing split: z .completedz Skipping z because z exists.z.tar.gzz$http://download.tensorflow.org/data/)filenamer+) ignore_errorsN) rmkdirr logginginfois_filershutilrmtreetarfileopen extractalltouch) r)r*r+ corpus_dir dataset_partspartpart_dircompleted_detectortar_nametar_pathtarrIZ/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/lhotse/recipes/speechcommands.pydownload_speechcommands-s8      rKrAc st|}|d|}g}t|D]bdkr7t|}|}Wdn1s1wYqdkrZt|}|}Wdn1sTwYqtj|rtt|}|fdd|D7}q| tj t dt |t |t |}||fS)z Returns the train_paths and valid_paths given a dataset part. :param speechcommands_version: str, dataset version. :param corpus_dir: Pathlike, the path of the data dir. :return: the train_paths and valid_paths. r1ztesting_list.txtNzvalidation_list.txtc$g|]}|drtj|qS.wavendswithospathjoin.0 file_pathrootrIrJ v  z&_split_train_valid..zrunning_tap.wav) rrQlistdirr>readstrip splitlinesrRisdirappendrSBACKGROUND_NOISEset)r)rA part_path train_pathsfile_objtrain_test_paths valid_paths file_pathsrIrWrJ_split_train_valid^s.  rirdc Ct|}|d|}g}g}|D]`}||}|}|d}|dddd} |dd} |dtkr.rkrlrNrmrnrNrorprrrsrt)rrQr[rRr_rbr}r~rWORDSr]SILENCEUNKNOWNr:r8rr rr`rrwrrrrr r )r)rArc test_pathsrhtest_recordingstest_supervisions test_pathrtest_path_splitsrrrzr{test_recording test_segmenttest_recording_settest_supervision_setrIrWrJ _prepare_testsl           r output_dirc Csht|}|sJd|tdd|d|f}t}|dur/t|}|jdddtt}t|dd D]x}td |d |t ||d |d dr^td|d |dq9|dkrrt ||\}}t |||\} } n|dkrt |||\} } n |dkrt ||\} } |dur| |d |d|d| |d |d|d| | d||<q9|S)a Returns the manifests which consist of the Recordings and Supervisions :param speechcommands_version: str, dataset version. :param corpus_dir: Path to the Speech Commands dataset. :param output_dir: Pathlike, the path where to write the manifests. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'recordings' and 'supervisions'. zNo such directory: z9Preparing Speech Commands v0.0{speechcommands_version}...r1r2NTr.z Dataset partsr3zProcessing Speech Commands v0.0z subset: speechcommandszjsonl.gz)rCrprefixsuffixzSpeech Commands v0.0z already prepared - skipping.rrr_supervisions_z .jsonl.gz _recordings_) recordings supervisions)ris_dirr8r9_SPLITSr7rdictr rrirrrto_file) r)rArrBsubsets manifestsrCrdrg recording_setsupervision_setrIrIrJprepare_speechcommandsKsj    r)r'r(F)N)5__doc__r8rQr;r= collectionsrpathlibrtypingrrrrrr tqdm.autor lhotser r lhotse.audior rlhotse.recipes.utilsrrlhotse.supervisionrr lhotse.utilsrrrr_DOWNLOAD_PATH_V1_TEST_DOWNLOAD_PATH_V1_DOWNLOAD_PATH_V2_TEST_DOWNLOAD_PATH_V2rrrrrastrboolrKrbrirrrrrIrIrIrJs      1  '  =  >  N