o 2wi( @sdZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z ddl mZddlmZddlmZmZdd lmZdd lmZmZdd lmZmZmZd efd dZ   ddedededefddZ ddede ede ee ee eeffffddZ!dS)z About the Aishell corpus Aishell is an open-source Chinese Mandarin speech corpus published by Beijing Shell Shell Technology Co.,Ltd. publicly available on https://www.openslr.org/33 N) defaultdict)Path)DictOptionalUnion)tqdm)$validate_recordings_and_supervisions) Recording RecordingSet) fix_manifests)SupervisionSegmentSupervisionSet)Pathlikeresumable_download safe_extractlinecCsH|dd}|dd}|dd}|dd}|d d }|}|S) u Modified from https://github.com/wenet-e2e/wenet/blob/main/examples/multi_cn/s0/local/aishell_data_prep.sh#L54 sed 's/ａ/a/g' | sed 's/ｂ/b/g' | sed 's/ｃ/c/g' | sed 's/ｋ/k/g' | sed 's/ｔ/t/g' > $dir/transcripts.t uａauｂbuｃcuｋkuｔt)replaceupper)rrS/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/lhotse/recipes/aishell.pytext_normalizes     r.F http://www.openslr.org/resources target_dirforce_downloadbase_urlreturnc CsB|d}t|}|jddd|d}d}d}||fD]}||}||dd} | d } | r?td |d | d qt|d |||dtj| ddt |} t | |dWdn1shwY||kr| d} t | D]} t | | } t | | dWdn1swYqz| q|S)aR Downdload and untar the dataset :param target_dir: Pathlike, the path of the dir to storage the dataset. :param force_download: Bool, if True, download the tars no matter if the tars exist. :param base_url: str, the url of the OpenSLR resources. :return: the path to downloaded and extracted directory with data. z/33Tparentsexist_okaishellzdata_aishell.tgzzresource_aishell.tgzNz .completedzSkipping download of z because z exists./)filenamer) ignore_errors)pathwav)rmkdiris_filelogginginforshutilrmtreetarfileopenroslistdirtouch)rrr url corpus_dirdataset_tar_nameresources_tar_nametar_nametar_path extracted_dircompleted_detectortarwav_dir sub_tar_namerrrdownload_aishell)s>    rBr8 output_dircCs"t|}|sJd||durt|}|jddd|d}i}t|ddd&}|D]}|}d |d d}t|}|||d <q0Wdn1sTwYtt }gd } t | d dD]} t d| g} g} |dd| } | dD]R}|j}|jd}||vrt d|t |dq||}|st d|qt|}| |t||d|jd d||d dd}| |qt| }t| }t||\}}t|||dur||d| d||d| d||d|| <qg|S)aL Returns the manifests which consist of the Recordings and Supervisions :param corpus_dir: Pathlike, the path of the data dir. :param output_dir: Pathlike, the path where to write the manifests. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'recordings' and 'supervisions'. zNo such directory: NTr"z3data_aishell/transcript/aishell_transcript_v0.8.txtrzutf-8)encoding r)traindevtestz2Process aishell audio, it takes about 102 seconds.)desczProcessing aishell subset: data_aishellr+z**/*.wavzNo transcript: z has no transcript.zNo such file: gChinese)id recording_idstartdurationchannellanguagespeakertextaishell_supervisions_z .jsonl.gzaishell_recordings_) recordings supervisions) ris_dirr,r3 readlinessplitjoinrrdictrr.r/rglobstempartswarningr-r from_fileappendr rSstriprr from_recordingsr from_segmentsr rto_file)r8rCtranscript_pathtranscript_dictfridx_transcriptcontent manifests dataset_partspartrZr[wav_path audio_pathidxrVrW recordingsegment recording_setsupervision_setrrrprepare_aishellTsx          rz)rFr)N)"__doc__r.r4r0r2 collectionsrpathlibrtypingrrr tqdm.autorlhotser lhotse.audior r lhotse.qar lhotse.supervisionr r lhotse.utilsrrrstrrboolrBrzrrrrsF      ,