o Si @s dZddlZddlZddlZddlmZddlmZmZm Z m Z ddl m Z ddl m Z mZmZmZddlmZm Z ddlmZdd lmZmZdd lmZmZmZd Z ddedeedeedefddZ ddedeedeeeee e effffddZ dS)a AISHELL-3 is a large-scale and high-fidelity multi-speaker Mandarin speech corpus published by Beijing Shell Shell Technology Co.,Ltd. It can be used to train multi-speaker Text-to-Speech (TTS) systems. The corpus contains roughly 85 hours of emotion-neutral recordings spoken by 218 native Chinese mandarin speakers and total 88035 utterances. Their auxiliary attributes such as gender, age group and native accents are explicitly marked and provided in the corpus. Accordingly, transcripts in Chinese character-level and pinyin-level are provided along with the recordings. The word & tone transcription accuracy rate is above 98%, through professional speech annotation and strict quality inspection for tone and prosody. N)Path)DictOptionalSequenceUnion)tqdm) RecordingSetSupervisionSegmentSupervisionSet$validate_recordings_and_supervisions) Recordingr) fix_manifests)manifests_existread_manifests_if_cached)Pathlikeresumable_download safe_extract)testtrain.F http://www.openslr.org/resources target_dirforce_downloadbase_urlreturncCst|}|jddd|d}d}||}|d}|r,td|d|d|St|d |||d t|}t||d Wd n1sNwY| |S) aR Download and untar the dataset :param target_dir: Pathlike, the path of the dir to storage the dataset. :param force_download: Bool, if True, download the tars no matter if the tars exist. :param base_url: str, the url of the OpenSLR resources. :return: the path to downloaded and extracted directory with data. Tparentsexist_okz/93zdata_aishell3.tgzz .completedz Skipping z because z exists./)filenamer)pathN) rmkdiris_filelogginginfortarfileopenrtouch)rrrurltar_nametar_pathcompleted_detectortarr-K/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/aishell3.pydownload_aishell3&s"   r/ corpus_dir output_dircCst|}|sJd|t}i}|dur)t|}|jdddt||dd}i}|d}|s:Jd|t|d 1}|D]$}|}| d sUt |d krVqD| d }|d |d }} | ||<qDWdn1sswY|dd} | sJd| t| d 8}i} |D])}|}| d st |d krq| d}t |dksJ|dd| |d <qWdn1swYt |ddD]} t | |ddrtd| dq|| } | d}|sJd|g}g}t|}t |D]}| d \}}| d|dd|}| dd }| |d}|dd}dddt| D}d d!dt| D}|s\td|qt|}t||d"|jd d#|||d$|||d |dd%d& }||||qWdn 1swYt|}t|}t||\}}t|||dur||d'| d(||d)| d(||d*|| <q|S)+a Returns the manifests which consist of the Recordings and Supervisions. When all the manifests are available in the ``output_dir``, it will simply read and return them. :param corpus_dir: Pathlike, the path of the data dir. :param output_dir: Pathlike, the path where to write the manifests. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'audio' and 'supervisions'. zNo such directory: NTraishell3) dataset_partsr1prefixz spk-info.txtzNo such file: r#r rzlabel_train-set.txt|zPreparing aishell3 parts)desc)partr1r4zaishell3 subset: z already prepared - skipping.z content.txtwavr)NNcS g|] \}}|ddkr|qS)r8rr-.0ixr-r-r.  z$prepare_aishell3.. cSrA)r8r;r-rBr-r-r.rFrGgChinesefemale)pinyin tones_pinyin tones_text) id recording_idstartdurationchannellanguagespeakergendertextcustomaishell3_supervisions_z .jsonl.gzaishell3_recordings_) recordings supervisions) ris_dirr2r!rr"r& readlinesstrip startswithlensplitrrr#r$getjoin enumeratewarningr from_filer rQappendrfrom_recordingsr from_segmentsr r to_file)r0r1r3 manifests speaker_infospeaker_info_pathfkrTrU label_path tone_labelsr= part_path scripts_pathrZr[linerNrV audio_pathtonesrK recordingsegment recording_setsupervision_setr-r-r.prepare_aishell3Fs                     r{)rFr)N)!__doc__r#shutilr%pathlibrtypingrrrrrlhotserr r r lhotse.audior lhotse.qar lhotse.recipes.utilsrr lhotse.utilsrrrr2boolstrr/r{r-r-r-r.sB     "