o 2wi< @sdZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z ddl mZddlmZmZddlmZmZdd lmZmZdd lmZmZmZ dd ed efddZ ddede ed e ee ee eeffffddZdS)a About the XBMU-AMDO31 corpus XBMU-AMDO31 is an open-source Amdo Tibetan speech corpus published by Northwest Minzu University. publicly available on https://huggingface.co/datasets/syzym/xbmu_amdo31 XBMU-AMDO31 dataset is a speech recognition corpus of Amdo Tibetan dialect. The open source corpus contains 31 hours of speech data and resources related to build speech recognition systems,including transcribed texts and a Tibetan pronunciation lexicon. (The lexicon is a Tibetan lexicon of the Lhasa dialect, which has been reused for the Amdo dialect because of the uniformity of the Tibetan language) The dataset can be used to train a model for Amdo Tibetan Automatic Speech Recognition (ASR). It was recorded by 66 native speakers of Amdo Tibetan, and the recorded audio was processed and manually inspected. The dataset has three splits: train, evaluation (dev) and test.Each speaker had approximately 450 sentences, with a small number of individuals having fewer than 200 sen. Subset Hours Male Female Remarks Train 25.41 27 27 18539 sentences recorded by 54 speakers Dev 2.81 2 4 2050 sentences recorded by 6 speakers Test 2.85 3 3 2041 sentences recorded by 6 speakers Licensing Information This dataset is distributed under CC BY-SA 4.0. N) defaultdict)Path)DictOptionalUnion)tqdm) fix_manifests$validate_recordings_and_supervisions) Recording RecordingSet)SupervisionSegmentSupervisionSet)Pathlikeis_module_available safe_extract. target_dirreturnc CsDd}t|}|jddd|d}|dd}d}d}d }|sTtd r,d d lm}ntd tdz| ||Wnt yN}zt |d}~wwtd|||fD]F} || } || dd} | d} | rytd| dqYt j | ddt| } t| |dWdn1swY| qY|S)z Downdload and untar the dataset :param target_dir: Pathlike, the path of the dir to storage the dataset. :return: the path to downloaded and extracted directory with data. z1https://huggingface.co/datasets/syzym/xbmu_amdo31Tparentsexist_ok xbmu_amdo31datawavz train.tar.gzz dev.tar.gzz test.tar.gzgitr)Repoz}In order to download the xbmu-amdo31 corpus from huggingface, please install the relevant dependencies: pip install gitpythonz(Start downloading the xbmu-amdo31 corpusNDoneiz .completedzSkipping tar of because z exists.) ignore_errors)path)rmkdiris_filerrr ImportErrorlogginginfo clone_from Exceptionprintshutilrmtreetarfileopenrtouch)rurl corpus_dirwav_dirtrain_tar_name dev_tar_name test_tar_namererrortar_nametar_path extracted_dircompleted_detectortarr8W/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/lhotse/recipes/xbmu_amdo31.pydownload_xbmu_amdo31*sH     r:r- output_dirc Cs<t|}|sJd||durt|}|jddd|d}i}t|ddd"}|D]}|}d |d d}|||d <q0Wdn1sPwYtt}gd } t | d dD]} t d| g} g} |dd| } d }| dD]a}|j dd }|jd}||vrt d|t |dq||}|st d|qt|}| ||d 7}tt|d||d|d|jd d||d}| |qt| }t| }t||\}}t|||dur||d| d||d| d||d|| <qc|S) aL Returns the manifests which consist of the Recordings and Supervisions :param corpus_dir: Pathlike, the path of the data dir. :param output_dir: Pathlike, the path where to write the manifests. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'recordings' and 'supervisions'. zNo such directory: NTrz$data/transcript/transcript_clean.txtrzutf-8)encoding r)traindevtestzProcess xbmu_amdo31 audio.)desczProcessing xbmu_amdo31 subset: rrz**/*.wav-zNo transcript: z has no transcript.zNo such file: _gtibetan)id recording_idstartdurationchannellanguagespeakertextxbmu_amdo31_supervisions_z .jsonl.gzxbmu_amdo31_recordings_) recordings supervisions)ris_dirrr* readlinessplitjoinrdictrr"r#rglobstempartswarningr r from_fileappendr strrKstripr from_recordingsr from_segmentsrr to_file)r-r;transcript_pathtranscript_dictflineidx_transcriptcontent manifests dataset_partspartrRrSwav_pathcount audio_pathidxrNrO recordingsegment recording_setsupervision_setr8r8r9prepare_xbmu_amdo31Zs~           ru)r)N)__doc__r"osr'r) collectionsrpathlibrtypingrrr tqdm.autorlhotserr lhotse.audior r lhotse.supervisionr r lhotse.utilsrrrr:r_rur8r8r8r9s6    1