o Si @s dZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z ddl mZddlmZmZddlmZmZdd lmZmZdd lmZmZmZd efd d Z   ddedededefddZ ddede ede ee ee eeffffddZ dS)aG Magicdata is an open-source Chinese Mandarin speech corpus by Magic Data Technology Co., Ltd., containing 755 hours of scripted read speech data from 1080 native speakers of the Mandarin Chinese spoken in mainland China. The sentence transcription accuracy is higher than 98%. Publicly available on https://www.openslr.org/68 N) defaultdict)Path)DictOptionalUnion)tqdm) fix_manifests$validate_recordings_and_supervisions) Recording RecordingSet)SupervisionSegmentSupervisionSet)Pathlikeresumable_download safe_extractlinecCsh|dd}|dd}|dd}|dd}|dd}|dd}|dd}|d d}|d d}|d d}|d d}|d d}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|}|S)u Modified from https://github.com/wenet-e2e/wenet/blob/main/examples/multi_cn/s0/local/magicdata_data_prep.sh#L41 sed 's/！//g' | sed 's/？//g' |\ sed 's/，//g' | sed 's/－//g' |\ sed 's/：//g' | sed 's/；//g' |\ sed 's/　//g' | sed 's/。//g' |\ sed 's/`//g' | sed 's/,//g' |\ sed 's/://g' | sed 's/?//g' |\ sed 's/\///g' | sed 's/·//g' |\ sed 's/\"//g' | sed 's/“//g' |\ sed 's/”//g' | sed 's/\\//g' |\ sed 's/…//g' | sed "s///g" |\ sed 's/、//g' | sed "s///g" | sed 's/《//g' | sed 's/》//g' |\ sed 's/\[//g' | sed 's/\]//g' | sed 's/FIL//g' | sed 's/SPK//' |\ tr '[a-z]' '[A-Z]' |\ u！u？u，u－u：u；u　u。`,:?/·"u“u”\u…u、z[ []u《 u《u》FILSPKu)replaceupper)rr!L/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/magicdata.pytext_normalizes>                             r#.F http://www.openslr.org/resources target_dirforce_downloadbase_urlreturnc Cs|d}t|}|jddd|d}d}d}d}|||fD]U}||} ||dd } | d } | rBtd |d | d qt|d|| |dtj| ddt | } t | |dWdn1skwY| q|S)aR Downdload and untar the dataset :param target_dir: Pathlike, the path of the dir to storage the dataset. :param force_download: Bool, if True, download the tars no matter if the tars exist. :param base_url: str, the url of the OpenSLR resources. :return: the path to downloaded and extracted directory with data. z/68Tparentsexist_ok magicdataztrain_set.tar.gzzdev_set.tar.gzztest_set.tar.gzNiz .completedzSkipping download z because z exists.r)filenamer') ignore_errors)path) rmkdiris_filelogginginforshutilrmtreetarfileopenrtouch) r&r'r(url corpus_dirtrain_tar_name dev_tar_name test_tar_nametar_nametar_path extracted_dircompleted_detectortarr!r!r"download_magicdataKs2   rDr; output_dirc Csbt|}|sJd||durt|}|jdddgd}i}|D]^}||d}t|ddd F}|D];}|d rItd |q9|}|d d ksY|d dkrZq9|d dd } d|dd} t | } | || <q9Wdn1swYq&t t } t |ddD]}td|g} g} ||}| dD]N}|j}|jd}||vrtd|t|dq||}|std|qt|}| |t||d|jd d||d}| |qt| }t| }t||\}}t|||dur'||d|d||d|d||d | |<q| S)!aL Returns the manifests which consist of the Recordings and Supervisions :param corpus_dir: Pathlike, the path of the data dir. :param output_dir: Pathlike, the path where to write the manifests. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'recordings' and 'supervisions'. zNo such directory: NTr*)traindevtestz TRANS.txtrzutf-8)encoding UtteranceIDzline is rz16_4013_20170819121429.wavz18_1565_20170712000170.wavr$ z/Process magicdata audio, it takes 6818 seconds.)desczProcessing magicdata subset: z**/*.wavzNo transcript: z has no transcript.zNo such file: gChinese)id recording_idstartdurationchannellanguagespeakertextmagicdata_supervisions_z .jsonl.gzmagicdata_recordings_) recordings supervisions)ris_dirr1r8 startswithr3r4splitjoinr#rdictrrglobstempartswarningr2r from_fileappendr rTstripr from_recordingsr from_segmentsrr to_file)r;rE dataset_partstranscript_dictpartr0f1ridx_transcriptidx_content manifestsr[r\wav_path audio_pathidxrWrX recordingsegment recording_setsupervision_setr!r!r"prepare_magicdatars              r{)r$Fr%)N)!__doc__r3osr5r7 collectionsrpathlibrtypingrrr tqdm.autorlhotserr lhotse.audior r lhotse.supervisionr r lhotse.utilsrrrstrr#boolrDr{r!r!r!r"sD   3 (