o Si @s dZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z ddl mZddlmZmZddlmZmZdd lmZmZdd lmZmZmZd efd d Z   ddedededefddZ ddede ede ee ee eeffffddZ dS)z Stcmds is an open-source Chinese Mandarin corpus by Surfingtech (www.surfing.ai), containing utterances from 855 speakers, 102600 utterances; Publicly available on https://www.openslr.org/resources/38 ST-CMDS (110 hours) N) defaultdict)Path)DictOptionalUnion)tqdm) fix_manifests$validate_recordings_and_supervisions) Recording RecordingSet)SupervisionSegmentSupervisionSet)Pathlikeresumable_download safe_extractlinecCs|dd}|}|S)u Modified from https://github.com/wenet-e2e/wenet/blob/main/examples/multi_cn/s0/local/stcmds_data_prep.sh#L42 paste -d' ' $data/utt.list $data/text.list | sed 's/，//g' | tr '[a-z]' '[A-Z]' | awk '{if (NF > 1) print $0;}' > $data/train/text u，)replaceupper)rrI/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/stcmds.pytext_normalizes r.F http://www.openslr.org/resources target_dirforce_downloadbase_urlreturnc Cs|d}t|}|jddd|d}d}|fD]R}||}||dd}|d} | r9td | d qt|d |||d tj|dd t |} t | |dWdn1sbwY| q|S)aR Downdload and untar the dataset :param target_dir: Pathlike, the path of the dir to storage the dataset. :param force_download: Bool, if True, download the tars no matter if the tars exist. :param base_url: str, the url of the OpenSLR resources. :return: the path to downloaded and extracted directory with data. z/38Tparentsexist_okstcmdszST-CMDS-20170001_1-OS.tar.gzNiz .completedzSkipping download of because z exists./)filenamer) ignore_errors)path) rmkdiris_filelogginginforshutilrmtreetarfileopenrtouch) rrrurl corpus_dirdataset_tar_nametar_nametar_path extracted_dircompleted_detectortarrrrdownload_stcmds$s*    r7r0 output_dirc Cst|}|sJd||durt|}|jddd|d}i}|dD]1}|j}td|t|dd d }|D] }t|}|||<q@Wdn1sUwYq)t t }d g} t | d d D]} td| g} g} |dD]T} | j}d t |dd}||vrtd|t| dq{||}| std| q{t| }| |t||d|jdd||d}| |q{t| }t| }t||\}}t|||dur||d| d||d| d||d|| <qh|S)aL Returns the manifests which consist of the Recordings and Supervisions :param corpus_dir: Pathlike, the path of the data dir. :param output_dir: Pathlike, the path where to write the manifests. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'recordings' and 'supervisions'. zNo such directory: NTrzST-CMDS-20170001_1-OSz**/*.txtzprocessing stcmds transcript rzutf-8)encodingtrainz=process stcmds audio, it needs waste about 2169 seconds time.)desczProcessing stcmds z**/*.wavrzNo transcript: z has no transcriptzNo such file: grChinese)id recording_idstartdurationchannellanguagespeakertextstcmds_supervisions_z .jsonl.gzstcmds_recordings_) recordings supervisions)ris_dirr&rglobstemr(r)r-rrdictrjoinlistwarningr'r from_fileappendr rCstripr from_recordingsr from_segmentsrr to_file)r0r8r%transcript_dict text_pathidxfr manifests dataset_partspartrJrK audio_pathrFrG recordingsegment recording_setsupervision_setrrrprepare_stcmdsGsv         re)rFr)N)!__doc__r(osr*r, collectionsrpathlibrtypingrrr tqdm.autorlhotserr lhotse.audior r lhotse.supervisionr r lhotse.utilsrrrstrrboolr7rerrrrsD    $