o 2wip@sdZddlZddlmZddlmZddlmZmZm Z ddl m Z ddl m Z ddlmZmZdd lmZdd lmZmZdd lmZd ed efddZ  ddedeeded eeeee eeffffddZdS)zM AISHELL2 (~1000 hours) if available(https://www.aishelltech.com/aishell_2). N) defaultdict)Path)DictOptionalUnion)tqdm)$validate_recordings_and_supervisions) Recording RecordingSet) fix_manifests)SupervisionSegmentSupervisionSet)PathlikelinereturncCs|dd}|dd}|dd}|dd}|d d }|d d }|d d }|} g}t|}t|D]#\}}|dkrWd||dkrOdkrWnn|dd }||q9d |}|}|S)zz Modified from https://github.com/wenet-e2e/wenet/blob/main/examples/multi_cn/s0/local/aishell2_data_prep.sh#L50 uＡAuＴTuＭMu𫖯u頫u，?- 'u一u鿿)replaceupperlist enumerateappendjoin)rnew_lineicharr#T/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/lhotse/recipes/aishell2.pytext_normalizes$       /(   r%r corpus_dir output_dirnum_jobsc CsVt|}|sJd||durt|}|jdddtt}gd}t|ddD]}td||d krO|d d d d }|d d d d}n|d d |d }|d d |d}i}t|ddd$} | D]} | } d | dd} t | } | || d<qqWdn1swYg} t j |d|d}|dD]F}|j}|jd}||vrtd|t|dq||}|std|qt||d||dd||d}| |qt |}t| }t||\}}t|||dur!||d|d ||d!|d ||d"||<q,|S)#aY Returns the manifests which consist of the Recordings and Supervisions :param corpus_dir: Pathlike, the path of the data dir. :param output_dir: Pathlike, the path where to write the manifests. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'recordings' and 'supervisions'. zNo such directory: NT)parentsexist_ok)traindevtestzEProcess aishell2 audio, it takes about 55 minutes using 40 cpu jobs.)desczProcessing aishell2 subset: r+z AISHELL-2iOSdataz trans.txtwavrzutf-8)encodingrrrz*.wav)pathpatternr(z**/*.wavzNo transcript: z has no transcript.zNo such file: gChinese)id recording_idstartdurationchannellanguagespeakertextaishell2_supervisions_z .jsonl.gzaishell2_recordings_) recordings supervisions)ris_dirmkdirrdictrlogginginfoopensplitrr%r from_dirrglobstempartswarningis_filer r;striprfrom_recordingsr from_segmentsr rto_file)r&r'r( manifests dataset_partsparttranscript_pathwav_pathtranscript_dictfridx_transcriptcontentrCrB audio_pathidxr>r?segment recording_setsupervision_setr#r#r$prepare_aishell2Zs~       rc)Nr)__doc__rG collectionsrpathlibrtypingrrr tqdm.autorlhotser lhotse.audior r lhotse.qar lhotse.supervisionr r lhotse.utilsrstrr%intrcr#r#r#r$s.      I