o Si@sdZddlZddlZddlmZddlmZddlmZm Z m Z ddl m Z ddl mZmZddlmZmZdd lmZmZdd lmZd efd d Z  ddede ededeeeee eeffffddZdS)zy optional TAL_CSASR(587 hours) if available(https://ai.100tal.com/dataset). It is a mandarin-english code-switch corpus. N) defaultdict)Path)DictOptionalUnion)tqdm) fix_manifests$validate_recordings_and_supervisions) Recording RecordingSet)SupervisionSegmentSupervisionSet)PathlikelinecCs|dd}|dd}|dd}|dd}|d d }|d d }|d d}|dd}|dd}tdd|}|dd}|dd}|}|S)us Modified from https://github.com/wenet-e2e/wenet/blob/main/examples/multi_cn/s0/local/tal_mix_data_prep.sh#L52 sed 's/Ａ/A/g' | sed 's/Ｃ/C/g' | sed 's/Ｄ/D/g' | sed 's/Ｇ/G/g' | sed 's/Ｈ/H/g' | sed 's/Ｕ/U/g' | sed 's/Ｙ/Y/g' | sed 's/ａ/a/g' | sed 's/Ｉ/I/g' | sed 's/#//g' | sed 's/=//g' | sed 's/；//g' | sed 's/，//g' | sed 's/？//g' | sed 's/。//g' | sed 's/\///g' | sed 's/！//g' | sed 's/!//g' | sed 's/\.//g' | sed 's/\?//g' | sed 's/：//g' | sed 's/,//g' | sed 's/"//g' | sed 's/://g' | sed 's/@//g' | sed 's/-/ /g' | sed 's/、/ /g' | sed 's/~/ /g' | sed "s/‘/'/g" | sed 's/Ｅ/E/g' | sed "s/’/'/g" | sed 's/《//g' | sed 's/》//g' | sed "s/[ ][ ]*$//g" | sed "s/\[//g" | sed 's/、//g' 210_40223_210_6228_1_1533298404_4812267_555 上面是一般现在对然后然后下面呢 HE IS ALWAYS FINISHIＮG uＡAuＣCuＤDuＧGuＨHuＵUuＹYuａauＩIuP#|[=]|；|，|？|。|[/]|！|[!]|[.]|[?]|：|,|"|:|@|-|、|~|《|》|[|]|、|\.uＥEuＮN)replaceresubupper)rr L/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/tal_csasr.pytext_normalizes           r" corpus_dir output_dirnum_jobsreturnc Cs$t|}|sJd||durt|}|jdddgd}i}|D]@}|d|d}t|dd d &}|D]}|} d | d d} t| } | || d <q=Wdn1sawYq&tt } t |ddD]}t d|g} |d|d} t j| d|d}| dD]C}|j}|}||vrt d|t |dq||}|st d|qt||d||d d||d}| |qt |}t| }t||\}}t|||dur||d|d||d|d||d| |<qq| S)aL Returns the manifests which consist of the Recordings and Supervisions :param corpus_dir: Pathlike, the path of the data dir. :param output_dir: Pathlike, the path where to write the manifests. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'recordings' and 'supervisions'. zNo such directory: NT)parentsexist_ok) train_setdev_settest_set TALCS_corpusz label.txtrzutf-8)encoding r#rzDProcess tal_csasr audio, it takes about 4 minutes using 40 cpu jobs.)desczProcessing tal_csasr subset: wavz*.wav)pathpatternr&z**/*.wavzNo transcript: z has no transcript.zNo such file: gChinese)id recording_idstartdurationchannellanguagespeakertexttal_csasr_supervisions_z .jsonl.gztal_csasr_recordings_) recordings supervisions)ris_dirmkdiropen readlinessplitjoinr"rdictrlogginginfor from_dirrglobstemwarningis_filer r9stripappendfrom_recordingsr from_segmentsrr to_file)r$r%r& dataset_partstranscript_dictparttranscript_pathfridx_transcriptcontent manifestsrAwav_pathr@ audio_pathidxr<r=segment recording_setsupervision_setr r r!prepare_tal_csasr3s|       rc)Nr#)__doc__rIr collectionsrpathlibrtypingrrr tqdm.autorlhotserr lhotse.audior r lhotse.supervisionr r lhotse.utilsrstrr"intrcr r r r!s.    "