o Sir @sdZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZddlmZmZddlmZmZdd lmZmZdd lmZmZdd lmZd Z d(de!de!de e!e!ffddZ"    d)dede e!e e!fde ede#de!dee!ee!e eeffff ddZ$   d*de e!efde e!efde e#d e e#d!e e#def d"d#Z% d(ded$e!d%e!de!de e eeff d&d'Z&dS)+a KsponSpeech is a large-scale spontaneous speech corpus of Korean. This corpus contains 969 hours of open-domain dialogue utterances, spoken by about 2,000 native Korean speakers in a clean environment. All data were constructed by recording the dialogue between two people freely conversing on a variety of topics and manually transcribing the utterances. The transcription provides a dual transcription consisting of orthography and pronunciation, and disfluency tags for the spontaneity of speech, such as filler words, repeated words, and word fragments. The original audio data has a PCM extension. During preprocessing, it is converted into a file in the FLAC extension and saved anew. KsponSpeech is publicly available on an open data hub site of the Korea government. The dataset must be downloaded manually. For more details, please visit: Dataset: https://aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=123 Paper: https://www.mdpi.com/2076-3417/10/19/6936 N)ThreadPoolExecutor)Path)DictListOptionalSequenceTupleUnion)tqdm) fix_manifests$validate_recordings_and_supervisions) Recording RecordingSet)manifests_existread_manifests_if_cached)SupervisionSegmentSupervisionSet)Pathlike)traindev eval_clean eval_otherdefault raw_contentnormalize_textreturncCst|dkrdS|d\}}|dkr||fS|dkrHtdd|}tdd|}|d d}|d d}|d d}td d |}||fSdS)a Normalizing KsponSpeech text datasets with '.trn' extension. Perform the following processing. 1. Separate file name and text labeling from raw content using separator ' :: '; 2. Remove noise labeling characters (e.g. `o/`, `b/`...); 3. Remove the actual pronunciation from the text labeling; use the spelling content; 4. Remove other special characters and double spaces from text labeling. :param raw_content: a raw text labeling content containing file name and text labeling. :param normalize_text: str, the text normalization type, "default" or "none". :return: a tuple with file name and normalized text labeling. rz :: nonerz[a-z]/z\((.*?)\)/\((.*?)\)z\1*+/z\s+ N)lensplitresubreplacestrip)rroriginal_content_idcontentr*N/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/ksponspeech.py normalize/s     r,all corpus_dir dataset_parts output_dirnum_jobsc Cst|}|sJd||dkrtt}nt|tr |g}i}|dur:t|}|jdddt||dddd}t|}t |d d D]}t d |t ||ddd rat d |dqEg}g} g} ||d} t | } | D]} | |t||| |quWdn1swYt | dddD]}|}|durq|\}}||| |qt|}t| }t||\}}t|||dur||d|d||d|d||d||<qEWd|S1swY|S)a Returns the manifests which consist of the Recordings and Supervisions. When all the manifests are available in the ``output_dir``, it will simply read and return them. :param corpus_dir: Pathlike, the path of the data dir. :param dataset_parts: string or sequence of strings representing dataset part names, e.g. 'train', 'dev'. By default, we will infer all parts. :param output_dir: Pathlike, the path where to write the manifests. :param num_jobs: int, number of parallel threads used for 'parse_utterance' calls. :param normalize_text: str, the text normalization type, "default" or "none". :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'audio' and 'supervisions'. zNo such directory: r-NT)parentsexist_ok ksponspeechzjsonl.gz)r0r1prefixsuffixlazyz Dataset parts)desczProcessing KsponSpeech subset: )partr1r6r7zKsponSpeech subset: z already prepared - skipping.z.trn ProcessingF)r9leaveksponspeech_supervisions_z .jsonl.gzksponspeech_recordings_) recordings supervisions)ris_dirset KSPONSPEECH isinstancestrmkdirrrr logginginforopenappendsubmitparse_utteranceresultrfrom_recordingsr from_segmentsr r to_file)r/r0r1r2r manifestsexr:r?r@futures trans_pathflinefuturerM recordingsegment recording_setsupervision_setr*r*r+prepare_ksponspeechSs            66r\>pcm_path flac_path sample_ratechannels bit_depthcCs>t|}t|}tj|||ddd\}}tj|||dd|S)NRAWPCM_16)rb samplerateformatsubtypeFLAC)rg)rsfreadwrite)r_r`rarbrcdata_r*r*r+ pcm_to_flacs ror:rVc Cst|}t||\}}d|vr|jdddd}||}|j}|s-td|dS|d}t||}t j ||d}t ||d|j d d |d } || fS) Nevalr r.)maxsplitzNo such file: z.flac) recording_idgrKorean)idrrstartdurationchannellanguagetext) rr,r#stemis_filerGwarning with_suffixror from_filerrv) r/r:rVr audio_pathnormalized_linerrr`rXrYr*r*r+rLs,   rL)r)r-Nr.r)r]r.r^)'__doc__rGr$concurrent.futures.threadrpathlibrtypingrrrrrr soundfilerj tqdm.autor lhotser r lhotse.audior rlhotse.recipes.utilsrrlhotse.supervisionrr lhotse.utilsrrCrEr,intr\rorLr*r*r*r+s        & f