o 2wi)%@s*dZddlZddlZddlZddlmZddlmZddlm Z m Z m Z ddl m Z ddl mZddlmZmZdd lmZdd lmZdd lmZmZdd lmZmZmZmZ  ddede ede edefddZ    d dede ede ededede ee ee eeffff ddZ!dS)!al The AliMeeting Mandarin corpus, originally designed for ICASSP 2022 Multi-channel Multi-party Meeting Transcription Challenge (M2MeT), is recorded from real meetings, including far-field speech collected by an 8-channel microphone array as well as near-field speech collected by each participants' headset microphone. The dataset contains 118.75 hours of speech data in total, divided into 104.75 hours for training (Train), 4 hours for evaluation (Eval) and 10 hours as test set (Test), according to M2MeT challenge arrangement. Specifically, the Train, Eval and Test sets contain 212, 8 and 20 meeting sessions respectively, and each session consists of a 15 to 30-minute discussion by 2-4 participants. AliMeeting covers a variety of aspects in real-world meetings, including diverse meeting rooms, various number of meeting participants and different speaker overlap ratios. High-quality transcriptions are provided as well. The dataset can be used for tasks in meeting rich transcriptions, including speaker diarization and multi-speaker automatic speech recognition. More details and download link: https://openslr.org/119/ N) defaultdict)Path)DictOptionalUnion)tqdm)$validate_recordings_and_supervisions) Recording RecordingSet) fix_manifests)normalize_text_alimeeting)SupervisionSegmentSupervisionSet)Pathlikeis_module_availableresumable_download safe_extract.F;https://speech-lab-share-data.oss-cn-shanghai.aliyuncs.com/ target_dirforce_downloadbase_urlreturnc Cs|d}t|}|jdddgd}|D]-}||}t|d|||dt|}t||dWdn1s>wYq|S) aR Downdload and untar the dataset :param target_dir: Pathlike, the path of the dir to storage the dataset. :param force_download: Bool, if True, download the tars no matter if the tars exist. :param base_url: str, the url of the OpenSLR resources. :return: the path to downloaded and extracted directory with data. z/AliMeeting/openlrTparentsexist_ok)zTrain_Ali_far.tar.gzzTrain_Ali_near.tar.gzzEval_Ali.tar.gzzTest_Ali.tar.gz/)filenamer)pathN)rmkdirrtarfileopenr)rrrurldataset_tar_namestar_nametar_pathtarr'W/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/lhotse/recipes/ali_meeting.pydownload_ali_meeting$s  r)farnone corpus_dir output_dirmicnormalize_text save_monoc#Cstdstdddl}|r|dkrtdd}|r!|s!td|}|d vr)d nd }t|}|s:Jd |tt}|durMt|}|j d d ddD]}|ra|d|} | j d d dg} g} |} |dkso|dkr||dr||dn|} | |d|d} | |d|d}t t | dd|dD]}|j }|d kr|d\}}}}|dd}z |jt|}Wntyt|dYqwt | |dd}|r | |j}|sd|d|}tj|d d d tj||d!}ntj||d!}| ||jD]x}|d krE|jd}t|d"kr4|\}}}}n t|d#kr?|\}}|dd}t|jD]E\}}|jd$kr|j}|j}|j}t |d%|d%||j!|t"||d"|d&vrwdnt t#d'd(||t$|%|d)d* } | | qJqqt&t'(| t)*| \}!}"t+|!|"|dur|",|d+|d,|-d-|!,|d+|d.|-d-|!|"d/||-<qO|S)0az Returns the manifests which consist of the Recordings and Supervisions :param corpus_dir: Pathlike, the path of the data dir. :param output_dir: Pathlike, the path where to write the manifests. :param mic: str, "near" or "far", specifies whether to prepare the near-field or far-field data. May also specify "ihm", "sdm", "mdm" (similar to AMI recipe), where "ihm" and "mdm" are the same as "near" and "far" respectively, and "sdm" is the same as "far" with a single channel. :param normalize_text: str, the text normalization type. Available options: "none", "m2met". :param save_mono: bool, if True, save the mono recordings for sdm mic. This can speed up feature extraction since all channels will not be loaded. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'recordings' and 'supervisions'. textgridz@To prepare AliMeeting data, please 'pip install textgrid' first.rNsdmzCsave_mono is True, but mic is not 'sdm'. Ignoring save_mono option.Fzjsave_mono is True, but output_dir is not specified. Please specify output_dir to save the mono recordings.)ihmnearr4r*zNo such directory: Tr)TrainEvalTestalimeeting_sdmr6r7_Ali_Ali_ audio_dir textgrid_dirz *.TextGridz Preparing )desc_z0 has annotation issues. Skipping this recording.z*.wavzsox z -c 1 )shellcheck) recording_id-)r4r3r2Chinese) normalize) idrBstartdurationchannellanguagespeakergendertextz alimeeting-_supervisions_z .jsonl.gz _recordings_) recordings supervisions).r ValueErrorr1loggingwarningris_dirrdictrrlistrglobstemsplitTextGridfromFilestrnameis_file subprocessrunr from_fileappendtierslen enumerate intervalsmarkminTimemaxTimer rJroundranger stripr r from_recordingsr from_segmentsrto_filelower)#r,r-r.r/r0r1mic_orig manifestspartoutput_dir_monorTrUcorpus_dir_split wav_paths text_paths text_path session_idr>rPspk_idtgwav_path wav_path_monocmd recordingtierpartsiintervalrKendrQsegment recording_setsupervision_setr'r'r(prepare_ali_meetingFs                      r)rFr)Nr*r+F)"__doc__rWrdr collectionsrpathlibrtypingrrrrlhotser lhotse.audior r lhotse.qar lhotse.recipes.utilsr lhotse.supervisionr r lhotse.utilsrrrrboolrar)rr'r'r'r(sZ       $