o Si@sdZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z ddl mZmZddlmZdd lmZmZdd lmZmZd Zgd ZddededefddZ  ddedeeeefdedeeeeeeeffffddZdedefddZdS)aW Multi-Domain Cantonese Corpus (MDCC), consists of 73.6 hours of clean read speech paired with transcripts, collected from Cantonese audiobooks from Hong Kong. It comprises philosophy, politics, education, culture, lifestyle and family domains, covering a wide range of topics. Manuscript can be found at: https://arxiv.org/abs/2201.02419 N)Path)DictSequenceUnion)tqdm)$validate_recordings_and_supervisions) Recording RecordingSet) fix_manifests)SupervisionSegmentSupervisionSet)Pathlikeis_module_availablezFhttps://drive.google.com/file/d/1epfYMMhXdBKA6nxPgUugb2Uj4DllSxkn/view)trainvalidtestF target_dirforce_downloadreturncCstdstdddl}t|}|jddd|d}|d}|s/|r/t|d ntd t |j t t |dd d |rH|rotd |d|t |}|j|dWd|S1sjwY|S)a5 Downloads the MDCC data from the Google Drive and extracts it. :param target_dir: the directory where MDCC data will be saved. :param force_download: if True, it will download the MDCC data even if it is already present. :return: the path to downloaded and extracted directory with data. gdownz0Please run 'pip install gdown' to download MDCC.rNT)parentsexist_okdatasetz.zipz# already exists. Skipping download.zRunning: gdown --fuzzy F)fuzzyquietz Extracting z to path)r ValueErrorrrmkdir with_suffixexistslogginginfoMDCC_URLdownloadstrzipfileZipFile extractall)rrr corpus_dir corpus_zipzfr,G/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/mdcc.py download_mdccs&     r.allr) dataset_parts output_dirc Cst|}|d}|sJd|d|di}|dks#|ddkr&t}nt|tr9|tvs6Jd||g}|D]}g}g}|d|d }|sXJd|d |dt|d } | } | d d } Wd n1srwYt| d|ddD]@} | d\} } }}|t| j } || } t t| }t j| |d}||t||d|jd|  |dd}||qt|}t|}t||\}}t||d|d urt|}|jddd||d|d||d|d||d||<q;|S)ac Create RecordingSet and SupervisionSet manifests for MDCC from a raw corpus distribution. :param corpus_dir: Pathlike, the path to the extracted corpus. :param output_dir: Pathlike, the path where to write the manifests. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'audio' and 'supervisions'. audiozMissing z in .r/rzUnknown dataset part: cnt_asr_z _metadata.csvz metadata in rNz Processing z metadata)desc,) recording_idgyue)idr9startdurationchanneltextgenderlanguage) recordings supervisionsT)rrmdcc_recordings_z .jsonl.gzmdcc_supervisions_)ris_dir MDCC_PARTS isinstancer%is_fileopen readlinesrstripsplitnamemake_recording_idr from_fileappendr r= read_textr from_recordingsr from_segmentsr rrto_file)r)r0r1 audio_dir manifestspartrBrCmetadataflinesline audio_path text_pathr@_r9 recordingsupervision_segmentr,r,r- prepare_mdcc;sb        rbrcCs d|jS)Nmdcc_)stemrr,r,r-rOs rO)F)r/N)__doc__r!r&pathlibrtypingrrr tqdm.autorlhotser lhotse.audiorr lhotse.qar lhotse.supervisionr r lhotse.utilsr rr#rGboolr.r%rbrOr,r,r,r-s4    " N