o Si+@sdZddlmZddlmZddlmZmZddlm Z ddl m Z m Z ddl mZddlmZdd lmZmZdd lmZmZmZmZmZdd lmZdd lmZmZdd lm Z m!Z!m"Z"m#Z# d9de ddfddZ$    d:de de de%de%de&de&dee'ee'eeeffffddZ(d Z)d!Z*d"d#e+e)e*DZ,d$e'de'fd%d&Z-d'e'de'fd(d)Z.d'e'de'fd*d+Z/d'e'de'fd,d-Z0d'e'de'fd.d/Z1d'e'de'fd0d1Z2d'e'de'fd2d3Z3d'e'de'fd4d5Z4d6e'de&ddfd7d8Z5dS);a Description taken from official website: https://arabicspeech.org/mgb2/ The Multi-Dialect Broadcast News Arabic Speech Recognition (MGB-2): The second edition of the Multi-Genre Broadcast (MGB-2) Challenge is an evaluation of speech recognition and lightly supervised alignment using TV recordings in Arabic. The speech data is broad and multi-genre, spanning the whole range of TV output, and represents a challenging task for speech technology. In 2016, the challenge featured two new Arabic tracks based on TV data from Aljazeera. It was an official challenge at the 2016 IEEE Workshop on Spoken Language Technology. The 1,200 hours MGB-2: from Aljazeera TV programs have been manually captioned with no timing information. QCRI Arabic ASR system has been used to recognize all programs. The ASR output was used to align the manual captioning and produce speech segments for training speech recognition. More than 20 hours from 2015 programs have been transcribed verbatim and manually segmented. This data is split into a development set of 10 hours, and a similar evaluation set of 10 hours. Both the development and evaluation data have been released in the 2016 MGB challenge )chaininfo)pathsystem)Path)matchsub)copy) punctuation)DictUnion) RecordingSetSupervisionSegmentSupervisionSet fix_manifests$validate_recordings_and_supervisions)load_kaldi_data_dir)manifests_existread_manifests_if_cached)Pathlikecheck_and_rglobis_module_availablerecursion_limit. target_dirreturnNcCs tddS)z Download and untar the dataset. NOTE: This function just returns with a message since MGB2 is not available for direct download. :param target_dir: Pathlike, the path of the dir to storage the dataset. z{MGB2 is not available for direct download. Please fill out the format https://arabicspeech.org/mgb2 to download the corpus.Nr)rrG/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/mgb2.py download_mgb2)s rTFP corpus_dir output_dir text_cleaning buck_walternum_jobs mer_threshc st|}|sJd|gd}i}|dur-t|}|jdddt||dddd}|D]T}td |t||ddd rJtd |d q/t|}t|}|d ksZ|dkr||jdddt||d||dt||d||dt||dd5} t||dd} | D]} | | d|d|dqWdn1swYWdn1swYt ||d\} } }|dur| t } |d krt | dksJdt | nq|dkrt | dksJdt | n]|dkr[tj||d d!|d"} tt||d#d$}td%ttfd&d'|D}Wdn 1s7wYt|} t | d(ksQJd)t | |dur[| t} t| | \} } t| | | |d*|d+| |d,|d+| | d-||<q/|S).a Returns the manifests which consist of the Recordings and Supervisions. When all the manifests are available in the ``output_dir``, it will simply read and return them. :param corpus_dir: Pathlike, the path of the data dir. :param output_dir: Pathlike, the path where to write the manifests. :param text_cleaning: Bool, if True, basic text cleaning is performed (similar to ESPNet recipe). :param buck_walter: Bool, use BuckWalter transliteration :param num_jobs: int, the number of jobs to use for parallel processing. :param mer_thresh: int, filter out segments based on mer (Match Error Rate) :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'audio' and 'supervisions'. .. note:: Unlike other recipes, output_dir is not Optional here because we write the manifests to the output directory while processing to avoid OOM issues, since it is a large dataset. .. caution:: The `text_cleaning` option removes all punctuation and diacritics. zNo such directory: )devtraintestNT)parentsexist_okmgb2zjsonl.gz) dataset_partsr#prefixsuffixlazyzProcessing MGB2 subset: )partr#r/r0z MGB2 subset: z already prepared - skipping.r*r(ztext.non_overlap_speechtextzsegments.non_overlap_speechsegmentszwav.scprwzwav//z/wav/i>Fiz+Expected 5365 supervisions for test, found iz*Expected 5002 supervisions for dev, found r)wavz*.wav)patternr&zxml/utf8z*.xmlicsg|]}t|qSr)make_supervisions).0pr'rr sz prepare_mgb2..i?z.Expected 375103 supervisions for train, found mgb2_recordings_z .jsonl.gzmgb2_supervisions_) recordings supervisions)ris_dirmkdirrrrr openwritereplacertransform_textfrom_buck_walterlenrfrom_dirrrjoinrlistr from_iterabler from_segmentscleaningrrto_file)r"r#r$r%r&r'r. manifestsr2f_inf_outlinerArB_ xml_pathssupervisions_listrr=r prepare_mgb2:s                   rYufآؤئبتجگخذزشضظغـقلنويٌَِْٰپچءأإڤاةثحدرسصطعفكمهىًٍُّٱz3|&}btjGx*z$DZg_qlnwyNaio`PJ'>sr]scCs |tSN) translate _backward_map)r^rrrrIs rIr3cCs tdd|S)Nz/[\u064B-\u0652\u06D4\u0670\u0674\u06D5-\u06ED]+r r3rrrremove_diacriticss recCs6d}t}t||}|D] }||vr||d}q |S)z;This function removes all punctuations except the verbatimu:﴿﴾`÷×؛<>_()*&^%][ـ،/:"؟.,'{}~¦+|!”…“–ـ )r setrG)r3arabic_punctuationsenglish_punctuationsall_punctuationsr<rrrremove_punctuationss  rkcCs|}tdd|S)Nz[^\u0600-\u06FF\s\da-z]+rb)lowerr rdrrrremove_non_alphanumerics rmcCs |}dd|D}d|S)z Remove single character word from text Example: I am in a a home for two years => am in home for two years Args: text (str): text Returns: (str): text with single char removed cSs$g|]}t|dks|r|qS)r )rJ isnumeric)r;wordrrrr>s$z+remove_single_char_word..rf)splitrL)r3words filter_wordsrrrremove_single_char_words  rscCs6ddddddddd d d d d d d}t|}||S)N0123456789%rfuف)u٠u١u٢u٣u٤u٥u٦u٧u٨u٩u٪rVuڤ|)str maketransr`)r3eastern_to_western trans_stringrrreast_to_west_nums"  rcCstdd|}tdd|}|S)Nz\s+rfz\s+\.\s+rrcrdrrrremove_extra_spaces  rcCs4t|}t|}t|}t|}t|}t|}|Sr_)rkrrermrsrrdrrrrP srPxml_pathcsHtdstdddlm}t|d}||d}fdd|d DS) Nbs4z@To prepare MGB2 data, please 'pip install beautifulsoup4' first.r) BeautifulSoupr5xmlcsg|]Z}dust|dkrt|dd|dd|d|ddddd t|dtt|dt|dd d dd d d|dDdttd|d ddqS)NWMERidrV starttime:endtime_uttr-)ndigitsrfcSsg|] }|jdur|jqSr_)string)r;elementrrrr>)s  z0make_supervisions...rArabicz\w+speaker(\d+)\w+whor )r recording_idstartdurationchannelr3languagespeaker) floatrrprGroundrLfind_allintrgroup)r;segmentr=rrr>s( z%make_supervisions..r)r ValueErrorrrrEr)rr'r xml_handlesouprr=rr:s    r:)r)TFr r!)6__doc__ itertoolsrloggingrosrrpathlibrrerr shutilr rr typingr r lhotserrrrr lhotse.kaldirlhotse.recipes.utilsrr lhotse.utilsrrrrrboolrrrY_unicode _buckwalterziprarIrerkrmrsrrrPr:rrrrsh