o }oiI @sddlZddlZddlZddlZddlmZmZmZgdZdZ dZ dZ de fdd Z dd e d efd dZdde dededee fddZdee deeeffddZd de de de fddZdS)!N)DictListUnion) read_wordidsset_grapheme_caseGRAPHEME_CASE_UPPERGRAPHEME_CASE_LOWERGRAPHEME_CASE_MIXEDget_heteronym_spansupperlowermixed wordid_mapc Cstj|s t|di}i}t|ddd@}tj|dd}t|D])\}}|dkr-q$|d}|d}|d } t|||<||vrGi||<| |||<q$Wd ||fS1s[wY||fS) ag Reads wordid file from WikiHomograph dataset, https://github.com/google-research-datasets/WikipediaHomographData/blob/master/data/wordids.tsv Args: wordid_map: path to wordids.tsv Returns: data_dict: a dictionary of graphemes with corresponding word_id - ipa_form pairs wordid_to_idx: word id to label id mapping not foundrutf-8encoding ) delimiterrN) ospathexists ValueErroropencsvreader enumeratelen) r data_dict wordid_to_idxftsv_fileilinegraphemeword_idipa_formr*R/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/tts/g2p/utils.pyr&s*   rTwordid_to_phonemes_fileto_lowercCstj|s t|di}t|ddd'}t|D]\}}|r%|}|d}|d||d<qWd|S1s@wY|S) a WikiHomograph and NeMo use slightly different phoneme sets, this function reads WikiHomograph word_ids to NeMo IPA heteronyms mapping. Args: wordid_to_phonemes_file: Path to a file with mapping from wordid predicted by the model to phonemes, e.g., NeMo/scripts/tts_dataset_files/wordid_to_ipa-0.7b_nv22.10.tsv to_lower: set to True to lower case wordid rrrrz rrN) rrrrrrr stripsplit)r,r-wordid_to_nemo_cmur#r%r&r*r*r+get_wordid_to_phonemesHs  r1Ftext remove_spacesdo_lowerexcludecCsxtj}|dur|D]}||d}q td|dd|}tdd|}|r2|dddd}|r8|}|S)a\ Remove punctuation marks form text Args: text: input text remove_spaces: set to True to remove spaces do_lower: set to True to lower case the text exclude: specify list of punctuation marks keep in the output, e.g., exclude=["'", "."] Return: processed text with punctuation marks removed N[] z + )string punctuationreplaceresubr.r )r2r3r4r5all_punct_markspr*r*r+remove_punctuation_s rB sentencessupported_heteronymsc Csg}g}|D]U}g}g}d}|D]<}|d} | D]2} t| ddd} | |vrG|| |}|t| } ||| f|| | }q|t| d7}qq||||q||fS)a Find heteronyms in sentences and returns span indices Args: sentences: sentences to find heteronyms in supported_heteronyms: heteronyms to look for Return: start_end: List[Tuple[int]] - start-end indices that indicate location of found heteronym in the sentence heteronyms: List[List[str]] - heteronyms found in sentences, each sentence can contain more than one heteronym r-TF)r4r3r)r r/rBindexr append) rCrD start_end heteronymssent cur_start_endcur_heteronyms start_idxwordword_by_hyphensub_word no_punct_wordend_idxr*r*r+r }s*     r casereturncCsH|dkr |}|S|dkr|}|S|dkr|}|Std|d)Nr r r zCase is not supported. Please specify either 'upper', 'lower', or 'mixed'.)r r r)r2rStext_newr*r*r+rsr)T)FFN)r )rrr>r;typingrrr__all__rrr strrboolr1rBr rr*r*r*r+s " #