o :iV @s~ddlZddlZddlmZmZddlmZddlmZm Z m Z ddl Z ddl m Z ddlmZmZddlmZmZddlmZmZzdd lmZWney]Zzeed dZ[wwe jejGd d d eZGd ddeZ GdddeZ!GdddeZ"Gddde"Z#GdddeZ$de de%de%fddZ&e'dkrdZ(e"Z)e)*e(gZ+e,dd-e+ddSdS) N)ABCabstractmethod)reduce)DictListOptional)CutSet)Style lazy_pinyin)to_finals_tone3 to_initials)ChineseTextNormalizerEnglishTextNormalizer)phonemize_espeakzl Please run pip install piper_phonemize -f https://k2-fsa.github.io/icefall/piper_phonemize.htmlc@szeZdZdZedeedeeefddZedeedeeefddZ edeeedeeefd d Z d S) Tokenizerz>Abstract base class for tokenizers, defining common interface.textsreturncCt)z4Convert list of texts to list of token id sequences.NotImplementedErrorselfrr3/home/ubuntu/LuxTTS/zipvoice/tokenizer/tokenizer.pytexts_to_token_ids/zTokenizer.texts_to_token_idscCr)z1Convert list of texts to list of token sequences.rrrrrtexts_to_tokens4rzTokenizer.texts_to_tokenstokenscCr)z>Convert list of token sequences to list of token id sequences.r)rrrrrtokens_to_token_ids9rzTokenizer.tokens_to_token_idsN) __name__ __module__ __qualname____doc__rrstrintrrrrrrrr,s  (rc@seZdZdZddeefddZdeedeeefdd Z deedeeefd d Z d eeedeeefd dZ dS)SimpleTokenizerz_The simplpest tokenizer, treat every character as a token, without text normalization. N token_filecCsd|_|durtddSi|_t|ddd/}|D]"}|d}|dt|d }}||jvs:J|||j|<qWdn1sJwY|jd |_ t |j|_ d |_dS) Args: tokens: the file that contains information that maps tokens to ids, which is a text file with '{token} {token_id}' per line. FNTInitialize Tokenizer without tokens file, will fail when map to ids.rutf-8encoding r_T) has_tokensloggingdebugtoken2idopen readlinesrstripsplitr$pad_idlen vocab_size)rr&flineinfotokenidrrr__init__Ds$     zSimpleTokenizer.__init__rrcC|||SNrrrrrrr]z"SimpleTokenizer.texts_to_token_idscsfddttD}|S)Ncg|]}t|qSrlist.0irrr gz3SimpleTokenizer.texts_to_tokens..ranger9rr tokens_listrrKrrcszSimpleTokenizer.texts_to_tokensrQcCd|jsJdg}|D]$}g}|D]}||jvr!td|q||j|q||q |SN/Please initialize Tokenizer with a tokens file. Skip OOV r0r3r1r2appendrrQtoken_ids_listr token_idstrrrrj  z#SimpleTokenizer.tokens_to_token_idsrB) rr r!r"rr#r@rr$rrrrrrrr%?s$      r%c@seZdZdZddeedefddZded eefd d Zd eed eee fd dZ d eed eeefddZ deeed eee fddZ dS)EspeakTokenizerz,A simple tokenizer with Espeak g2p function.Nen-usr&langcCsd|_||_|durtddSi|_t|ddd/}|D]"}|d}|dt |d }}||jvs=J|||j|<q Wdn1sMwY|jd |_ t |j|_ d |_dS) a& Args: tokens: the file that contains information that maps tokens to ids, which is a text file with '{token} {token_id}' per line. lang: the language identifier, see https://github.com/rhasspy/espeak-ng/blob/master/docs/languages.md FNr(r)r*r+r-rr.r/T) r0r_r1r2r3r4r5r6r7r$r8r9r:)rr&r_r;r<r=r>r?rrrr@s&      zEspeakTokenizer.__init__textrc Csbzt||j}tdd|}|WSty0}ztd|jd|gWYd}~Sd}~ww)NcS||SrBrxyrrrz%EspeakTokenizer.g2p..zTokenization of z texts failed: )rr_r Exceptionr1warningrr`rexrrrg2ps zEspeakTokenizer.g2prcCrArBrCrrrrrrDz"EspeakTokenizer.texts_to_token_idscs fddttD}|S)Ncsg|] }|qSr)rkrHrrrrLsz3EspeakTokenizer.texts_to_tokens..rNrPrrrrszEspeakTokenizer.texts_to_tokensrQcCrRrSrVrXrrrrr\z#EspeakTokenizer.tokens_to_token_ids)Nr^) rr r!r"rr#r@rrkr$rrrrrrrr]s&       r]c@sVeZdZd*deefddZdeedeeefdd Zd edefd d Z deedeeefd dZ deeedeeefddZ d edeefddZ d edeefddZ d edeefddZd edeefddZddZd edeefddZddZd edefd!d"Zd edefd#d$Zd%edefd&d'Zd%edefd(d)ZdS)+EmiliaTokenizerNphoner&cCs|dks Jd|dt|_t|_d|_|dur"tddSi|_t|ddd /}| D]"}| d }|d t |d }}||jvsNJ|||j|<q1Wdn1s^wY|jd |_ t|j|_d|_dS)r'rmz1Only support phone tokenizer for Emilia, but get .FNzXInitialize Tokenizer without tokens file, will fail when map to ids.r)r*r+r-rr.r/T)renglish_normalizerr chinese_normalizerr0r1r2r3r4r5r6r7r$r8r9r:)rr& token_typer;r<r=r>r?rrrr@s.       zEmiliaTokenizer.__init__rrcCrArBrCrrrrrrDz"EmiliaTokenizer.texts_to_token_idsr`cCs ||SrB)map_punctuationsrr`rrrpreprocess_texts zEmiliaTokenizer.preprocess_textc Cstt|D] }|||||<qg}|D]^}||}g}tt|D]I}||}|ddkr9||d} n1|ddkrG||d} n#|ddkrU||d} n|ddkra|dg} n td|q%|| 7}q%| |q|S)Nr.zhrenpinyintagzjNo English or Chinese characters found, skipping segment of unknown language: ) rOr9rt get_segment tokenize_ZH tokenize_ENtokenize_pinyinr1rhrW) rrrJ phoneme_listr`segments all_phonemeindexsegphonemerrrrs2        zEmiliaTokenizer.texts_to_tokensrQcCrRrSrVrXrrrrs  z#EmiliaTokenizer.tokens_to_token_idsc Csz<|j|}tt|}t|tjddd}g}|D]}|ddr+|ddvs1| |q| | |q|WSt yX}zt d|gWYd}~Sd}~ww)NT)style tone_sandhineutral_tone_with_fiver12345z&Tokenization of Chinese texts failed: )rp normalizerGjiebacutr r TONE3isalpharWextendseperate_pinyinrgr1rh)rr`segsfullphonesrcrjrrrrz*s*  zEmiliaTokenizer.tokenize_ZHc Csdz|j|}t|d}tdd|}|WSty1}ztd|gWYd}~Sd}~ww)Nr^cSrarBrrbrrrreErfz-EmiliaTokenizer.tokenize_EN..z&Tokenization of English texts failed: )rorrrrgr1rhrirrrr{As  zEmiliaTokenizer.tokenize_ENc Csz4|dr |ds J|dd}|ddr#|ddvs/td|dgWS||WStyP}ztd|gWYd}~Sd}~ww) N<>rrrzHStrings enclosed with <> should be pinyin, but got: z. Skipped it. zTokenize pinyin failed: ) startswithendswithlstripr6rr1rhrrg)rr`rjrrrr|Ks  zEmiliaTokenizer.tokenize_pinyincCsJg}t|dd}t|ddd}|dkr||d|dkr#|||S)z8 Separate pinyin into initial and final F)strictT)rr0)r r rW)rr`pinyinsinitialfinalrrrr\s  zEmiliaTokenizer.seperate_pinyincCs|dd}|dd}|dd}|dd}|d d }|d d }|d d}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|S)Nu，,u。rnu！!u？?u；;u：:u、u‘'u“"u”u’u⋯u…u···u ・・・z...)replacersrrrrrqs               z EmiliaTokenizer.map_punctuationsc Cs0g}g}d}d}td}||}t|D]$\}}||s$||r*|dq||r5|dq|dqt|t|ksEJt t|D]>}|dkr\|||7}||}qK|dkrk|||7}||}qK|||dfvrz|||7}qK|||f||}||}qK|||f| |}|S)u Split a text into segments based on language types (Chinese, English, Pinyin, tags, etc.) Args: text (str): Input text to be segmented Returns: List[str]: Segmented text parts with their language types Example: Input: 我们是小米人,是吗? Yes I think so!霍...啦啦啦 Output: [('我们是小米人,是吗? ', 'zh'), ('Yes I think so!', 'en'), ('霍...啦啦啦', 'zh')] rz[<[].*?[>\]]|.rurvotherr) recompilefindall enumerate is_chinese is_pinyinrW is_alphabetr9rOsplit_segments) rr`r~typestemp_seg temp_lang _part_patternrJpartrrrrys8            zEmiliaTokenizer.get_segmentcCstg}|D]3\}}td|}|D]&}|sq||r"||dfq||r/||dfq|||fqq|S)a split segments into smaller parts if special strings enclosed by [] or <> are found, where <> denotes pinyin strings, [] denotes other special strings. Args: segments (list): A list of tuples where each tuple contains: - temp_seg (str): The text segment to be split. - temp_lang (str): The language code associated with the segment. Returns: list: A list of smaller segments. z([<[].*?[>\]])rwrx)rr7rrWis_tag)rr~resultrrpartsrrrrrs     zEmiliaTokenizer.split_segmentscharcCs|dkr |dkr dSdS)Nu一u龥TFrrrrrrrszEmiliaTokenizer.is_chinesecCs(|dkr|dks|dkr|dkrdSdS)NAZazTFrrrrrrszEmiliaTokenizer.is_alphabetrcC|dr |dr dSdS)NrrTFrrrrrrrrzEmiliaTokenizer.is_pinyincCr)N[]TFrrrrrrrzEmiliaTokenizer.is_tagNrm)rr r!rr#r@rr$rrtrrrzr{r|rrrryrboolrrrrrrrrrlsB      !    =rlcs:eZdZd deeffdd Zdedefdd ZZS) DialogTokenizerNrmr&cs4tj||d|r|jd|_|jd|_dSdS)N)r&rqz[S1]z[S2])superr@r3spk_a_idspk_b_id)rr&rq __class__rrr@s  zDialogTokenizer.__init__r`rcCstdd|}||}|S)Nz\s*(\[S[12]\])\s*z\1)rsubrrrsrrrrts zDialogTokenizer.preprocess_textr)rr r!rr#r@rt __classcell__rrrrrsrc@s~eZdZddeefddZdeedeeefdd Zdeedeeefd d Z d eeedeeefd dZ dS)LibriTTSTokenizerNrr&c CsR||_|dvs Jzddl}Wnty"}zt|dd}~ww|jj|_d|_|dur6t ddS|dkrWddl }| |_ |j ||j d|_|j |_nMi|_t|d d d /}|D]"}|d }|dt|d } } | |jvsJ| | |j| <qfWdn1swY|jd|_t|j|_d|_dS)a0 Args: type: the type of tokenizer, e.g., bpe, char, phone. tokens: the file that contains information that maps tokens to ids, which is a text file with '{token} {token_id}' per line if type is char or phone, otherwise it is a bpe_model file. )bperrmrNz+ Please run pip install espnet_tts_frontendFr(rzr)r*r+r-r.r/T)typetacotron_cleaner.cleanersrg RuntimeErrorcleanerscustom_english_cleanersrr0r1r2 sentencepieceSentencePieceProcessorspload piece_to_idr8get_piece_sizer:r3r4r5r6r7r$r9) rr&rqtacotron_cleanerrjspmr;r<r=r>r?rrrr@sB          zLibriTTSTokenizer.__init__rrcCsJ|jdkrtt|D] }|||||<q |j|S|||S)Nr)rrOr9rrencoderr)rrrJrrrr2s  z$LibriTTSTokenizer.texts_to_token_idscsttD] }|||<q|jdkr&fddttD}|S|jdkr:fddttD}|S|jdkrG|jjtd}|S)NrcrErrFrHrKrrrLErMz5LibriTTSTokenizer.texts_to_tokens..rmcsg|] }t|dqS)r^)rlowerrHrKrrrLGsr)out_type)rOr9rrrrr#)rrrJrQrrKrr=s    z!LibriTTSTokenizer.texts_to_tokensrQcCsv|jsJd|jdksJdg}|D]$}g}|D]}||jvr*td|q||j|q||q|S)NrTrz-BPE tokenizer does not support this function.rU)r0rr3r1r2rWrXrrrrOs  z%LibriTTSTokenizer.tokens_to_token_ids)Nr) rr r!rr#r@rr$rrrrrrrrs"+       rcut_set tokenizerr_cs~dkrtn*dkrt|dn dkrtndkr"tndkr*tntddfd d }||}|S) Nemiliaespeak)r_dialoglibrittssimplezUnsupported tokenizer: rncsLt|jdksJt|j|f|jdj}|gd}||jd_|S)Nr.r)r9 supervisionsr`rr)rr`rrrr _prepare_cutts   z add_tokens.._prepare_cut)rlr]rrr% ValueErrormap)rrr_rrrr add_tokensfs   r__main__u我们是5年小米人,是吗? Yes I think so! mr king, 5 years, from 2019 to 2024.霍...啦啦啦超过90%的人...?!9204ztokens: |).r1rabcrr functoolsrtypingrrrrlhotserpypinyinr r pypinyin.contrib.tone_convertr r zipvoice.tokenizer.normalizerr rpiper_phonemizerrgrjrdefault_loggersetLevelINFOrr%r]rlrrr#rrr`rrrprintjoinrrrrsF  @J/`