o i$@sdZddlZddlZddlZddlmZmZmZddlm Z ddl m Z m Z m Z e r/ddlZe r6ddlZe eZddiZd d ZGd d d e Zd gZdS) zTokenization class for VITS.N)AnyOptionalUnion)PreTrainedTokenizer)is_phonemizer_availableis_uroman_availablelogging vocab_filez vocab.jsoncCs td}||}|du}|S)Nz [^\x00-\x7F])recompilesearch) input_stringnon_roman_patternmatch has_non_romanrg/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/vits/tokenization_vits.pyhas_non_roman_characters%s  rc seZdZdZeZddgZ       d& d'fd d Zed d Z ddZ ddZ ddZ d(de dedeed ee ee efffddZde d ee fddZdee d e fddZddZd d!Zd)d"e d#ee d eee dffd$d%ZZS)* VitsTokenizera Construct a VITS tokenizer. Also supports MMS-TTS. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. language (`str`, *optional*): Language identifier. add_blank (`bool`, *optional*, defaults to `True`): Whether to insert token id 0 in between the other tokens. normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the input text by removing all casing and punctuation. phonemize (`bool`, *optional*, defaults to `True`): Whether to convert the input text into phonemes. is_uroman (`bool`, *optional*, defaults to `False`): Whether the `uroman` Romanizer needs to be applied to the input text prior to tokenizing. input_idsattention_maskNTFreturnc st|dd} t| |_Wdn1swYdd|jD|_||_||_||_||_ ||_ t j d|||||||d| dS)Nutf-8encodingcSsi|]\}}||qSrr).0kvrrr Wsz*VitsTokenizer.__init__..) pad_token unk_tokenlanguage add_blank normalize phonemize is_uromanr) openjsonloadencoderitemsdecoderr$r%r&r'r(super__init__) selfr r"r#r$r%r&r'r(kwargs vocab_handle __class__rrr0Hs(  zVitsTokenizer.__init__cCs t|jSN)lenr,r1rrr vocab_sizejs zVitsTokenizer.vocab_sizecs(fddtjD}|j|S)Ncsi|]}||qSr)convert_ids_to_tokens)rir8rrr!osz+VitsTokenizer.get_vocab..)ranger9updateadded_tokens_encoder)r1vocabrr8r get_vocabns zVitsTokenizer.get_vocabcCst|jt|j}d}d}|t|krMd}|D]}|||t||kr8||7}|t|7}d}nq|sG|||7}|d7}|t|ks|S)zfLowercase the input string, respecting any special token ids that may be part or entirely upper-cased.rFT)listr,keysr>r7lower)r1rall_vocabulary filtered_textr; found_matchwordrrrnormalize_textss"    zVitsTokenizer.normalize_textcCs|jdkr |dd}|S)z4Special treatment of characters in certain languagesronuțuţ)r$replace)r1textrrr_preprocess_chars  zVitsTokenizer._preprocess_charrMis_split_into_wordsr&c s|dur|nj}|r|}|}t|r.jr.ts%tdn t }| |}j rNt s8t dtj |dddddd}tdd |}||fS|r`d ttfd d |}||fS) a Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. normalize (`bool`, *optional*, defaults to `None`): Whether or not to apply punctuation and casing normalization to the text inputs. Typically, VITS is trained on lower-cased and un-punctuated text. Hence, normalization is used to ensure that the input text consists only of lower-case characters. kwargs (`dict[str, Any]`, *optional*): Keyword arguments to use for the tokenization. Returns: `tuple[str, dict[str, Any]]`: The prepared text and the unused kwargs. NaCText to the tokenizer contains non-Roman characters. To apply the `uroman` pre-processing step automatically, ensure the `uroman` Romanizer is installed with: `pip install uroman` Note `uroman` requires python version >= 3.10Otherwise, apply the Romanizer manually as per the instructions: https://github.com/isi-nlp/uromanzEPlease install the `phonemizer` Python package to use this tokenizer.zen-usespeakT)r$backendstrippreserve_punctuation with_stressz\s+ rAcs |jvSr6)r,)charr8rrs z8VitsTokenizer.prepare_for_tokenization..)r&rJrNrr(rloggerwarningurUromanromanize_stringr'r ImportError phonemizerr subjoinrCfilterrR)r1rMrOr&r2rGuromanrr8rprepare_for_tokenizations6    z&VitsTokenizer.prepare_for_tokenizationcCs@t|}|jr|dgt|dd}||ddd<|}|S)z]Tokenize a string by inserting the `` token at the boundary between adjacent characters.rrBN)rCr%_convert_id_to_tokenr7)r1rMtokens interspersedrrr _tokenizes zVitsTokenizer._tokenizerfcCs*|jrt|dkr|ddd}d|S)NrBrdrA)r%r7r`)r1rfrrrconvert_tokens_to_strings z&VitsTokenizer.convert_tokens_to_stringcCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)r,getr#)r1tokenrrr_convert_token_to_idsz"VitsTokenizer._convert_token_to_idcCs |j|S)z=Converts an index (integer) in a token (str) using the vocab.)r.rj)r1indexrrrres z"VitsTokenizer._convert_id_to_tokensave_directoryfilename_prefixc Cstj|std|ddStj||r|dndtd}t|ddd}|t j |j d d d d d Wd|fS1sEwY|fS)NzVocabulary path (z) should be a directory-rAr wrrrdTF)indent sort_keys ensure_ascii ) ospathisdirrXerrorr`VOCAB_FILES_NAMESr)writer*dumpsr,)r1rnror frrrsave_vocabularys   zVitsTokenizer.save_vocabulary)rrNTTTF)rN)FNr6)__name__ __module__ __qualname____doc__rzvocab_files_namesmodel_input_namesr0propertyr9r@rJrNstrboolrtupledictrrcrCrhrirlrerr~ __classcell__rrr4rr/sD "  A 0r)rr*rvr typingrrrtokenization_utilsrutilsrrr r^rbrZ get_loggerrrXrzrr__all__rrrrs"    H