o i1#@sdZddlZddlmZddlmZmZddlZddl m Z ddl m Z ddl mZd d lmZe eZd d iZed dGddde ZdgZdS)z Tokenization class for SpeechT5.N)copyfile)AnyOptional)PreTrainedTokenizer)logging)requires)EnglishNumberNormalizer vocab_filezspm_char.model) sentencepiece)backendsc s,eZdZdZeZddgZ      d/d eee e fd d ffd d Z d0ddZ e ddZe ddZejddZddZddZddZde d ee fddZddZd d!Zd"d#Zd1d eefd$d%Z d2d&eed'eeed(ed eeffd)d* Zd1d+e d,ee d ee fd-d.ZZS)3SpeechT5Tokenizera Construct a SpeechT5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. bos_token (`str`, *optional*, defaults to `""`): The begin of sequence token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. normalize (`bool`, *optional*, defaults to `False`): Whether to convert numeric quantities in the text to their spelt-out english counterparts. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Attributes: sp_model (`SentencePieceProcessor`): The *SentencePiece* processor that is used for every conversion (string, tokens and IDs). input_idsattention_maskFNsp_model_kwargsreturnc sj|durin||_||_||_d|_tjdi|j|_|j|tj d||||||jd|dS)N) bos_token eos_token unk_token pad_token normalizer) rr r _normalizerspmSentencePieceProcessorsp_modelLoadsuper__init__) selfr rrrrrrkwargs __class__ro/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/speecht5/tokenization_speecht5.pyr#Qs    zSpeechT5Tokenizer.__init__cKs0|d|j}|r d|}|r||}||fS)Nr )popr normalizer)r$textis_split_into_wordsr%rrrr(prepare_for_tokenizationns  z*SpeechT5Tokenizer.prepare_for_tokenizationcCs |jSN)r get_piece_sizer$rrr( vocab_sizev zSpeechT5Tokenizer.vocab_sizecCs|jdur t|_|jSr/)rr r1rrr(r+zs zSpeechT5Tokenizer.normalizercCs ||_dSr/)r)r$valuerrr(r+r3cs(fddtjD}|j|S)Ncsi|]}||qSr)convert_ids_to_tokens).0ir1rr( sz/SpeechT5Tokenizer.get_vocab..)ranger2updateadded_tokens_encoder)r$vocabrr1r( get_vocabs zSpeechT5Tokenizer.get_vocabcCs|j}d|d<|S)Nr )__dict__copy)r$staterrr( __getstate__s zSpeechT5Tokenizer.__getstate__cCs<||_t|ds i|_tjdi|j|_|j|jdS)Nrr)r>hasattrrrrr r!r )r$drrr( __setstate__s  zSpeechT5Tokenizer.__setstate__r,cCs|jj|tdS)zPTake as input a string and return a list of strings (tokens) for words/sub-words)out_type)r encodestr)r$r,rrr( _tokenizeszSpeechT5Tokenizer._tokenizecCs |j|S)z0Converts a token (str) in an id using the vocab.)r piece_to_id)r$tokenrrr(_convert_token_to_ids z&SpeechT5Tokenizer._convert_token_to_idcCs|j|}|S)z=Converts an index (integer) in a token (str) using the vocab.)r IdToPiece)r$indexrJrrr(_convert_id_to_tokens z&SpeechT5Tokenizer._convert_id_to_tokencCspg}d}d}|D]#}||jvr$|s|d7}||j||7}d}g}q||d}q||j|7}|S)z:Converts a sequence of tokens (string) in a single string.Fr)T)all_special_tokensr decodeappendstrip)r$tokenscurrent_sub_tokens out_stringprev_is_specialrJrrr(convert_tokens_to_strings  z*SpeechT5Tokenizer.convert_tokens_to_stringcCs$|dur ||jgS|||jgS)z=Build model inputs from a sequence by appending eos_token_id.N) eos_token_id)r$ token_ids_0 token_ids_1rrr( build_inputs_with_special_tokenss z2SpeechT5Tokenizer.build_inputs_with_special_tokensrZr[already_has_special_tokenscsV|r tj||ddSdg}|durdgt||Sdgt|dgt||S)NT)rZr[r]r r)r"get_special_tokens_masklen)r$rZr[r] suffix_onesr&rr(r^s z)SpeechT5Tokenizer.get_special_tokens_masksave_directoryfilename_prefixcCstj|std|ddStj||r|dndtd}tj|jtj|kr?tj |jr?t |j||fStj |jsgt |d}|j }||Wd|fS1sbwY|fS)NzVocabulary path (z) should be a directory-rOr wb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr isfileropenr serialized_model_protowrite)r$rarbout_vocab_fileficontent_spiece_modelrrr(save_vocabularys" (     z!SpeechT5Tokenizer.save_vocabulary)rrrrFN)Fr/)NF) __name__ __module__ __qualname____doc__rkvocab_files_namesmodel_input_namesrdictrGrr#r.propertyr2r+setterr=rArDlistrHrKrNrXintr\boolr^tuplert __classcell__rrr&r(r"sR*       ( r)rxreshutilrtypingrrr rtokenization_utilsrutilsrutils.import_utilsrnumber_normalizerr get_loggerrurhrkr__all__rrrr(s       =