o ei @svdZddlmZddlmZddlmZddlmZddl m Z e e Z d d iZed d Gd ddeZdgZdS)z Tokenization class for SpeechT5.)Any)SentencePieceBackend)logging)requires)EnglishNumberNormalizer vocab_filezspm_char.model) sentencepiece)backendsc seZdZdZeZddgZdZ      dd ee e fd Bd d ffd d Z dddZ e ddZejddZdd eefddZ ddeedeed Bded eeffdd Z ddeedeed Bd eefddZZS) SpeechT5Tokenizera Construct a SpeechT5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. bos_token (`str`, *optional*, defaults to `""`): The begin of sequence token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. normalize (`bool`, *optional*, defaults to `False`): Whether to convert numeric quantities in the text to their spelt-out english counterparts. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Attributes: sp_model (`SentencePieceProcessor`): The *SentencePiece* processor that is used for every conversion (string, tokens and IDs). input_idsattention_maskFNsp_model_kwargsreturnc s@||_d|_|dur||d<tjd||||||d|dS)Nr)r bos_token eos_token unk_token pad_token normalize)r _normalizersuper__init__) selfr rrrrrrkwargs __class__rp/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/speecht5/tokenization_speecht5.pyrMs  zSpeechT5Tokenizer.__init__cKs0|d|j}|r d|}|r||}||fS)Nr )popr normalizer)rtextis_split_into_wordsrrrrr"prepare_for_tokenizationjs  z*SpeechT5Tokenizer.prepare_for_tokenizationcCs|jdur t|_|jSN)rr)rrrr"r%rs zSpeechT5Tokenizer.normalizercCs ||_dSr))r)rvaluerrr"r%xs cCs$|dur ||jgS|||jgS)z=Build model inputs from a sequence by appending eos_token_id.N) eos_token_id)r token_ids_0 token_ids_1rrr" build_inputs_with_special_tokens|s z2SpeechT5Tokenizer.build_inputs_with_special_tokensr,r-already_has_special_tokenscsV|r tj||ddSdg}|durdgt||Sdgt|dgt||S)NT)r,r-r/rr)rget_special_tokens_masklen)rr,r-r/ suffix_onesr rr"r0s z)SpeechT5Tokenizer.get_special_tokens_maskcCs8|jg}|durt||dgSt|||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. SpeechT5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of zeros. Nr)r+r1)rr,r-eosrrr"$create_token_type_ids_from_sequencessz6SpeechT5Tokenizer.create_token_type_ids_from_sequences)rrrrFN)Fr))NF)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesis_fastdictstrrrr(propertyr%setterlistintr.boolr0r4 __classcell__rrr r"r sP*      r N)r8typingr tokenization_utils_sentencepiecerutilsrutils.import_utilsrnumber_normalizerr get_loggerr5loggerr9r __all__rrrr"s