o
i@' @ s d Z ddlZddlZddlZddlmZ ddlmZ ddlm Z ddl
m Z m Z e r2dd l
mZ ndZe eZd
d d ZG d
d de ZdgZdS )z Tokenization class for model T5. N)copyfile)Optional )PreTrainedTokenizerFast)is_sentencepiece_availablelogging ) T5Tokenizerz spiece.modelztokenizer.json)
vocab_filetokenizer_filec s e Zd ZU dZeZddgZeZg Z e
e e d< d fd
d Z
ed d
Zd dedee dee fddZ d de
e dee
e de
e fddZ d de
e dee
e de
e fddZdd Zdd Z ZS ) T5TokenizerFasta`
Construct a "fast" T5 tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models).
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as
"" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by
calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method
additional_special_tokens (`list[str]`, *optional*):
Additional special tokens used by the tokenizer.
add_prefix_space (`bool`, *optional*):
Whether or not the tokenizer should automatically add a prefix space
from_slow (`book`, *optional*, defaults to `False`):
Whether or not the tokenizer should be converted from a slow one. If `add_prefix_space` is set, this will be set to `True`.
input_idsattention_mask
prefix_tokensNd c s |d ur3dd |D }
t |
dk r |dd t|D 7 }n!|dkr2|t |
kr2td| d| dn d d t|D }
|
}|d urKtd
d | d < t jd||||||||d
| || _|| _d S )Nc S s g | ]
}d t |v r|qS )
d s z,T5TokenizerFast.__init__..r c S g | ]}d | dqS r >r r ir r r r f r zBoth extra_ids (z!) and additional_special_tokens (zk) are provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensc S r r r r r r r r n r zXYou set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizersT from_slow)r
r eos_token unk_token pad_token extra_idsadditional_special_tokensadd_prefix_spacer ) lenrange
ValueErrorlogger warning_oncesuper__init__r
_extra_ids) selfr
r r" r# r$ r% r& r' kwargs extra_tokens __class__r r r. V s<
zT5TokenizerFast.__init__c C sZ | t jv r+t j| }|d ur||kr|S |d u r+td| d| d| d| d t |S )NzGThis tokenizer was incorrectly instantiated with a model max length of z which will be corrected in Transformers v5.
For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on z( automatically truncating your input to zM when padding/encoding.
- If you want to encode/pad to sequences longer than z you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.)r max_model_input_sizeswarningswarn
FutureWarning) pretrained_model_name_or_pathmax_model_lengthinit_max_model_lengthdeprecated_max_model_lengthr r r !_eventually_correct_t5_max_length s$
z1T5TokenizerFast._eventually_correct_t5_max_lengthsave_directoryfilename_prefixreturnc C s | j stdtj|std| d d S tj||r"|d ndtd }tj | j
tj |krDt | j
| t d| |fS )NzhYour fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.zVocabulary path (z) should be a directory- r
zCopy vocab file to )
can_save_slow_tokenizerr* ospathisdirr+ errorjoinVOCAB_FILES_NAMESabspathr
r info)r0 r> r? out_vocab_filer r r save_vocabulary s z T5TokenizerFast.save_vocabulary token_ids_0 token_ids_1c C s8 || j g }|du r| j| S || j g }| j| | S )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:
- single sequence: `X `
- pair of sequences: `A B `
Args:
token_ids_0 (`list[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
N) eos_token_idr )r0 rN rO r r r build_inputs_with_special_tokens s
z0T5TokenizerFast.build_inputs_with_special_tokensc C s< | j g}|du rt|| dg S t|| | | dg S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of zeros.
Nr )rP r( )r0 rN rO eosr r r $create_token_type_ids_from_sequences s z4T5TokenizerFast.create_token_type_ids_from_sequencesc C s t ttdd | jS )Nc S s t td| d uS )Nz)boolresearch)r r r r s z5T5TokenizerFast.get_sentinel_tokens..)listsetfilterr& r0 r r r get_sentinel_tokens s z#T5TokenizerFast.get_sentinel_tokensc s fdd D S )Nc s g | ]} |qS r )convert_tokens_to_ids)r tokenr[ r r r s z:T5TokenizerFast.get_sentinel_token_ids..)r\ r[ r r[ r get_sentinel_token_ids s z&T5TokenizerFast.get_sentinel_token_ids)NNr r r r NN)N)__name__
__module__ __qualname____doc__rI vocab_files_namesmodel_input_namesr slow_tokenizer_classr rX int__annotations__r. staticmethodr= r r tuplerM rQ rS r\ r_
__classcell__r r r3 r r ) sH
&0
r )rc rD rU r6 shutilr typingr tokenization_utils_fastr utilsr r tokenization_t5r
get_loggerr` r+ rI r __all__r r r r s
A