o
ei @ sd d dl Z d dlZd dlmZmZmZmZ d dlmZ ddl m
Z
dddZ G d d
d
e
Z d
gZ
dS ) N) Tokenizerdecoderspre_tokenizers
processors)Unigram )TokenizersBackendz spiece.modelztokenizer.json)
vocab_filetokenizer_filec s e Zd ZdZeZddgZeZ d fd d
Z d d Z
d
d Z dde e
e B dededB dedef
fdd
Z ZS )
LasrTokenizera
Construct a LASR tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models).
This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`, *optional*):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as
"" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by
calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method
additional_special_tokens (`list[str]`, *optional*):
Additional special tokens used by the tokenizer.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dict. If not provided, a minimal vocabulary is created using the special tokens.
input_idsattention_maskd Nc s || _ |d ur6dd |D } t| dk r |dd t|D 7 }n!|dkr5|t| kr5td| d| dn d d t|D } | }|d urI|| _n*t|d
ft|d
ft|d
fd g| _t|d d d D ]}
| jd
|
dd
f qdtt| jddd| _ d | j _
t t
t jddddg| j _tjdddd| j _t jd|||||d| tjddgg dd| jfgd| j _d S ) Nc S s g | ]
}d t |v r|qS )
Y s z*LasrTokenizer.__init__.. c S g | ]}d | dqS r >r r ir r r r [ r zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to LasrTokenizer. In this case the additional_special_tokens must include the extra_ids tokensc S r r r r r r r r c r g ) ▁g r r r F)unk_id
byte_fallbackr alwaysT) replacementprepend_schemesplit) eos_token unk_token pad_token extra_idsadditional_special_tokens$Ar )r- r z$Br )singlepairspecial_tokensr )
_extra_idslenrange
ValueError
_vocab_scoresr appendr r
_tokenizer
normalizerr SequenceWhitespaceSplit Metaspace
pre_tokenizerr decodersuper__init__r TemplateProcessing eos_token_idpost_processor) selfr( r) r* r+ r, vocabr kwargs extra_tokensr __class__r r r? J sd
zLasrTokenizer.__init__c C s t ttdd | jS )zQGet the list of sentinel tokens (extra_id tokens) from additional_special_tokens.c S s t td| d uS )Nz)boolresearch)r r r r z3LasrTokenizer.get_sentinel_tokens..)listsetfilterr, rC r r r get_sentinel_tokens s z!LasrTokenizer.get_sentinel_tokensc s fdd D S )z&Get the token IDs for sentinel tokens.c s g | ]} |qS r )convert_tokens_to_idsr tokenrQ r r r s z8LasrTokenizer.get_sentinel_token_ids..)rR rQ r rQ r get_sentinel_token_ids s z$LasrTokenizer.get_sentinel_token_idsFT token_idsskip_special_tokens clean_up_tokenization_spaces group_tokensreturnc sT t |tr|g}|rdd t|D } fdd|D }t jd|||d|S )Nc S s g | ]}|d qS )r r )r token_groupr r r r rM z)LasrTokenizer._decode..c s g | ] }| j kr|qS r ) pad_token_idrT rQ r r r s )rW rX rY r )
isinstanceint itertoolsgroupbyr> _decode)rC rW rX rY rZ rE rG rQ r rb s
zLasrTokenizer._decode)r r r r NNN)FNT)__name__
__module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr modelr? rR rV r_ rN rI r rb
__classcell__r r rG r r ! s: $L
r )r` rJ
tokenizersr r r r tokenizers.modelsr tokenization_utils_tokenizersr rg r __all__r r r r s