o €wÖiyã@sbdZddlmZmZddlmZddlmZmZm Z m Z m Z ddl m Z Gdd„deƒZdgZd S) z Processor class for Bros. é)ÚOptionalÚUnioné)ÚProcessorMixin)Ú BatchEncodingÚPaddingStrategyÚPreTokenizedInputÚ TextInputÚTruncationStrategy)Ú TensorTypec"söeZdZdZdgZdZd"‡fdd„ Z              d#d eee e ee e fd e d ee e e fd ee e efdeededeedee dee de de de de de deee efdef dd„Zdd„Zdd„Zed d!„ƒZ‡ZS)$Ú BrosProcessora‘ Constructs a Bros processor which wraps a BERT tokenizer. [`BrosProcessor`] offers all the functionalities of [`BertTokenizerFast`]. See the docstring of [`~BrosProcessor.__call__`] and [`~BrosProcessor.decode`] for more information. Args: tokenizer (`BertTokenizerFast`, *optional*): An instance of ['BertTokenizerFast`]. The tokenizer is a required input. Ú tokenizer)Ú BertTokenizerÚBertTokenizerFastNc s |durtdƒ‚tƒ |¡dS)Nz"You need to specify a `tokenizer`.)Ú ValueErrorÚsuperÚ__init__)Úselfr Úkwargs©Ú __class__©úe/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/bros/processing_bros.pyr)szBrosProcessor.__init__TFrÚtextÚadd_special_tokensÚpaddingÚ truncationÚ max_lengthÚstrideÚpad_to_multiple_ofÚreturn_token_type_idsÚreturn_attention_maskÚreturn_overflowing_tokensÚreturn_special_tokens_maskÚreturn_offsets_mappingÚ return_lengthÚverboseÚreturn_tensorsÚreturncKs4|jd||||||||| | | | | ||dœ|¤Ž}|S)z¶ This method uses [`BertTokenizerFast.__call__`] to prepare text for the model. Please refer to the docstring of the above two methods for more information. )rrrrrrrr r!r"r#r$r%r&r'Nr)r )rrrrrrrrr r!r"r#r$r%r&r'rÚencodingrrrÚ__call__/s(ñðzBrosProcessor.__call__cOó|jj|i|¤ŽS)zÁ This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r Ú batch_decode©rÚargsrrrrr,\ózBrosProcessor.batch_decodecOr+)z» This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )r Údecoder-rrrr0cr/zBrosProcessor.decodecCs|jj}tt |¡ƒS©N)r Úmodel_input_namesÚlistÚdictÚfromkeys)rÚtokenizer_input_namesrrrr2jszBrosProcessor.model_input_namesr1)NTFNNrNNNFFFFTN)Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú attributesÚtokenizer_classrrr rr3ÚboolÚstrrr rÚintr rr*r,r0Úpropertyr2Ú __classcell__rrrrr sr ðþý ü ûúùø ÷ ö õ ô óòñð î-r N)r:ÚtypingrrÚprocessing_utilsrÚtokenization_utils_baserrrr r Úutilsr r Ú__all__rrrrÚs   V