o
i! @ s d Z ddlZddlZddlmZ ddlmZmZ ddlm Z ddl
m Z m Z m
Z
ddlmZmZ dd lmZ G d
d d e d d
ZeeZG dd de ZdgZdS )z
Processor class for Donut.
N)contextmanager)OptionalUnion )
ImageInput)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)loggingc @ s e Zd Zi ZdS )DonutProcessorKwargsN)__name__
__module__ __qualname__ _defaults r r ^/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/donut/processing_donut.pyr
s r
F)totalc
s e Zd ZdZddgZdZdZd fdd Z dd ee d
ee
e e e e
ef d ee fd d
Zedd Zedd ZdddZedd Zedd Z ZS ) DonutProcessora
Constructs a Donut processor which wraps a Donut image processor and an XLMRoBERTa tokenizer into a single
processor.
[`DonutProcessor`] offers all the functionalities of [`DonutImageProcessor`] and
[`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. See the [`~DonutProcessor.__call__`] and
[`~DonutProcessor.decode`] for more information.
Args:
image_processor ([`DonutImageProcessor`], *optional*):
An instance of [`DonutImageProcessor`]. The image processor is a required input.
tokenizer ([`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`], *optional*):
An instance of [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. The tokenizer is a required input.
image_processor tokenizerAutoImageProcessor
AutoTokenizerNc sR d }d|v rt dt |d}|d ur|n|}t || | j| _d| _d S )Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.F) warningswarn
FutureWarningpopsuper__init__r current_processor_in_target_context_manager)selfr r kwargsr __class__r r r 9 s
zDonutProcessor.__init__imagestextr$ c K s | j r
| j||fi |S |du r|du rtd| jtfd| jji|}|dur5| j|fi |d }|durP|durE|d dd | j|fi |d }|du rV|S |du r\|S |d |d < |d |d< |S )
a
When used in normal mode, this method forwards all its arguments to AutoImageProcessor's
[`~AutoImageProcessor.__call__`] and returns its output. If used in the context
[`~DonutProcessor.as_target_processor`] this method forwards all its arguments to DonutTokenizer's
[`~DonutTokenizer.__call__`]. Please refer to the docstring of the above two methods for more information.
NzBYou need to specify either an `images` or `text` input to process.tokenizer_init_kwargs
images_kwargs text_kwargsadd_special_tokensF input_idslabels) r" r!
ValueError
_merge_kwargsr
r init_kwargsr
setdefault) r# r' r( audiovideosr$
output_kwargsinputs encodingsr r r __call__I s0 zDonutProcessor.__call__c C s | j j}t|ddg S )Nr- r. )r model_input_nameslist)r# image_processor_input_namesr r r r9 s s z DonutProcessor.model_input_namesc c s2 t d d| _| j| _dV | j| _d| _dS )z
Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning TrOCR.
z`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your images inputs, or in a separate call.TNF)r r r" r r! r r# r r r as_target_processory s
z"DonutProcessor.as_target_processorFc C s@ |du r | j }i }|r td|tj}|du rn|| d }d|vr(n|d|dd }|tdtd }t|}td| d|tj} | du rZ| |d}n|
} t|}
t| } t|
d| |tjtj B } | dur|
d } d| v rd| v r| j
| d|d }
|
rt|
dkr|
d
}
|
||< n?g ||< | d D ]%}| }||v r|d
d kr|d
d dkr|dd
}|| | qt|| dkr|| d
||< ||| t| d }|dd d kr
|g| j
|dd d|d S |s|r|r|gS |S |r g S d|iS )zS
Convert a (generated) token sequence into an ordered JSON format.
Nz z
text_sequence)r get_added_vocabresearch
IGNORECASEstartindexlenescapereplacegroupDOTALLstrip
token2jsonsplitappendfind)r# tokensrA rB outputpotential_start start_tokenkey key_escaped end_tokenstart_token_escapedend_token_escapedcontentvalueleafr r r rS s\
$ ,zDonutProcessor.token2jsonc C t dt | jS )Nzg`feature_extractor_class` is deprecated and will be removed in v5. Use `image_processor_class` instead.)r r r image_processor_classr< r r r feature_extractor_class
z&DonutProcessor.feature_extractor_classc C rc )Nz[`feature_extractor` is deprecated and will be removed in v5. Use `image_processor` instead.)r r r r r< r r r r rf z DonutProcessor.feature_extractor)NN)NNNN)FN)r r r __doc__
attributesrd tokenizer_classr r r r strr: r r
r r
r8 propertyr9 r r= rS re r
__classcell__r r r% r r % s4
*
:
r )rg rH r
contextlibr typingr r image_utilsr processing_utilsr r r tokenization_utils_baser
r utilsr r
get_loggerr loggerr __all__r r r r s
0