o
ei @ s d Z ddlZddlmZ ddlmZmZmZ ddlm Z m
Z
ddl m Z m
Z
G dd d ed
d Ze
eZe G d d
d
eZd
gZdS )z
Processor class for Donut.
N )
ImageInput)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)auto_docstringloggingc @ s e Zd Zi ZdS )DonutProcessorKwargsN)__name__
__module__ __qualname__ _defaults r r h/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/donut/processing_donut.pyr s r F)totalc sp e Zd Zd fdd Ze ddedB deee B eB e B dB de
e fddZ e
d d
Zdd d
Z ZS )DonutProcessorNc s t || d S )N)super__init__)selfimage_processor tokenizerkwargs __class__r r r # s zDonutProcessor.__init__imagestextr c K s |d u r |d u r t d| jtfd| jji|}|d ur(| j|fi |d }|d urC|d ur8|d dd | j|fi |d }|d u rI|S |d u rO|S |d |d< |d |d< |S ) NzBYou need to specify either an `images` or `text` input to process.tokenizer_init_kwargs
images_kwargs text_kwargsadd_special_tokensF input_idslabels)
ValueError
_merge_kwargsr r init_kwargsr
setdefault)r r r r
output_kwargsinputs encodingsr r r __call__& s, zDonutProcessor.__call__c C s | j j}t|ddg S )Nr" r# )r model_input_nameslist)r image_processor_input_namesr r r r, F s z DonutProcessor.model_input_namesFc C s@ |du r | j }i }|r td|tj}|du rn|| d }d|vr(n|d|dd }|tdtd }t|}td| d|tj} | du rZ| |d}n|
} t|}
t| } t|
d| |tjtj B } | dur|
d } d| v rd| v r| j
| d|d }
|
rt|
dkr|
d
}
|
||< n?g ||< | d D ]%}| }||v r|d
d kr|d
d dkr|dd
}|| | qt|| dkr|| d
||< ||| t| d }|dd d kr
|g| j
|dd d|d S |s|r|r|gS |S |r g S d|iS )zS
Convert a (generated) token sequence into an ordered JSON format.
Nz z
text_sequence)r get_added_vocabresearch
IGNORECASEstartindexlenescapereplacegroupDOTALLstrip
token2jsonsplitappendfind)r tokensr2 r3 outputpotential_start start_tokenkey key_escaped end_tokenstart_token_escapedend_token_escapedcontentvalueleafr r r rD L s\
$ ,zDonutProcessor.token2json)NN)FN)r r
r r r r strr- r r r r r+ propertyr, rD
__classcell__r r r r r ! s
r )__doc__r9 image_utilsr processing_utilsr r r tokenization_utils_baser r utilsr r
r
get_loggerr loggerr __all__r r r r s
e