o
eis @ s d Z ddlZddlZddlZddlmZ ddlmZ ddlm Z m
Z
m Z m Z m
Z
ddlmZ ddlmZmZ dd lmZ eeeef eeeeeef B eeeeef B eeeeeef B ZeedB eedB eedB eedB B B B ZG d
d d e d d
ZG dd de d d
Z G dd de
d d
Z eG dd de Z deeeeef dedeeef fddZ dededefdd Z d d Z!d d Z"d!d" Z#d&d$d%Z$dgZ%dS )'z Processor class for KOSMOS-2. N ) BatchFeature)
ImageInput) ImagesKwargsProcessingKwargsProcessorMixin
TextKwargsUnpack)
AddedToken)
BatchEncoding TextInput)auto_docstringc @ s2 e Zd ZU dZedB ed< eed< edB ed< dS )Kosmos2ImagesKwargsa
bboxes (`Union[list[tuple[int]], list[tuple[float]], list[list[tuple[int]]], list[list[tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional* defaults to 64):
The number of (consecutive) places that are used to mark the placeholders to store image information.
This should be the same as `latent_query_num` in the instance of `Kosmos2Config` you are using.
first_image_token_id (`int`, *optional*):
The token id that will be used for the first place of the subsequence that is reserved to store image
information. If unset, will default to `self.tokenizer.unk_token_id + 1`.
Nbboxesnum_image_tokensfirst_image_token_id)__name__
__module__ __qualname____doc__
NestedList__annotations__int r r l/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/kosmos2/processing_kosmos2.pyr ' s
r F)totalc @ s e Zd ZU dZeed< dS )Kosmos2TextKwargsz
add_eos_token (`bool`, defaults to `False`):
Whether or not to include `EOS` token id in the encoding when `add_special_tokens=True`.
add_eos_tokenN)r r r r boolr r r r r r 8 s
r c
@ s@ e Zd ZU eed< eed< dddddddddd ddid Zd
S ) Kosmos2ProcessorKwargs text_kwargs
images_kwargsTFr ) add_special_tokenspaddingstridereturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mappingreturn_token_type_idsverboser r @ )r r! N)r r r r r r _defaultsr r r r r A s
r c
s e Zd Zd! fdd Ze d"dedB deee B dee de
fd d
Z d d Z d
d Z
d#deee B dedB dededB deee B f
ddZd$ddZd$ddZedd Zdedeee eee B defd d Zd eeef eeeeef B deeef fd d Z ZS )%Kosmos2Processor c s d|_ d| _d| _d| _d| _d| _d| _d| _d | _d
| _ d | _
d | _ | j| j| j| j| j| j| j| j| j | j
| j g | _ || _
d
d t| j
D }g }| j | D ]
}|t|dddd qQ|| t || dS )z
num_patch_index_tokens (`int`, *optional*, defaults to 1024):
The number of tokens that represent patch indices.
Fzzzzzzz z z c S s" g | ]
}d t |d dqS )
)strzfill.0xr r r
s " z-Kosmos2Processor.__init__..T)lstriprstrip
normalizedN)r( eod_token boi_token eoi_token eoc_token eol_token bop_token eop_token boo_token eoo_token dom_token grd_token
tag_tokensnum_patch_index_tokensrangeappendr
add_tokenssuper__init__)selfimage_processor tokenizerrG kwargspatch_index_tokens
tokens_to_addtoken __class__r r rL X s>
zKosmos2Processor.__init__NimagestextrP returnc sJ |d u r |d u r t djtfdjji|}|d dd }|d dd}|d dd }|d d d
}|d d } |d d }
|d d
d } t } |d urdj|fi |d }
| |
|d urƈj
||||d}| r|st |t rjj
| }nt |trfdd|D }|d d o||d d < |d u r|
nd
|d d < |d u r| nd |d d
< jd%d|i|d }| | | |d d < |
|d d < | |d d
< |d ur#|d ur#|d u rjjd }| }t|d }tt||| }dgdg| dg }g }g }| d }t |t r |g}| d g| d< |D ]7}|d | | ||| d }|| t|}|rFdg| }|dgt|t| 7 }|| q"t |trtdd t|jD dd d}|d \}}|d \}}|d d o||d d < d |d d
< jd%d|| gi|d }t|jd | krjjdkr҇ fd d|D } fd d|D } fd d| d D | d< n'jjd kr fd d|D } fd!d|D } fd"d| d D | d< t |t r| d u r|d }| d d | d< |d }| t|| d |d#| d$ | S )&Nz*You have to specify either images or text.tokenizer_init_kwargsr! r r r* r r r Fr" r# return_tensors)r c s g | ]
} j j | qS r )rO bos_token)r5 srM r r r7 z-Kosmos2Processor.__call__..rW r input_idsattention_maskc S s g | ]
\}}|t |fqS r len)r5 idxr6 r r r r7 r^ c S s | d S Nr )r6 r r r s z+Kosmos2Processor.__call__..)keyrf rightc s& g | ]}|j jg t| qS r rO pad_token_idrc r4 max_len_paddedrM r r r7 & c " g | ]
}|d g t | qS r rb r4 rm r r r7 c ro rp rb r4 rq r r r7 rr leftc s& g | ]}j jg t| | qS r rj r4 rl r r r7 rn c " g | ]
}d g t | | qS rp rb r4 rq r r r7 rr c rt rp rb r4 rq r r r7 rr )r` ra image_embeds_position_mask)data tensor_typer )
ValueError
_merge_kwargsr rO init_kwargspop
setdefaultr rN updatepreprocess_examples
isinstancer2 r[ list unk_token_idr rH rI copyrc sorted enumerater` padding_sider ) rM rV rW rP
output_kwargsr r r r r" r# rZ encodingimage_encoding
text_encodingwith_bos start_indeximage_token_ids base_image_embeds_position_maskr` ru
all_input_idstext_idsmask
sorted_length_min_len_not_paddedrd r rl r __call__ s
zKosmos2Processor.__call__c C s |du rdS t |tstd|D ];}|du rqt |ts |g}|D ])}t |trGt|dkr8tdd |D sKt|dkrGtdd |D sKtdq"qdS ) a
Check `bboxes` for a single text example. It could be
- `None`: no bounding box associated to a text.
- A list with each element being the bounding boxes associated to one ` ... ` pair found
in a text. This could be:
- `None`: no bounding box associated to a ` ... ` pair.
- A tuple of 2 integers: A single bounding box specified by patch indices.
- A tuple of 4 float point number: A single bounding box specified by (normalized) coordinates.
- A list containing the above 2 tuple types: Multiple bounding boxes for a
` ... ` pair.
Nz@`bboxes` (for a single text example) should be `None` or a list. c s | ]}t |tV qd S N)r r r4 r r r / zAKosmos2Processor._check_bboxes_for_single_text..r0 c s r r )r floatr4 r r r r 0 r a' Each element in `bboxes` (for a single text example) should be either `None`, a tuple containing 2 integers or 4 float point numbers, or a list containing such tuples. Also make sure the arguments `texts` and `bboxes` passed to `preprocess_text` are both in batches or both for a single example.)r r rx tuplerc all)rM r bboxelementr r r _check_bboxes_for_single_text s&
z.Kosmos2Processor._check_bboxes_for_single_textc C s. | }|d ur| d| }| ||}|S )N )strip_insert_patch_index_tokens)rM rW imager img_info_tokensr r r _preprocess_single_example9 s
z+Kosmos2Processor._preprocess_single_exampler* textsr r c sB j g| }dj g| jg d}t|tr d}|g}|du r+dgt| }nt|ts3|g}t|t|krJtdt| dt| d|sU| |g}n |durmt|tsbtd|D ]}| qdndgt| }t|t|krtd t| dt| d fd
d t |||D }|s|d }|S )
a- Add image and bounding box information to `texts` as image and patch index tokens.
Args:
texts (`Union[TextInput, list[TextInput]]`): The texts to be processed.
images (`ImageInput`, *optional*): The images associated to `texts`.
bboxes (`Union[list[tuple[int]], list[tuple[float]], list[list[tuple[int]]], list[list[tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional*, defaults to 64):
The number of image tokens (used as latent queries). This should corresponds to the `latent_query_num`
attribute in `Kosmos2Config`.
Returns:
`Union[TextInput, list[TextInput]]`: The processed texts with image and patch index tokens.
r TFNzGThe number of examples in `texts` and `images` should be the same. Got v.s. instead.zS`bboxes` should be `None` or a list (as a batch) when `texts` is passed as a batch.zGThe number of examples in `texts` and `bboxes` should be the same. Got c s" g | ]
\}}} ||| qS r )r )r5 rW r r r rM r r r7 { s z8Kosmos2Processor.preprocess_examples..r )
r< joinr= r r2 rc r rx r zip) rM r rV r r
img_tokensbatchedr6 resultr r r r~ C sD
z$Kosmos2Processor.preprocess_examplesTc C s | | jd }|rt|S |S re )splitr= +clean_text_and_extract_entities_with_bboxes)rM rW cleanup_and_extractcaptionr r r post_process_generation s z(Kosmos2Processor.post_process_generationc s( j |fd|i|} fdd|D S )a
Post-process the output of the model to decode the text.
Args:
generated_outputs (`torch.Tensor` or `np.ndarray`):
The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
or `(sequence_length,)`.
skip_special_tokens (`bool`, *optional*, defaults to `True`):
Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
**kwargs:
Additional arguments to be passed to the tokenizer's `batch_decode method`.
Returns:
`list[str]`: The decoded text.
skip_special_tokensc s g | ] } j |d dqS )F)r )r )r5 rW r] r r r7 s zDKosmos2Processor.post_process_image_text_to_text..) batch_decode)rM generated_outputsr rP generated_textsr r] r post_process_image_text_to_text s z0Kosmos2Processor.post_process_image_text_to_textc C s | j j}| jj}|| dg S )Nru )rO model_input_namesrN )rM tokenizer_input_namesimage_processor_input_namesr r r r s z"Kosmos2Processor.model_input_namesc C sP |d u s
t |dkr |S ttjd|d}t |t |kr,tdt | dt | dd}g }t||D ]\\}}| \}} |||| | }|d u rOq5t|t rW|g}g }
t
dd |D sftd |D ]} | | \} }
|
| d
|
qht |
dkrq5d |
}|d | d
q5|t |k r|||d d |}|S )Nr z.+?)stringzuThe number of elements in `bboxes` should be the same as the number of ` ... ` pairs in `text`. Got r r c s s | ]}|d uV qd S r r )r5 boxr r r r s z>Kosmos2Processor._insert_patch_index_tokens..zTThe multiple bounding boxes for a single phrase should not contain any `None` value.r z z )
rc r refinditerrx r spanrI r r r #_convert_bbox_to_patch_index_tokensr )rM rW r matched_phrasescurr_posbuffermatchedr r endpatch_index_stringsr
patch_index_1
patch_index_2 position_strr r r r sB
z+Kosmos2Processor._insert_patch_index_tokensr c C sh t |dkr |\}}ntt| j}t||\}}dt|d d}dt|d d}||fS )Nr r/ r0 r1 )rc r mathsqrtrG coordinate_to_patch_indexr2 r3 )rM r idx_1idx_2num_patches_per_sidetoken_1token_2r r r r s
z4Kosmos2Processor._convert_bbox_to_patch_index_tokens)r- )NN)NNr* )T)r r r rL r
r r r r r r r r r BboxInputr r2 r~ r r propertyr r r r r
__classcell__r r rT r r, V sR 1
#
B
*-
r, r r rX c C s | \}}}}||kr||kst dt|| }t|| }t|| d }t|| d } || | }
| | | } |
| fS )a Convert a bounding box to a pair of patch indices.
Args:
bbox (`tuple[float, float, float, float]`):
The 4 coordinates of the bounding box, with the format being (x1, y1, x2, y2) specifying the upper-left and
lower-right corners of the box. It should have x2 > x1 and y2 > y1.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`tuple[int, int]`: A pair of patch indices representing the upper-left patch and lower-right patch.
zTThe coordinates in `bbox` should be `(x1, y1, x2, y2)` with `x2 > x1` and `y2 > y1`.r_ )rx r floorceil) r r x1y1x2y2ul_xul_ylr_xlr_yul_idxlr_idxr r r r s r r r c C s d| }| | }| | }|| }|| }| |kr-|| }|| } || | }
|| | } n=||ks5||krJ|| }|| } || | }
|| | } n || |d }|| |d } || |d }
|| |d } || |
| fS )a
Given a grid of length `num_patches_per_side` and the indices of the upper-left and lower-right corners of a
bounding box, returns the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
Args:
ul_idx (`int`): the index of the grid cell that corresponds to the upper-left corner of the bounding box.
lr_idx (`int`): the index of the grid cell that corresponds to the lower-right corner of the bounding box.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`tuple[float]`: the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
g ?r r ) r r r cell_sizer r r r r r r r r r r patch_index_to_coordinate s( r c C s4 d}t || }g }|D ]}|d}| \}}}|s,d}|dd |dd f}|d} g }
| D ];} t d| } t d| dd }
| rp|
rp|r_|
t| dt|
df q5|
t| dt|
df q5|r|||||
f q |
D ]}d|d d |d d
}||||gf q~q |S ) a Extract entities contained in `text`. The bounding bboxes is given in the form of patch indices.
This function is only intended to be used within `clean_text_and_extract_entities_with_bboxes` where further
processing happens, including converting to normalized coordinates and whitespace character cleaning up.
Examples:
```python
>>> text = " An image of a snowman warming himself by a fire."
>>> entities = extract_entities_with_patch_indices(text)
>>> entities
[(' a snowman', (31, 41), [(44, 863)]), (' a fire', (130, 137), [(5, 911)])]
```z(?:(([^<]+)))?r Nr r. zr_ r/ z>r N)rc r sub)r rW entity_namestartr adjusted_start adjusted_endadjusted_entityr r r adjust_entity_positionsi s
r c C s | }t| t| }g }|D ]5\}\}}}t|t| } t|t| }
|| | }|| |
}| }||||f|f q||fS )z9Remove the spaces around the text and the entities in it.)r rc r8 r9 rI ) rW entitiesnew_textleading_spaces new_entitiesr r r r entity_name_leading_spacesentity_name_trailing_spacesr r r _cleanup_spacess s r c
sp t dd| }t| }g }|D ]#}|dd |d }}t|| } fdd|D } ||| f qt||S )a Remove the tag tokens from `text`, extract entities in it with some cleaning up of white characters.
Examples:
```python
>>> text = " An image of a snowman warming himself by a fire."
>>> clean_text, entities = clean_text_and_extract_entities_with_bboxes(text)
>>> clean_text
'An image of a snowman warming himself by a fire.'
>>> entities
[('a snowman', (12, 21), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('a fire', (41, 47), [(0.171875, 0.015625, 0.484375, 0.890625)])]
```r r r r c s g | ] }t |d |d qS )r r_ )r )r5 r r r r r7 s z?clean_text_and_extract_entities_with_bboxes..)r r r r rI r )
rW r processed_textr r itemr r r bboxes_in_coordsr r r r s
r )r )&r r r r image_processing_utilsr image_utilsr processing_utilsr r r r r tokenization_pythonr
tokenization_utils_baser r utilsr
r r r r r r r r r r, r r r r r r __all__r r r r sD
0 * -: