o *i @srddlZddlZddlmZddlmZddlmZddlZ ddl m Z ddl m Z mZddlmZmZddlmZmZeeZerHddlZeGd d d ZeGd d d Zd eeBde fddZdZdZGdddeeZeGdddZ de j de j fddZ!de j"de#e$e$e$fde#e$e$e$fde j"fddZ%de j de#e&e&fde j"fd d!Z'Gd"d#d#Z(dS)$N) dataclass)Enum)BytesIO)Image)SerializableImagedownload_image)assert_opencv_installedis_opencv_installed) ImageChunk ImageURLChunkc@s(eZdZUdZeeed<ejed<dS) ImageEncodingzA tokenized image. Attributes: tokens: The token ids. image: The image as a numpy array. Examples: >>> import numpy as np >>> image_encoding = ImageEncoding(tokens=[1, 2, 3], image=np.array([[0., 0.5, 1.]])) tokensimageN) __name__ __module__ __qualname____doc__listint__annotations__npndarrayrrc/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/image.pyr s  r c@s*eZdZUdZeed<eed<eed<dS)SpecialImageIDsaSpecial image tokens ids. Attributes: img: The image token id. img_break: The image break token id. img_end: The image end token id. Examples: >>> special_image_ids = SpecialImageIDs(img=1, img_break=2, img_end=3) img img_breakimg_endN)rrrrrrrrrrr&s   rchunkreturncCst|tr|jS|dr$|dd}t|}t t |S|dr9t t | dddS|drFt |St d |) zGet a serializable image from a chunk. Args: chunk: The chunk to get the image from. Returns: The image as a PIL Image object. z data:image,filezfile://rbhttpzUnsupported image url scheme ) isinstancer rget_url startswithsplitbase64 b64decoderopenrreplacer RuntimeError)rdata image_datarrrimage_from_chunk8s   r1)g3<4'?gwg M?gy{ ?)gB91?gwt.?g U?c@s"eZdZdZdZedddZdS) MultiModalVersionzVersion of the image tokenizer.m1r ImageConfigcCs |jdkr tddSt|j)Nr3i)namer4NotImplementedErrorselfrrrconfigYs   zMultiModalVersion.configN)rr4)rrrrr3propertyr:rrrrr2Ts r2c@s.eZdZUdZeed<eed<dZeed<dS)r4z'Configuration for the image tokenizers.image_patch_sizemax_image_sizer!spatial_merge_sizeN)rrrrrrr>rrrrr4as r4rcCsJ|jdkr|S|jdkr|d}td|jd}||d||dS)zUConvert a PIL image to RGB. We ensure transparent background becomes white. RGBRGBAWHITE)rr)modeconvertrnewsizepaste)rwhite_bgrrr_convert_to_rgbjs    rHnp_imagemeanstdcCs|d}t|jdksJd|j|jdt|kr$t|ks3nJd|jd|d||||}|dddS) aNormalize a tensor image with mean and standard deviation. Args: np_image: Image to be normalized. mean: Mean for each channel. std: Standard deviation for each channel. Returns: Normalized image with shape (C, H, W). go@znp_image.shape=z, mean=z, std=rr!)lenshape transpose)rIrJrKrrr normalizexs @ rQnew_sizecCs4ttjtjt|tjd|tjd}t|t t S)zTransform an image to a numpy array with the given size. Args: image: Image to be transformed. new_size: New size of the image. Returns: Transformed image with shape (C, H, W). )dtype) interpolation) rcv2resizerarrayrHfloat32 INTER_CUBICrQ DATASET_MEAN DATASET_STD)rrRrIrrrtransform_images " r\c@s|eZdZdZdededdfddZedefdd Zd e j de e e ffd d Z d e eBdefddZede fddZdS) ImageEncoderz&Image encoder for the image tokenizer. image_config special_idsrNcCs||_||_dS)zInitialize the image encoder. Args: image_config: Configuration for the image tokenizer. special_ids: Special image tokens ids. N)r^r_)r9r^r_rrr__init__s zImageEncoder.__init__cCs|jSN)r^r8rrr mm_configszImageEncoder.mm_configrcCs|j\}}t||jj||jj}|dkr"t||}t||}|d|jj|jjd}|d|jj|jjd}||fS)Nr!)rEmaxr^r=roundr<r>)r9rwhratio width_tokens height_tokensrrr_image_to_num_tokenss   z!ImageEncoder._image_to_num_tokenscontentcCst|}||\}}|dksJ|dksJ|jjg||jjg|}|jj|d<||jj|jj||jj|jjf}t ||}t ||dS)zConverts an image chunk to an image encoding. Args: content: image chunk to be converted. Returns: Image encoding. r)r r) r1rjr_rrrr^r<r>r\r )r9rkrrerf image_tokensnew_image_sizeprocessed_imagerrr__call__s      zImageEncoder.__call__cCs|jjSra)r_rr8rrr image_tokenszImageEncoder.image_token)rrrrr4rr`r;rbrtuplerrjr r r rprqrrrrr]s r]))r*logging dataclassesrenumriornumpyrPILrmistral_common.imagerrmistral_common.importsrr &mistral_common.protocol.instruct.chunkr r getLoggerrloggerrUr rr1rZr[strr2r4rHrrrfloatrQrr\r]rrrrsF         "