o ίi@sdZddlZddlmZddlmZmZmZmZeGdddZ e dgdd ddddddd dd d Z e d gdd ddddddddd d Z e e dZ dS)a Define conversation format for each training phases and language models. Modified from LLaVA codebase: https://github.com/haotian-liu/LLaVA/blob/main/llava/conversation.py NOTE: - an example of required json format is: data = { "image": IMAGE_PATH, or "images": LIST of IMAGE_PATH, "conversations": [ {"from": "human", "value": "hello"}, {"from": "assistant", "value": "Hi, how can I help you today?"}, {"from": "human", "value": "Who are you?"}, {"from": "assistant", "value": "I am a multimodal large language model created by FAIR. I can assist you with questions related to images and videos."}, ] } N) dataclass)CallableDictListUnionc @seZdZUeed<eed<eed<eed<eed<eed<eed<dZeed <d Zeed <d Z eed <d Z eed <d Z eed< d!de de dede efddZ d"dede de fddZdee effddZddZd S)# Conversationsystem conversations bos_token sep_system sep_question sep_answerplace_image_token <|image|> image_token pre_system pre_question pre_answer eos_tokenimage num_images num_patches media_typereturnc Csg}|j|j|j}d}|dkr+|jD]}|ddkr)|dd|j||d<qn8|jdddd d d dd d d d d d d |jdd<||jdd|j|||jdd<|jD]T}|rq|ddkrqqf|ddkrd } |r| |7} | |j|d|j7} d| i} d}qf|ddkr|j t |d|j } | d| i| | qftd|dd|j|dd|dd<|dd|j|dd<|S)zR Each turn of conversation is a dict with source and target keys. T multi_imagefromhumanvaluer r