o ۾ih& @s ddlZddlZddlmZddlmZmZddlmZddl m Z m Z ddl Z ddlm ZddlmZddlmZdd lmZdd lmZmZmZmZmZdd lmZmZmZm Z e raddl!Z"ned e#d Z"d e$fddZ%ddde j&de'de$de$fddZ(ddde j&de'de$de$fddZ)ddddejde$de$de$fdd Z*ddddejde$de$de$fd!d"Z+d#dd$ej,de$de$fd%d&Z-d#dd$ej,de$de$fd'd(Z.d)ede/e0e$e'ffd*d+Z1d,efd-d.Z2dd/d0d1eed2e"j3j4d3e5fd4d5Z6dd/d0d1eed2e"j3j4d3e5dee0e'effd6d7Z7dd/d0d8e/e0e$efd2e"j3j4d3e5dee0e$e'efddffd9d:Z8 dGd;e$dZ; dGd?e$d@e9e$e fdBdejfdAdBZ< dGdCe$dDe9e$e fdBde0ej,e9e$e fffdEdFZ=dS)HN) defaultdict) GeneratorSequence)groupby) TYPE_CHECKINGAny)Image) LazyLoader)MultiModalHasher)BatchedTensorInputsMultiModalFieldElemMultiModalKwargsItemMultiModalPlaceholderDictMultiModalSharedField) AudioMediaIO ImageMediaIOMediaConnector VideoMediaIOtorchnamecCs<|dkrddlm}tjdtdd|Stdtd|) NMEDIA_CONNECTOR_REGISTRYr )rz`vllm.multimodal.utils.MEDIA_CONNECTOR_REGISTRY` has been moved to `vllm.multimodal.media.MEDIA_CONNECTOR_REGISTRY`. The old name will be removed in v0.17.) stacklevelzmodule z has no attribute )mediarwarningswarnDeprecationWarningAttributeError__name__)rrr I/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/utils.py __getattr__!s r"WAVformataudio sampling_rater%returncCst}|j||f|dS)zEncode audio as base64.) audio_format)r encode_base64)r&r'r%audio_ior r r!encode_audio_base642sr,cC4t|||d}tjd|d}d|d|S)zEncode audio as a data URL.r$.r&data:;base64,)r, mimetypes types_mapgetlower)r&r'r% audio_b64mimetyper r r!encode_audio_url=sr7RGBPNG image_moder%imager;cCst|d}|j||dS)z Encode a pillow image to base64 format. By default, the image is converted into RGB format before being encoded. )r;) image_format)rr*)r<r;r%image_ior r r!encode_image_base64Is r?cCr-)z| Encode a pillow image as a data URL. By default, the image is converted into RGB format before being encoded. r:r.r<r/r0)r?r1r2r3r4)r<r;r% image_b64r6r r r!encode_image_urlXs rAJPEGframescCst}t|}|j||dS)N) video_format)rrr*)rCr%r>video_ior r r!encode_video_base64hsrFcCsDt||d}|dkrd}n tjd|d}d|d|S)Nr$jpegz video/jpegr.videor/r0)rFr4r1r2r3)rCr% video_b64r6r r r!encode_video_urlrs  rJ mm_positionscCs0dd|D}t|ddd}dd|DS)a/ Given a `MultiModalPlaceholderDict`, output a sequence of keys to sort the dictionary by `offset` (starting index in the input sequence) in ascending order. Returns: A list of `(modality, idx)`, which can be used to access an item by `mm_positions[modality][idx]`. css0|]\}}t|D] \}}|||fVq qdSN) enumerate).0modalityitemsidxitemr r r! s z'argsort_mm_positions..cSs |djS)Nr)offsetxr r r!s z&argsort_mm_positions..keycSsg|] \}}}||fqSr r )rNrOrQ_r r r! sz(argsort_mm_positions..)rPsorted)rK flat_itemssorted_flat_itemsr r r!argsort_mm_positionss  r_elemcCst|jtsdStj|jdS)N)data) isinstancefieldrr hash_kwargsra)r`r r r!_get_group_hashs reFdevice pin_memoryrPrgrhcsVttttft}|D]}|D] \}}|||qq fdd|DS)Ncs(i|]\}}||djj|dqS)rrf)rc reduce_data)rNrYelemsrfr r! s z#_batch_mm_items..)rstrlistr rPappend)rPrgrhrjrRrYr`r rfr!_batch_mm_itemss roccspdd|D}ddt|D}d}|D]}t||||||d}||fV||7}q|t|ks6JdS)a Group consecutive items (possibly from different requests) into batches. Items must be split across groups if any of the following occurs, as the batch would otherwise be invalid: - They have different fields (e.g. mixed image and embedding inputs). - They have different values in `MultiModalSharedField`. Args: items: List of `MultiModalKwargsItem`. device: The device to place the grouped tensors on. pin_memory: Whether to pin memory for faster host-to-device transfer. Yields: A tuple `(num_items, grouped_kwargs)`, where: - `kwargs` is a dictionary of keyword arguments to pass to the model; - `num_items` is the corresponding number of items. cSs.g|]}tddt|dddDqS)css |] \}}|t|fVqdSrL)re)rNrYr`r r r!rSs  6group_and_batch_mm_items...cS|dSNrr )kvr r r!rWz5group_and_batch_mm_items...rX)tupler\rP)rNrRr r r!r[s z,group_and_batch_mm_items..cSs"g|] \}}tdd|DqS)css|]}dVqdS)r Nr )rNrZr r r!rSsrp)sum)rNrZgroupr r r!r[s"rrfN)rrolen)rPrgrh group_ids group_sizes start_idx group_size group_datar r r!group_and_batch_mm_itemss  r~ mm_kwargsccsRt|dddD]\}}dd|D}t|||dD] \}}|||fVqq dS)a Group consecutive items (possibly from different requests) into batches. Items must be split across groups if any of the following occurs, as the batch would otherwise be invalid: - They have different fields (e.g. mixed image and embedding inputs). - They have different values in `MultiModalSharedField`. To simplify the implementation of `embed_multimodal`, we add another restriction that the items in a batch must belong to the same modality. Args: mm_kwargs: List of `(modality, item)`. device: The device to place the grouped tensors on. pin_memory: Whether to pin memory for faster host-to-device transfer. Yields: A tuple `(modality, num_items, grouped_kwargs)`, where: - `modality` is the modality of the batch; - `kwargs` is a dictionary of keyword arguments to pass to the model; - `num_items` is the corresponding number of items. cSrqrrr rUr r r!rWrtz-group_mm_kwargs_by_modality..rXcSsg|]\}}|qSr r )rNrZrRr r r!r[sz/group_mm_kwargs_by_modality..rfN)rr~)rrgrhrOrw items_lst num_itemsmm_kwargs_batchr r r!group_mm_kwargs_by_modalitysr audio_urlaudio_io_kwargscC&|sdnd|i}t|dd}||S)a+ Args: audio_url: URL of the audio file to fetch. audio_io_kwargs: Additional kwargs passed to handle audio IO. Warning: This method has direct access to local files and is only intended to be called by user code. Never call this from the online server! Nr&/media_io_kwargsallowed_local_media_path)r fetch_audio)rrrmedia_connectorr r r!r    r image_urlimage_io_kwargscCr)a+ Args: image_url: URL of the image file to fetch. image_io_kwargs: Additional kwargs passed to handle image IO. Warning: This method has direct access to local files and is only intended to be called by user code. Never call this from the online server! Nr<rr)r fetch_image)rrrrr r r!r rr video_urlvideo_io_kwargscCr)a+ Args: video_url: URL of the video file to fetch. video_io_kwargs: Additional kwargs passed to handle video IO. Warning: This method has direct access to local files and is only intended to be called by user code. Never call this from the online server! NrHrr)r fetch_video)rrrrr r r!r5rrrL)>r1r collectionsrcollections.abcrr itertoolsrtypingrrnumpynp numpy.typingnptPILrvllm.utils.import_utilsr hasherr inputsr r rrrrrrrr torch.typesrglobalsrlr"ndarrayintr,r7r?rANDArrayrFrJrmrur_retypesDeviceboolror~rdictfloatrrrr r r r!s                  3 )