o پi^ @s`dZddlZddlZddlZddlZddlmZddlmZddl Z ddl Z ddl Z ddl mZddlmZmZddlmZddlmZd efd d Zd d ZddZddZddZddZddZddZddZddZ ddZ! d/d!e"e#d"e#d e$e"e#e"e#e"e#ffd#d$Z%d%e j&d&e j'j(d e j&fd'd(Z)d&e j'j(d)e j&d*e"d+ed,fd-d.Z*dS)0a Utilities for multi-modal models. This python file mainly contains utilities that were used in the image processing logic of llava-next including operations such as anyres and anyres_max Currently supports the anyres and anyres_max operation for CLIP and SigLip. For more information, you may refer to the paper or the blog LLaVA-NeXT : https://llava-vl.github.io/blog/2024-01-30-llava-next/ LLaVA-Onevision : https://arxiv.org/pdf/2408.03326 N)BytesIO)Literal)Image)get_tensor_model_parallel_rank$get_tensor_model_parallel_world_size) tensor_model_parallel_all_gather)flatten_nested_listreturncCs0|durdSt|trtddt|DSdS)NFcss|]}t|VqdSN)has_valid_data).0itemrR/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/multimodal/mm_utils.py 7sz!has_valid_data..T) isinstancelistanyr)datarrrr 3s  r cCs|\}}d}d}td}|D]=\}}t||||} t|| t|| } } t| | ||} ||| } | |ksC| |krK| |krK| }| }||f}q|S)a Selects the best resolution from a list of possible resolutions based on the original size. Args: original_size (tuple): The original size of the image in the format (width, height). possible_resolutions (list): A list of possible resolutions in the format [(width1, height1), (width2, height2), ...]. Returns: tuple: The best fit resolution in the format (width, height). Nrinf)floatminint) original_sizepossible_resolutionsoriginal_widthoriginal_heightbest_fitmax_effective_resolutionmin_wasted_resolutionwidthheightscaledownscaled_widthdownscaled_heighteffective_resolutionwasted_resolutionrrrselect_best_resolution;s*     r'cCs|j\}}|\}}||}||}||kr"|}tt|||} n |} tt|||}||| f} td||fd} ||d} || d} | | | | f| S)a1 Resize and pad an image to a target resolution while maintaining aspect ratio. Args: image (PIL.Image.Image): The input image. target_resolution (tuple): The target resolution (width, height) of the image. Returns: PIL.Image.Image: The resized and padded image. RGB)rrr)sizermathceilresizernewpaste)imagetarget_resolutionrr target_width target_heightscale_wscale_h new_width new_height resized_image new_imagepaste_xpaste_yrrrresize_and_pad_imagecs   r<c Cs^g}|j\}}td||D]}td||D]}||||||f}||}||qq |S)a Divides an image into patches of a specified size. Args: image (PIL.Image.Image): The input image. patch_size (int): The size of each patch. Returns: list: A list of PIL.Image.Image objects representing the patches. r)r*rangecropappend) r0 patch_sizepatchesr r!ijboxpatchrrrdivide_to_patchess   rFcst|trEd|vrEdvsJdtd|}ttt|dttt|dfddtddd D}fd d|D}t|t urN|}nt |}t ||\}}||fS) a Calculate the shape of the image patch grid after the preprocessing for images of any resolution. Args: image_size (tuple): The size of the input image in the format (width, height). grid_pinpoints (str): A string representation of a list of possible resolutions. patch_size (int): The size of each image patch. Returns: tuple: The shape of the image patch grid in the format (width, height). xiPiii1patch_size should be in [224, 336, 384, 448, 512]\((\d+)x(\d+)\)rc0g|]}tdddD]}||fqqSr=r rBrC range_end range_startrr z/get_anyres_image_grid_shape..rOcg|] }fdd|DqS)cg|]}|qSrrr dimr@rrrUz:get_anyres_image_grid_shape...rr pairr[rrrU) rstrrefindalltuplemaprr=typerast literal_evalr') image_sizegrid_pinpointsr@matchesrr r!r)r@rSrTrget_anyres_image_grid_shapes      rkc sxt|trdd|vrdzjdWnty'}z jdWYd}~nd}~wwdvs0Jdtd|}ttt|dttt|dfd d t ddd D}fd d |D}t |t urm|}nt |}t|j|}t||}d jvrjdnjd}djvrjdnjd} t||} || | f} | g| } fdd | D} tj| ddS)a^ Process an image with variable resolutions. Args: image (PIL.Image.Image): The input image to be processed. processor: The image processor object. grid_pinpoints (str): A string representation of a list of possible resolutions. Returns: np.array: An np array containing the processed image patches. rGr shortest_edgeNrHrJrKrLcrMrNrPrQrRrrrUrVz(process_anyres_image..rOcrW)crXrrrYr[rrrUr\z3process_anyres_image...rr]r[rrrUr_ crop_sizer!cs$g|]}|dddqS)r( pixel_valuesr) preprocessconvert)r image_patch) processorrrrUaxis)rr`r* Exceptionrarbrcrdrr=rerrfrgr'r<__dict__rmrFr-npstack) r0rrrierjrbest_resolution image_paddedrmrlrAimage_original_resize image_patchesr)r@rrrSrTrprocess_anyres_imagesJ               rcCstttj|ddS)NT)validate)ropenrpybase64 b64decode)r0rrrload_image_from_base64srcCs|j\}}||kr |S|jdkr|d}||kr1t|j||f|}||d||df|St|j||f|}||||ddf|S)NLr(rr))r*moderprr.r/)pil_imgbackground_colorr r!resultrrr expand2square s   rc Cs|\}}|jdd\}}||}||}||kr:||}t||} || d} |dd| || ddf} | S||}t||} || d} |dddd| || f} | S)a Unpads a PyTorch tensor of a padded and resized image. Args: tensor (torch.Tensor): The image tensor, assumed to be in CxHxW format. original_size (tuple): The original size of the image (height, width). Returns: torch.Tensor: The unpadded image tensor. rONr))shaper) tensorrrrcurrent_height current_widthoriginal_aspect_ratiocurrent_aspect_ratio scale_factorr7paddingunpadded_tensorr6rrr unpad_images     rc Cs|\}}||}||}||kr*||}t||}||d} |d| |f} | S||}t||} || d} ||d| f} | S)z^ Unpads a PyTorch tensor of a padded and resized image and returns the new shape. r)r) rrrrrrrrr7r new_shaper6rrrunpad_image_shape:s    rcst|dd}g|dkr-|D]}t|tdd|jD}||dd}|qnd|vrC|D]}t|||j}|q3n||dStfddDr[t j dd S) Nimage_aspect_ratiopadcss|] }t|dVqdS)Nrr rGrrrrXsz!process_images..rnranyresc3s |] }|jdjkVqdS)rN)rr new_imagesrrrdsrt) getattrrrc image_meanror?rimage_grid_pinpointsallrxry)imagesimage_processor model_cfgrr0rrrprocess_imagesRs*    rr)sizesnum_gpusc st}|dkrgdg|dg|fSddt|D}dg|tt|fdddd}|D]}tt|fddd }|||||7<q0tt}tt}t|D]} ||| |t|| q^||fS) aq Generate load balancing assignment and metadata for distributing data across GPUs. The load is determined by the total image sizes, not the number of images. Args: sizes: The size of each image num_gpus: Number of GPUs to balance across Returns: shuffle_indices: Indices to reorder data for balanced loading gpu_sample_counts: Number of samples assigned to each GPU grouped_sizes_per_gpu: Total size assigned to each GPU Example: ``` sizes = [1000, 100, 200, 50] num_gpus = 2 ``` rcSsg|]}ttqSr)rr)r _rrrrUz0get_dp_encoder_lb_assignment..c|Sr rrB)rrrz.get_dp_encoder_lb_assignment..T)keyreversecrr rr) gpu_loadsrrrr)r)lenr=sortedrr?rrextend) rr n_samplesgpu_assignmentslarge_to_small_indicesidxmin_gpushuffle_indicesgpu_sample_countsgpu_idr)rrrget_dp_encoder_lb_assignmentjs$     r image_input vision_modelc Cs|jd}t}||d|}|||}dd|dd|f}tjj||}t}||||d|df} || } | j } t | dd} | d|df} | S)aWRun a vision model with data parallelism (DP) sharding. The function will shard the input image tensor on the first dimension and run the vision model Args: image_input (torch.Tensor): Image input tensor. vision_model (torch.nn.Module): Vision model. Returns: torch.Tensor: Output image embeddings rrO)rr).rZN) rrrZtorchnn functionalrrlast_hidden_state contiguousr) rr num_chunks mp_world_sizenum_chunks_per_ranknum_padded_chunksrimage_input_paddedrankimage_input_per_rankvision_embeddingsrrrrun_dp_sharded_vision_models    rrn grid_thw_list rope_type)rope_3drope_2dc'sHddlm}m}m}|}|dkr|tdS|}ddD} dgt| t| |\} } } dgt| } | | || |d}t |dkr\t fdd|D}ntj dj dfj jd}|d krz|jd|jdn|j|jt| }fd d|D}|d kr|j ddkr||t|}t|trtj |dd }n2t|jd d }tj d|fj jd}n|j ddkr||t|}n tj d|jfj jd}|j d}||kr!||}|d krtj ||j d|j df|j|j d}ntj ||j df|j|j d}tj ||gdd }n|}|j|dd }ttj}t|D]}||}|| |}||||q5fdd| D}d gt }d}t|D]6}| |} | dkr| ||| }!||}"d}#|!D]}$||$}%|"|#|#|%||$<|#|%7}#q|| 7}qdtj |dd }&|&S)aRun a vision model with data parallelism (DP) sharding. The function will shard the input image tensor on the first dimension and run the vision model. This function is used to run the vision model with mrope. Args: vision_model (torch.nn.Module): Vision model. pixel_values (torch.Tensor): Image/Video input tensor. grid_thw_list: List of grid dimensions for each image rope_type: Type of rope used in the vision model. Different rope types have different dimension to do ViT. "rope_3d" for 3D rope (e.g., Qwen2.5-VL) "rope_2d" for 2D rope (e.g., Kimi-VL) Returns: torch.Tensor: Output image embeddings Example: ``` vision_model.out_hidden_size = 64 vision_model.spatial_merge_size = 2 pixel_values.shape = (1350, channel) grid_thw_list = [[1, 10, 100], [1, 10, 10], [1, 10, 20], [1, 50]] tp_size = 2 ``` r)get_attention_tp_groupget_attention_tp_rankget_attention_tp_sizerO)grid_thwcSsg|]}t|qSr)r+prod)r rrrrrUrz5run_dp_sharded_mrope_vision_model..cs$g|]}||dqSrNrr rB)cum_patches_per_imagernrrrUrs)devicedtypercsg|]}|qSrrr)rrrrU6r\r hidden_sizeNr))rrcsg|]}|qSrr)r r@)embed_dim_reduction_factorrrrU|s)sglang.srt.layers.dp_attentionrrrrr itertools accumulaterrcatemptyrrrmerge_kernel_sizespatial_merge_sizemaxrrrconfigout_hidden_size all_gatherTensorr=r?)'rrnrrrrrtp_size tp_rank_localpatches_per_imageimage_to_tp_rankrgrouped_pixel_values_lencum_gpu_sample_countsimage_idxs_localpixel_values_localmax_len_per_ranklocal_grid_thw_listimage_embeds_localout_dim current_len padding_sizerimage_embeds_local_paddedgathered_embedsrank_embeddingsr start_idxend_idxpatches_per_output_imageoriginal_order_embeddings current_idxcount rank_images rank_embed embed_startimg_idx img_patchesout_embeddingsr)rrrrnr!run_dp_sharded_mrope_vision_models!                    r)r))+__doc__rfrr+raiortypingrnumpyrxrrPILrsglang.srt.distributedrr'sglang.srt.distributed.communication_oprsglang.srt.utilsrboolr r'r<rFrkrrrrrrrrrcrrrModulerrrrrrsb     (#(C F "