o ٷi:@sBddlmZmZmZddlZddlZddlmZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&ddl'm(Z(ddl)m*Z*erddl+m,Z,nede-dZ.ede-dZ/ee0Z1Gddde$Z2dS)) TYPE_CHECKINGAnycastN)CUDAGraphWrapper) CUDAGraphMode) get_pp_group)set_forward_context) init_logger)MRotaryEmbedding)supports_mrope)VllmModelForPooling) SamplingType) instrument) LazyLoader)cdiv) EagleProposer)CachedRequestState)GPUModelRunnerIntermediateTensorsPerLayerAttnMetadata)maybe_create_ubatch_slices) OmniOutput)SchedulerOutputxgrxgrammarxgr_torch_compilez:xgrammar.kernels.apply_token_bitmask_inplace_torch_compilecsreZdZfddZfddZedddRfd d Zd efd d ZdSfdd Z dTddZ dTddZ e de jee jBeBdefddZe           dUdededBdeded ed!ed"ed#ed$ed%ed&ed'edBdee je jffd(d)ZdTd*d+Zdeefd,d-Zdeeeeffd.d/Zdefd0d1Zde jd2ed3ejddddf d4d5Zd3ejdee effd6d7Z!d8e d9eddfd:d;Z"dTdedBd?e dedBfd@dAZ$ dVdddBedCe%dBfdDdEZ&dFee dGe jddfdHdIZ'    dWdJe jdBdKe jdBdCe%dBdGe jdBdLee e(ff fdMdN Z)d?e dOedBddfdPdQZ*Z+S)XOmniGPUModelRunnercs(tj|i|d|_d|_d|_dSN)super__init__$_omni_per_req_additional_information_omni_num_scheduled_tokens_np_omni_last_model_output)selfargskwargs __class__U/home/ubuntu/.local/lib/python3.10/site-packages/vllm_omni/worker/gpu_model_runner.pyr&s zOmniGPUModelRunner.__init__c st|||jD];}|D]6}|jD]0}t|dd}t|dd}|durC|dkrC|jj|d}|jd|krCtj ||j |j d|_ qqq dS)aOverride to fix scheduler_metadata buffer size for FA3 + CUDA graph. The upstream FlashAttentionMetadataBuilder pre-allocates scheduler_metadata with (max_num_seqs + 1) entries, but FA3's get_scheduler_metadata() can return up to (max_num_seqs * max_num_splits + 1) entries, causing a RuntimeError during CUDA graph capture. After calling the parent implementation we resize any too-small buffers. scheduler_metadataNmax_num_splitsr)dtypedevice) rinitialize_metadata_builders attn_groupsmetadata_buildersgetattrscheduler_config max_num_seqsshapetorchzerosr-r.r*) r#kv_cache_configkernel_block_sizeskv_cache_group attn_groupbuildersmr+requiredr&r(r)r/,s$    z/OmniGPUModelRunner.initialize_metadata_buildersz Loading (GPU)) span_namereturnNcstj|i|t|jdd}|dur~||_|jj}|dus!Jt|jdddu}|r:|r:t||j t j d|_t t|jddpHt|j jd}t|j|jj}|j|tjd|_|j|||jdd |_|j|||jdd |_|j|||jdd |_dSdS) N talker_mtptalker) runtime_modemtp_hidden_sizer hidden_sizer-F)r-numpy)r load_modelr2modelrAcompilation_configcudagraph_modehas_full_cudagraphsr vllm_configrFULLint model_confighf_text_configmax max_num_reqsmax_cudagraph_capture_size _make_bufferr6int32talker_mtp_input_idsr-talker_mtp_inputs_embedslast_talker_hidden text_step)r#r$r%rArKhas_separate_talkerrEmax_batch_sizer&r(r)rHFs(   zOmniGPUModelRunner.load_model req_statec Cs4g}g}g}g}d}|jD]Y}|j}|durq |} | d} dur+|| | d} dur;|| | d} durI|| | d} durW|| | d} | durft| }q t| r|j j |j |j|j j|||||d\|_|_dStj|j |j j|||||d \|_|_dS) aInitialize M-RoPE positions for multimodal inputs. Extracts multimodal feature metadata (image grids, video grids, audio features) and computes M-RoPE positions for proper positional encoding of multimodal tokens. Args: req_state: Cached request state containing multimodal features Raises: AssertionError: If the model does not support M-RoPE FNimage_grid_thwvideo_grid_thwsecond_per_grid_tsaudio_feature_lengthsuse_audio_in_video) mm_features hf_configr^r_r`rarb)rdr^r_r`rarb)rcdataget_datagetappendtolistboolitemr get_modelrIget_mrope_input_positionsprompt_token_idsrPrdmrope_positionsmrope_position_deltar get_input_positions_tensor) r#r]r^r_r`rarb mm_featuremm_itemmm_inputtuse_audio_in_video_valuer(r(r)_init_mrope_positionscsT        z(OmniGPUModelRunner._init_mrope_positionsscheduler_outputrcs.t|t|ddsdS||dS)a@Calculate M-RoPE positions for scheduled tokens. Delegates to the upstream implementation first, then applies a fixup pass for models that pre-compute 2D spatial decode positions (e.g. GLM-Image). This avoids duplicating the full upstream method while still supporting non-linear decode position patterns. Models opt-in by declaring ``precomputed_mrope_decode = True`` as a class attribute. When set, ``get_mrope_input_positions`` is expected to return positions covering **both** prefill and decode tokens. precomputed_mrope_decodeFN)r_calc_mrope_positionsr2rl)_fixup_precomputed_mrope_decode_positions)r#rxr&r(r)rzs z(OmniGPUModelRunner._calc_mrope_positionscCsddlm}d}t|jjD]r\}}|j|}|jdusJ|jj|}|j|}||j |j } ||| krEt d| |} t d|| } n|} d} || 7}| dkr|} || } | | }|jj d}||kr||jdd| |f|jj dd| | | f<|| 7}qdS)aOverwrite linear decode M-RoPE positions with pre-computed ones. For image-generation models (like GLM-Image) that output tokens in 2D grid order, ``get_mrope_input_positions`` returns positions for the full sequence (prefill + decode). The upstream runner only uses the prefill portion and falls back to linear increments for decode. This method patches the decode slice with the correct pre-computed values. r)&length_from_prompt_token_ids_or_embedsNr,) vllm.utilsr| enumerate input_batchreq_idsrequestsronum_computed_tokens_cpunum_scheduled_tokensrn prompt_embedsrRr5cpu)r#rxr| mrope_pos_ptrindexreq_idreqnum_computed_tokensrnum_prompt_tokensprompt_part_lencompletion_part_len dst_start decode_start decode_endtotal_precomputedr(r(r)r{s4      zt(| dddur| j#}t(t)|j*}t)j+|j,|d}|-|j.}t/|}t0|j|d|z|| _#Wn t1yYnwWnt1y}zt23d|WYd}~nd}~wwzit(| d ddurx| j4}i}t5|t6r(|}nDd d l7m8}t5||rl|j9:D]2\}}|j;duret)*t(|d d }t)j+|j;|d}|-|j<}t/|=||<q9|j>||<q9|rxt0|j|d|Wnt1y}zt23d|WYd}~nd}~ww| r| j?dur| j?dkr|jj@n| j?|j|<|jAr|B||jCd kr|D||E|j|qRtFjG}|j }|jH}|I}tJ|jKD]:\} }|j|}|j&| }!|jL| }"||j v}#|jM| }$|jj N|}%|jOr;|jPr;|%durd |_On'|jjQdusJ|jjQ|}&||&d}'|jO|'}(|!|(8}!|jRSdg|'|!|_&|sn|jT| })|!tU|)|jV}*|*dkr]|jRE|)dn2|*d krm|jRS|)|* dn!|$tU|jRkr|jR|$d=|%dur|jjW|%|$}+|+|jjX|%<|#s|"durtY|j%|"D] \},}-|,S|-qn|%dusJ|"dusJ|"|_%|jj N|}%|%dur|jPr|$d kr|jZ|}.|.|$ d|_R|E|q|!|jj[|%<|"dur|jj\]|"|%|s|!}/|!tU|)}0|)|jj^|%|/|0f<|0|jjX|%<|j_||q|D]}1|j`|1|j_|1|q|ja|b||jcdS)abUpdate the cached states and the persistent batch with the scheduler output. The updated states are used by the `_prepare_inputs` function to create the input GPU tensors for the model. The SamplingMetadata is updated and copied to the GPU if there is a new/resumed/paused/finished request in the batch. N)r.z!You did not set `task` in the API) rrnrrcsampling_paramspooling_params generator block_idsroutput_token_ids lora_requestrrFprompt_embeds_cpuzError decoding prompt embeds: additional_informationr)AdditionalInformationPayload tensor_dtypefloat32additional_information_cpuz'Error decoding additional information: r,)dfinished_req_idsrpopnum_prompt_logprobsrremove_requestfree_encoder_mm_hashes encoder_cacherkeysreq_id_to_indexscheduled_cached_reqsresumed_req_idsscheduled_new_reqsrrr sampling_typer RANDOM_SEEDr6 Generatorr. manual_seedseedis_pooling_modeltaskrr rlpoolerget_pooling_updatesapplyrrnrrcrrrr2npr- frombufferrereshaper5 from_numpysetattr Exceptionloggererrorr isinstancedictvllm_omni.enginerentriesitems tensor_data tensor_shapecopy list_dataprompt_logprobs vocab_size uses_mroperwuses_xdrope_dim_init_xdrope_positionsrhr is_last_rankscheduled_spec_decode_tokens_get_valid_sampled_token_countr~r new_block_idsnum_output_tokensrgprev_num_draft_lenuse_async_schedulingprev_req_id_to_indexrextend new_token_idslen num_tokensrnum_tokens_no_speczip all_token_idsr block_table append_row token_ids_cpuupdate_req_spec_token_ids add_requestcondense_may_reorder_batchrefresh_metadata)2r#rxrmm_hashscheduled_req_idscached_req_idsrunscheduled_req_ids reqs_to_add new_req_datarrrrrI to_updater]payloadr-arrpe_cpue payload_info info_dictrkentrydtrreq_datascheduled_spec_tokensvalid_sampled_token_countirrresumed_from_preemptionr req_indexprev_req_index num_accepted num_rejectedrnum_new_tokensend_idxrnew_idsresumed_token_idsstart_token_indexend_token_indexrequestr(r(r)_update_statessD                                                    z!OmniGPUModelRunner._update_states hidden_statescCst|jdr|jjrt|tr|j}|j}||fSt|tjr'|}i}||fSt|t s1t|t r;|d}i}||fSt dt |)Nhave_multimodal_outputsrzInvalid hidden states type: ) hasattrrIrrrtext_hidden_statesmultimodal_outputsr6Tensorlisttuple ValueErrortype)r#rrrr(r(r)extract_multimodal_outputss$  z-OmniGPUModelRunner.extract_multimodal_outputsFTrrcudagraph_runtime_modeforce_attentionuniform_decodeallow_microbatching skip_eplb is_profilecreate_mixed_batch remove_lorais_graph_capturingnum_active_loras activate_lorac 3 Cs| dur| dk} |jjj} | r| jrtgtgfS|dus&|s&J|r+|jn|}||jj ks5J|jj }|r[|r?Jt |d|d}||}|d}dg||g}|}n8|r{|raJt |t ||}|g|}||dkrz|||d<nt ||}||}|g|}|d||7<t ||ksJt||ksJtj|tjd}t| }tj|tjd}|j||||d||p|tjk|| | d \}}}}}|dur|}n||ksJd |d |d |j}|jdur|jn|}t|||||jjj\}} td || d}!|j|||| d \}"}#|s|tjkrq|r-dg||dg}$n|}$|$|j jd|<d|j j|d<|j !|"|\}%}|%|j#jd|d<|j#!|tjk}&|j$||||&ri| n|| |"d\}!}|%|j&||| | ||j'ksJ|(}'|j)r|jj*s|+|\}(})i|'|,|}'n9|j-rd}(|j.j/d|})|(}'n&t0t0|ddddr|j1j/d|}(|j.j/d|})n |j1j/d|}(d})|j2r|j3j/ddd|f}*n|j4dkr|j5j/ddd|f}*n|j6j/d|}*t7j8rd}+n|j9dur$|j:j;|j'|jj<|j=d|_9|>|dd}+| dur@| dj}|dur@||dd<|?|(|)wt@|!|j||||| |#dYt0|j:dddurtA|j:dr|},|,|j'kru|jBj/jCd},|D|jBj/d|,|jEj/d|,|jFj/d|,|jGj/d|,}-d|jH_I|j:d|(|*|+|)d|'}-Wdn 1swYWdn 1swY|jJr|-\}.}n|-}.|K|.\}.}/|jLr|jLMrtN|jOtPsJ| r|tjQkp| o|tjko|jLjR }0|jHjSr | r d}0|jOjT||0| |#dWdn 1s"wY|U|s5|jVd|dtW|d}1tX|1jY|j=dd}2|.|.|2fS)a= Run a dummy forward pass to warm up/profile run or capture the CUDA graph for the model. Args: num_tokens: Number of tokens to run the dummy forward pass. cudagraph_runtime_mode: used to control the behavior. - if not set will determine the cudagraph mode based on using the self.cudagraph_dispatcher. - CUDAGraphMode.NONE: No cudagraph, for warm up and profile run - CUDAGraphMode.PIECEWISE: Piecewise cudagraph. - CUDAGraphMode.FULL: Full cudagraph, attention metadata is needed. force_attention: If True, always create attention metadata. Used to warm up attention backend when mode is NONE. uniform_decode: If True, the batch is a uniform decode batch. skip_eplb: If True, skip EPLB state update. is_profile: If True, this is a profile run. create_mixed_batch: If True, create a mixed batch with both decode (1 token) and prefill (multiple tokens) requests. remove_lora: If False, dummy LoRAs are not destroyed after the run num_active_loras: Number of active LoRAs to capture for. activate_lora: Backward-compatible override for LoRA activation. Nrr,rrFF) rnum_reqsnum_scheduled_tokens_npmax_num_scheduled_tokensuse_cascade_attnr force_eagerforce_uniform_decodeforce_has_loraforce_num_active_lorasz7Cudagraph runtime mode mismatch in dummy_run. Expected z , but got .z+ubatch_slices: %s, ubatch_slices_padded: %s)num_tokens_paddednum_reqs_paddednum_tokens_unpadded ubatch_slices)rr max_query_lenr%for_cudagraph_capture slot_mappingsrIhas_preprocess) batch_sizer-r.)rnum_tokens_across_dpr batch_descriptorr% slot_mappingrBrA input_ids positionsintermediate_tensors inputs_embeds)use_cudagraphsrr(T)is_dummyr) non_blockingr()ZrMrPmultimodal_configmm_encoder_onlyr6tensorvalid_runtime_modesuniform_decode_query_lenr3max_num_batched_tokensr4minrsumrrarrayrVrOones&_determine_batch_execution_and_paddingrNONErrrparallel_config num_ubatchesrdebug_get_slot_mappingsrNseq_lens copy_to_gpu_get_cumsum_and_arangequery_start_loc_build_attention_metadatamaybe_dummy_run_with_lora lora_configmax_num_tokens_init_model_kwargssupports_mm_inputsis_encoder_decoder_prepare_mm_inputs_dummy_mm_kwargsenable_prompt_embedsr2gpur2r/rrorxdrope_positionsr0r is_first_rankr1rImake_empty_intermediate_tensorsr-r.#sync_and_slice_intermediate_tensorsmaybe_randomize_inputsrrrWr5rArXrYrZrJ cache_diruse_aux_hidden_state_outputsr speculative_config use_eaglerdrafterr PIECEWISE enforce_eagercudagraph_specialize_lora dummy_run_register_layerwise_nvtx_hooks eplb_stepcumsumrto)3r#rr rrrrrrrrrr mm_configr&rSnum_decode_tokensnum_prefill_tokensrnum_scheduled_tokens_listmin_tokens_per_reqrr$num_sampled_tokens_cudagraph_mode batch_desc should_ubatchr+_r"r#r%ubatch_slices_padded attn_metadataslot_mappings_by_groupr(rFcum_num_tokenspad_attn model_kwargsr/r2r0r1num_tokens_padded_talker_mtpoutputsrrr3 logit_indiceslogit_indices_devicer(r(r) _dummy_runs(                            "      zOmniGPUModelRunner._dummy_runc Cszt|dg}|s WdS|D]}t|ddpt|dd}|dur!qt|dd}d}|durmt|tjr=|d}n0t|dd}t|dd}|durm|durmtt|d d } tj || d } | |} t | }|dur||j vrt|j |d |t|d d} | duri} t| tr| } nIt| dd} t| tr| D]9\}}t|dd}|durtt|dd } tj || d } | t|dd} t | | |<qt|dd| |<q| r||j vrt|j |d| qWdSty }ztd|WYd}~dSd}~ww)zDecode per-request prompt_embeds and additional_information for newly scheduled requests and store them to CPU in the request state. This version avoids hard dependency on payload classes by duck-typing.rNr request_idrrrer5r-rrFrrrrrrr(rrz7Error decoding prompt_embeds / additional_information: )r2rr6rdetachrf contiguousrr-rrrrrrrrrrr)r#rxnew_reqsnrr payload_perrer5rrrrrrrrrr(r(r)"_decode_and_store_request_payloadsLsZ           +z5OmniGPUModelRunner._decode_and_store_request_payloadscCsg}|jjD]?}|j|}|durt|ddnd}|r@t|tr@||d|vr?|d}t|dr?t d|d|j q|iq|S)zQGather per-request additional_information stored in request state in batch order.Nrthinker_reply_part_per_requestr5z [OMNI] req=z1 has thinker_reply_part_per_request queue shape: ) rrrrgr2rrrhrrrDr5)r#per_req_runtime_inforr]infoqr(r(r)&_gather_runtime_additional_informations     z9OmniGPUModelRunner._gather_runtime_additional_informationcCsLg}tt|jjD]}t|jj|}t||}||||fq |S)zUCompute (start, end) token spans for each request within the flattened step sequence.)rangerrrrOrIrrh)r#rreq_token_spansr start_offset sched_tokensr(r(r)_compute_request_token_spanss  z/OmniGPUModelRunner._compute_request_token_spansc Cs^i}z ||d<W|Sty.}ztd|ddl}|WYd}~|Sd}~ww)zBuild extra keyword arguments passed to the model for this step, including: - runtime_additional_information: per-request additional information stored in request state runtime_additional_informationz0[OMNI DEBUG] Error building model_kwargs_extra: rN)rrrr traceback print_exc)r#model_kwargs_extrarrr(r(r)_build_model_kwargs_extrasz,OmniGPUModelRunner._build_model_kwargs_extrarrc Csz[t|jdrS|jjrVt|jjD]G\}}|j|}|dur%t|ddnd}t |j j |} t ||} | | | } } || | } |jj | fi|}| ||qWdSWdSWdSty} ztd|jjd| d|ddl}|WYd} ~ dSd} ~ ww)z_Process model-provided per-request additional_information updates and merge into request state.has_postprocessNrzError merging for requests:z additional information update: z!, with the multimodal_outputs as r)rrIrr~rrrrgr2rOrIr postprocess$_merge_additional_information_updaterrrrr)r#rrrrxrrr] req_infosrrsrhidden_states_slice update_dictrr(r(r)'_process_additional_information_updatess4      z:OmniGPUModelRunner._process_additional_information_updatesc Cst|jjD]_\}}|j|}t|dd}t|jj|}t|j}t d||}t||} t | |} | dkr9q| dkre|dure|||| j |j |j dd} t|jj|} |j| | | | qdS)zOverlay per-request prompt_embeds for the prefill portion and collect additional_information slices for this step. Returns a map req_id -> dict.rNrT)r-r.r5)r~rrrr2rOrrrnrRr<rfr-r.rIrr2copy_) r#rrrr]rr prompt_lenprompt_remainingr overlay_lensrcrr(r(r)+_collect_additional_information_for_prefills$      z>OmniGPUModelRunner._collect_additional_information_for_prefillr|rcCsL|j|}|dur dSt|dd}t|tr"|dur$||dSdSdS)zBUpdate per-request additional_information stored in request state.Nr)rrgr2rrupdate)r#r|rr]rr(r(r)_update_request_informations  z.OmniGPUModelRunner._update_request_informationcCs~|jD]}t|dd}t|tr||j|qt|jdr9t|jdi}t|tr;|D]\}}|||q.dSdSdS)Nr) rr2rrrrrrr)r#rxnew_reqr cached_infosrrr(r(r)_update_additional_informations     z1OmniGPUModelRunner._update_additional_informationrrcCs^|dus |jjjdkr |St|trt|ni}t|dd}|r)|ds)||d<||d<|S)zAttach MiMoAudio-specific fields into req_infos if applicable. This helper is intentionally small and self-contained so that it can be unit-tested to prevent regressions when updating MiMoAudio handling. N!MiMoAudioForConditionalGenerationrcr)rIr'__name__rrr2rg)r#r]rrrcr(r(r)"_maybe_attach_mimo_audio_req_infoss  z5OmniGPUModelRunner._maybe_attach_mimo_audio_req_infosnum_input_tokensr1c'sj}tj}|jj}d}|jre|re|se|j|jd}|| \}} Wdn1s3wY|j j |j j d||| d} |jj d|| ||\} } i||} nl|jr|r|jj d|jddd}|dkr|j j |}|j j |d}||jj |<|jj d|} |} |j j d|} n*t|j d dr|j j d|} |jj d|} |} n|j j d|} d} |} |jr|jj ddd|f}n|jdkr|jj ddd|f}n|jj d|}|rd}n|dusJ|||d }|r"jr"|}| d |i|j!j"}t#j$fd d |Dt#j%d}||_&| durB|'|t(|j d rY|j j)rYg}|j*jj+r[|,t-|j!j"D]\}}|j./|}|durvt|ddnd}|0|||}t1|j2j3|}t1||}|||}}t1|t1|}| dur| ||nd}|j j4d| |||d|\} }!}"| durt5j6| j7d|!j7df|!j8|!j9d} t(|j dr|dkr|":d\}#}$t;t<|t<|d}%|j=j |%| |j>j |%|!|j?j |%|#|j@j |%|$|A||B||"tC||!j7d}&|!d|&| |||&<tD| t5jErJ| |&krJ| | |||&<qat(|j drY|F|| | | ||| |fS)zIAlign with v0.14.0 preprocess and omni's additional information handling.N)r)multimodal_embeddings is_multimodalF)as_tupler,r)r/r)Tencoder_outputscsg|]}j|qSr()r).0ridrxr(r) wsz2OmniGPUModelRunner._preprocess..rFr)r/ input_embedsr)r.r-rA mtp_inputsr()Gtotal_num_scheduled_tokensrrVrPrPrOmaybe_get_ec_connector_outputr_execute_mm_encoder_gather_mm_embeddingsrIembed_input_idsr/rTr2rrQrN_extract_mm_kwargsrS is_token_idsnonzerosqueezenumelr2rrorrUr0rXscheduled_encoder_inputsrrrrr>rVr!rrr)rM async_chunkrr~rrgrrOrIr preprocessr6emptyr5r.r-rslicerrWrXrYrZrhrr<rr_talker_mtp_forward)'r#rxrr1rrVrPec_connector_output mm_embeds is_mm_embedinputs_embeds_scheduledr/r2rv token_ids_idx token_idstokens_to_embedsr0rrrdecode_req_idsrrr]rrrrrspan_len embed_slice req_input_ids req_embedsrrYrZ decode_sliceseg_lenr(rr) _preprocesss                    zOmniGPUModelRunner._preprocessrr2cCslt|}|dkr dS|j||tj|tjdddd\}}}}}t|jts(tj }|j }|j j d|}|j j d|} |jj d|} |jj d|} td|j||d||| | | \} } Wdn1siwY| d} t|jdd }t|D]1\}}|jj|}t|jj|}| ||d|||d<|| ||di}|||qdS) NrrFr,F)rrrrr)r r,rtalker_mtp_output_keycode_predictor_codes)rr@rr?rVrrArrrArrWrTrXrYrZrrMr}rfr~r2rIr~rrrrOrIrr)r#rr2decode_batch_sizermrnrpr"rrrYrZrcode_predictor_codes_cpuout_keyidxrrrrr(r(r)rs>  z&OmniGPUModelRunner._talker_mtp_forwardr/r0rvc s\|}tjd||||d||}t|ts)t|jdr)|jj|fi|}||_|S)z?Inject omni-specific kwargs into forward and cache model outputr.make_omni_outputNr() rr_model_forwardrrrrIrr")r#r/r0r1r2rvr model_outputr&r(r)rs z!OmniGPUModelRunner._model_forwardupdcCst|tsdS|j|}|durdSt|di}t|ts i}t|}|D])\}}t|tjr>| d ||<q(t|t rMdd|D||<q(|||<q(t |d|dS)NrrcSs.g|]}t|tjr|dn|qS)r)rr6rr}rfr~)rrkr(r(r)rs"zKOmniGPUModelRunner._merge_additional_information_update..) rrrrgr2rr6rr}rfr~rr)r#rrr]existingmergedrvr(r(r)rs$       z7OmniGPUModelRunner._merge_additional_information_update)r@N)rxr)rxrr@N) NFFTFFFTFrNr)NNNN),r __module__ __qualname__rr/rrHrrwrzr{rr6inference_moderrrrr rOrrjr r{rrrrobjectrndarrayrstrrrrrrrrrrr __classcell__r(r(r&r)r%s  <  ,$       I6  "      1# "r)3typingrrrrGrr6vllm.compilation.cuda_graphr vllm.configrvllm.distributed.parallel_statervllm.forward_contextr vllm.loggerr +vllm.model_executor.layers.rotary_embeddingr %vllm.model_executor.models.interfacesr *vllm.model_executor.models.interfaces_baser vllm.sampling_paramsr vllm.tracingrvllm.utils.import_utilsrvllm.utils.math_utilsrvllm.v1.spec_decode.eaglervllm.v1.worker.gpu_input_batchrvllm.v1.worker.gpu_model_runnerrrrvllm.v1.worker.ubatch_utilsr0vllm_omni.model_executor.models.output_templatesrvllm.v1.core.sched.outputrglobalsrrrrrr(r(r(r)s<