o پi@s>dZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ddlmZmZmZmZmZddlZddlmZddlmZddlmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(dd l)m*Z*m+Z+dd l,m-Z-m.Z.m/Z/dd l0m1Z1dd l2m3Z3dd l4m5Z5ddl6m7Z7m8Z8m9Z9m:Z:m;Z;mZ>ddl?m@Z@ddlAmBZBddlCmDZDddlEmFZFddlGmHZHddlImJZJmKZKmLZLmMZMddlNmOZOmPZPmQZQmRZRddlSmTZTddlUmVZVddlWmXZXddlYmZZZm[Z[ddl\m]Z]ddl^m_Z_ddl`maZambZbmcZcmdZdmeZeddlfmgZgddlhmiZimjZjmkZkddllmmZmdd lnmoZodd!lpmqZqdd"lrmsZsdd#ltmuZudd$lvmwZwdd%lxmyZydd&lzm{Z{dd'l|m}Z}dd(l~mZdd)lmZdd*lmZmZdd+lmZmZmZmZdd,lmZdd-lmZdd.lmZdd/lmZdd0lmZmZdd1lmZmZmZdd2lmZdd3lmZdd4lmZdd5lmZmZmZdd6lmZdd7lmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZdd8lmZdd9lmZmZmZdd:lmZmZdd;lmZddlmZeσgd?Zgd@ZejdAejdAejdBejdCiZdDdEZdFdGZdHZeeۡZdIejdJejfdKdLZGdMdNdNej߃Ze GdOdPdPZGdQdRdReZdSeeeejffdTdUZdVdWZe GdXdYdYZdS)Z2ModelRunner runs the forward passes of the models.N) defaultdict) dataclass)CallableListOptionalTupleUnion)nn) BailingHybridConfigFalconH1ConfigGraniteMoeHybridConfigJetNemotronConfig JetVLMConfigKimiLinearConfig Lfm2Config Lfm2MoeConfigNemotronH_Nano_VL_V2_ConfigNemotronHConfig Qwen3_5ConfigQwen3_5MoeConfigQwen3NextConfig) DeviceConfig) LoadConfig LoadFormat) AttentionArch ModelConfig ModelImpl)#adjust_config_with_unaligned_cpu_tp)GPU_MEMORY_TYPE_WEIGHTS)register_forward_hook_for_model) get_pp_group get_tp_groupget_world_groupinit_distributed_environmentinitialize_model_parallelset_custom_all_reduceset_mscclpp_all_reduceset_torch_symm_mem_all_reduce)use_symmetric_memory) monkey_patch_vllm_parallel_state)ElasticEPStateManager)envs) EPLBManager)ExpertDistributionMetricsExpertDistributionRecorder'get_global_expert_distribution_recorder'set_global_expert_distribution_recorder)ExpertLocationMetadata(compute_initial_expert_location_metadata#get_global_expert_location_metadata#set_global_expert_location_metadata)ExpertLocationUpdater)NPUGraphRunner)deep_gemm_wrapper)ATTENTION_BACKENDSattn_backend_wrapper)is_nsa_enable_prefill_cp)TboAttnBackend) DpPaddingModeget_attention_tp_groupinitialize_dp_attentionset_dp_buffer_lenset_is_extend_in_batch)LogitsProcessorOutput)RoutedExpertsCapturerget_global_experts_capturerset_global_experts_capturer)get_moe_a2a_backend)EmbeddingPoolerOutput) fp8_dtype)create_sampler)apply_torchao_config_to_model) LoRAManager)LoRARef)sanity_check_mm_pad_shift_value)BaseTokenToKVPoolAllocator)ReqToTokenPool)CPUGraphRunner)CudaGraphRunnerset_torch_compile_config)CaptureHiddenMode ForwardBatch ForwardModePPProxyTensors)register_forward_hooks)GraphInputBuffers)ModelRunnerKVCacheMixin)PiecewiseCudaGraphRunner)DefaultModelLoaderget_model_loader)!RemoteInstanceWeightLoaderBackendregister_memory_region;trigger_init_weights_send_group_for_remote_instance_request)set_default_torch_dtype)default_weight_loader)SamplingBatchInfo) ServerArgsget_global_server_args$set_global_server_args_for_scheduler)SpeculativeAlgorithm)MultiprocessingSerializercpu_has_amx_supportdynamic_import empty_contextenable_show_time_costget_available_gpu_memoryget_cpu_ids_by_nodeget_local_ip_autoinit_custom_process_groupis_hipis_host_cpu_arm64is_npulog_info_on_rank0monkey_patch_p2p_access_checkrequire_attn_tp_gatherrequire_gathered_bufferrequire_mlp_tp_gather%reserve_rope_cache_for_long_sequences set_cuda_archslow_rank_detector)PytHooks)!create_offloader_from_server_args get_offloader set_offloader)monkey_patch_torch_reductionsregister_sgl_tp_rank)TorchMemorySaverAdapter) WeightChecker)FlattenedTensorBucketFlattenedTensorMetadata)init_npu_backend) aiter flashinferfa3fa4tritonflashmla cutlass_mla trtllm_mlaascendnsa)rrrrrrfp8_e4m3fp8_e5m2bf16cC,|tvrt|td|ddSdS)NAdded z to MLA_ATTENTION_BACKENDS.)MLA_ATTENTION_BACKENDSappendloggerinfo backend_namerZ/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.pyadd_mla_attention_backends rcCr)Nrz6 to CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDS.)1CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDSrrrrrrr*add_chunked_prefix_cache_attention_backends   rimodelreturncCs|jj}|dkr |jjS|jS)N$Qwen3OmniMoeForConditionalGeneration) __class____name__thinkerr)rmodel_cls_namerrrresolve_language_modelsrcs(eZdZdZfddZddZZS)RankZeroFilterz_Filter that only allows INFO level logs from rank 0, but allows all other levels from any rank.cst||_dSN)super__init__ is_rank_zero)selfrrrrrs  zRankZeroFilter.__init__cCs|jtjkr |jSdS)NT)levelnologgingINFOr)rrecordrrrfilter s zRankZeroFilter.filter)r __module__ __qualname____doc__rr __classcell__rrrrrs rc@s6eZdZUeeefed<eed<dZe e ed<dS)ModelRunnerOutput logits_output can_run_graphNexpert_distribution_metrics) rrrr rBrV__annotations__boolrrr.rrrrrs rc&@seZdZdZ       ddededededed ed ed ed ed ededeedeedeede dee dee deef$ddZ ddZ defddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)d*Zd+ed,eefd-d.Z  dd/ed0ed1eeege fd2e d3ee eff d4d5Z 6dd7d8Zd9d:Z 6dd;d<Zd=d>Z dd0eefd?d@Z!dAdBZ" ddCee#ee$e%j&dDffd0eefdEdFZ'dGdHZ( IddJedKed3ee%j&fdLdMZ)dNdOZ*dPe+fdQdRZ, ddPe+fdSdTZ-dPe+fdUdVZ.e/dWdXZ0e/dYdZZ1e/d[d\Z2e/d]d^Z3e/d_d`Z4e/dadbZ5e/dcddZ6dedfZ7dgdhZ8didjZ9dkdlZ:ddme fdndoZ; ddpedme fdqdrZd3e fdwdxZ?dydzZ@dd{efd|d}ZAd~dZBddZCddZDddZEdefddZF  ddeGde d3e$eHeIffddZJ  ddeGde d3e#e$eHeIeKfe ffddZL ddeGd3e$eHeIffddZM  ddeGde ded3eHfddZN    ddeGde deeIde ded3eOf ddZP  ddeGde deeIde ded3eOf ddZQdeHdeRfddZSdeHdeGd3e%j&fddZTdeHdeGd3dfddZUe/d3e fddZVdefddZW ddedeedeefddZXdefddZYddZZddZ[dS) ModelRunnerrNF model_configmem_fraction_staticgpu_idtp_ranktp_size moe_ep_rank moe_ep_sizepp_rankpp_size nccl_port server_argsdp_rank attn_cp_rank moe_dp_rankis_draft_workerreq_to_token_pooltoken_to_kv_pool_allocatordraft_model_idxcCs||_| j|_||_||_||_||_||_| jr| jnd|_||_ | |_ | |_ | j |_ ||_ | j|_||_| |_| |_||_|j|_|j|_|j|_t| j|_| j|_||_||_|j|_|j|_|jjt j!k|_"|j#|_#d|_$d|_%||_&d|_'d|_(d|_)d|_*|j+r|jst,j-| | j.| j/dd}d|_*zt0|j1dd}|2dd|_*|d |_3Wnd|_3Y| j4rt5|6t7| t8}|j"|_"|jd kr|9|:|;}t<=|j>|_?t@tA| | d tB|d |_CtDjE2rtFG|HtIjJr tIK|| |L||M|jrtN|jjOd tPQ|jRjSjTv|_U|j dkr8|jUs8Jdi|_Vi|_WdS)NrFT) model_pathmodel_revisionis_draft_model eagle_configuse_aux_hidden_state eagle_aux_hidden_state_layer_idscpu)r) model_runnerpp_proxy_tensorsz4Pipeline Parallel is not compatible with this model.)Xrdevicerrrrrenable_dp_attentiondp_sizerrr attn_cp_sizer moe_dp_sizer dist_portrr is_generation is_multimodal'is_multimodal_chunked_prefill_supportedrf from_stringspeculative_algorithmspec_algorithm page_sizerr is_hybrid_swais_hybrid_swa_compressattention_archrMLAuse_mla_backendattention_chunk_sizeforward_pass_idinit_new_workspacerremote_instance_transfer_engine*remote_instance_transfer_engine_session_id+remote_instance_transfer_engine_weight_infoeagle_use_aux_hidden_state is_eagle3rfrom_server_argsspeculative_draft_model_path speculative_draft_model_revisiongetattr hf_configgetrshow_time_costrkmodel_specific_adjustmentrerdinit_threads_binding$init_shared_mooncake_transfer_engineinit_torch_distributedtorchget_device_moduleStreamforward_streamr~r|r_weight_checkerr,SGLANG_DETECT_SLOW_RANKrzexecuteinit_mindspore_runnerr8ENABLE_JIT_DEEPGEMMupdate_deep_gemm_config initialize!check_quantized_moe_compatibilityrM vocab_sizeinspect signaturerforward parameters support_pp_model_update_group_weights_send_group)rrrrrrrrrrrrrrrrrrrdraft_model_configrglobal_server_argsmin_per_gpu_memoryrrrrs         zModelRunner.__init__cCs\|jjtjkr*tr,ddlm}||j|j |j|j |j |j |j|j ddSdSdS)Nr)init_ms_distributed) world_sizerank local_rankrport)r model_impllowerr MINDSPORE_is_npu*sglang.srt.model_executor.mindspore_runnerrrrrrrr)rrrrrrs   z!ModelRunner.init_mindspore_runnerrc sj}tjjjd_jrjsBtt |j j dj dkr6t jr6tdtttj|tj djjrMjsMtnd_t_jjr^tjndt_ jrj!durj"durt#j$j!_"j j%du}jr|rj j%nt&j j'j j(}j j)j*ddkrd}n j j)j*ddkrd}t+j$d d_,t+j$d |_-j-j,_.t+j j)d d}|dkr׈j.|_.|rj/0sj/0sj.|ksJd fd dt1j,j-dD}fddt1j,j-dD}|j _2|j _3t+j$dd}|s&t4j$t5j6t+j$dd} j7dkr:| r:8|j9rB:|j;rV|j|j<|j?rcddl@mA} | BC|tD|jEdurxjFdn|jEjGjHjI_EJjKdksjKdkrLMNOnjKdvrMOn d_Pd_QM|jRrtSj$|jRjTrӈj$UjVWXdS)N)enable)rrrrz"Initial expert_location_metadata: )r MiMoV2MTPr Step3p5MTP start_layer end_layerloop_numz%PP is not compatible with MTP models.c(g|]}tjdr|jjvr|qS)full_attention_layer_ids)hasattrrr).0 layer_idxrrr   z*ModelRunner.initialize..cr()swa_attention_layer_ids)r*rr1r+r.rrr/!r0torchao_appliedFsupports_torch_tpzGPlease specify the heavy channel type for double sparsity optimization.)enable_batch_invariant_modecudamusa)npur)Yrrcreateenable_memory_savermemory_saver_adapter1remote_instance_weight_loader_use_transfer_engine$remote_instance_init_transfer_enginerr5r3rrrr,#SGLANG_LOG_EXPERT_LOCATION_METADATArrrr4r1r/init_new enable_eplbr- eplb_managerr6expert_location_updaterelastic_ep_backendr+initrIsampler load_modelrrr^rnum_nextn_predict_layersmaxnum_hidden_layersnum_attention_layersr architecturesrr%r&num_effective_layersris_noneranger1r)rJrdtorchao_configrapply_torch_tp enable_lorainit_lora_managerenable_double_sparsityds_heavy_channel_type ValueError#init_double_sparsity_channel_configenable_deterministic_inferencesglang.srt.batch_invariant_opsr4configure_kv_cache_dtypeinit_memory_poolminmax_running_requestsmax_total_num_tokensrrsizeinit_routed_experts_capturerr init_cublasinit_attention_backend kernel_warmupinit_device_graphs graph_runnergraph_mem_usage forward_hooksrWrset_eagle3_layers_to_capturerinit_piecewise_cuda_graphsprealloc_symmetric_memory_pool) rrrmodel_has_mtp_layersmodel_num_layersr'r)r1r2r3r4rr.rr s                        zModelRunner.initializec CsR|jjst|jdr|jj}nd}ttjtj |j ||j |j |j |jddS)Nnum_fused_shared_expertsr)r"rrl num_tokensr\r)rdisable_shared_experts_fusionr*rrlrErCr9rdenable_return_routed_expertsrr]rr\r)rrlrrrr_ws   z(ModelRunner.init_routed_experts_capturerc Cszddlm}Wnty }z tdWYd}~dSd}~ww||_t}|j|ddtj |d|j |_ dS)Nr)TransferEnginezWPlease install mooncake for using remote instance transfer engine: pip install mooncake P2PHANDSHAKErdma:) mooncake.enginerp ImportErrorrwarningrrnr r,MOONCAKE_DEVICEr get_rpc_portr)rrpelocal_iprrrr=s z0ModelRunner.remote_instance_init_transfer_enginecCsx|j}|jrtdd|_d|_|jr%|js%d|_td|j j j |j r-|jt vr0d|_|js:ttddSdS)NzQDouble sparsity optimization is turned on. Use triton backend without CUDA graph.rTzIAutomatically turn off --chunked-prefill-size as it is not supported for z"Chunked prefix cache is turned on.)rrSrrattention_backenddisable_cuda_graphrrchunked_prefill_sizerr model_typerrdisable_chunked_prefix_cachers)rrrrrrs0z%ModelRunner.model_specific_adjustmentc Cst|jjdd}dury|dd}dur{|d}|j|jdkr.td|jd|j|j|j}t|jjdd}|durBdS||dkr[td|d|d |jd |jd |||dkr}td |d |d|d|jd|jd dSdSdS)Nquantization_configweight_block_sizerztp_size z must be divisible by ep_size moe_intermediate_sizezmoe_intermediate_size z# must be divisible by moe_tp_size (z) which is tp_size (z) divided by moe_ep_size (z).zBFor quantized MoE models, please make sure (moe_intermediate_size=z / moe_tp_size=z) % weight_block_size_n=z- == 0 where moe_tp_size is equal to tp_size (z) divided by ep_size (zE). You can fix this by setting arguments `--tp` and `--ep` correctly.)rrrrrrrUhf_text_config)rrrweight_block_size_n moe_tp_sizerrrrr sD     z-ModelRunner.check_quantized_moe_compatibilityc s\t}tdz tjjWn$t y9t djdjdt j ddjdj wjdkrijjd krfd }jjrejjd }z d d lm}||Wn0Yn+n*d }n'jdkrqd}njdkryd}njdkrd}njdkrd}njdkrd}tjj}jjsttj }|r|}njjrdjj}ndj}tjj t jj!t"jj#j$sDjdkrt%st&rtj'j()j*t+jt j d<tj'j(,jjtj-.dfdd}nt dt/|jj0jj1jj|jj2dt3jj4j0j5j6j7jj8dt9jj:d t;rDt<jtjjt=j>d!kt=j?d"} t@_AtB_CtD_Etjj} jd!krj$s| | d#krd$} | d%| d&| d'| d#7} tjF rtG| t | td(t|d)d*|| d)d+| S),NzInit torch distributed begin.zContext: self.device=z self.gpu_id=z( os.environ.get('CUDA_VISIBLE_DEVICES')=CUDA_VISIBLE_DEVICESz self.tp_rank=z self.tp_size=r6mooncake,r)epncclxpuxcclhpuhcclrgloor8r7mccltcp://ztcp://127.0.0.1: LOCAL_SIZEzsgl_kernel::shm_allgathercstj|gj|dS)N)dim)rcatr)datarr.rr_sz-ModelRunner.init_torch_distributed.._zrinit_cpu_threads_env and shared memory based AllReduce is disabled, only intel amx backend and arm64 are supported)backendrrrdistributed_init_methodtimeout)tensor_model_parallel_sizeattention_data_parallel_sizepipeline_model_parallel_sizeexpert_model_parallel_size%attention_context_model_parallel_sizemoe_data_model_parallel_sizeduplicate_tp_group)rrr) distributed cpu_groupg?zQThe memory capacity is unbalanced. Some GPUs may be occupied by other processes. zmin_per_gpu_memory=z, local_gpu_memory=z, local_gpu_memory * 0.9=z%Init torch distributed ends. elapsed=.2fz s, mem usage= GB)Htime perf_counterrrrrr set_devicer ExceptionrvosenvironrrrrrCmooncake_ib_devicesplitrrset_device_filterrlenable_p2p_checkrtr,'SGLANG_DISTRIBUTED_INIT_METHOD_OVERRIDEdist_init_addrrr&disable_custom_all_reducer'enable_mscclppr(enable_torch_symm_memr_is_cpu_amx_available _is_cpu_arm64ops sgl_kernelinit_cpu_threads_envlocal_omp_cpuidstrr library register_faker$rr dist_timeoutr%rrrr enable_pdmuxr?rrrrr#rrr"tp_groupr!pp_groupr>attention_tp_group'SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK RuntimeError) rticrr mooncake_epbefore_avail_memorydist_init_method_overridedist_init_methodrrlocal_gpu_memorymsgrr.rrs  0                     z"ModelRunner.init_torch_distributedcCs|jjdko |jjdkp)|jjo|jjdkp)|jjo|jjdkp)|jjo)|jjdk}|rCddlm }|t |j |jj p>|jj ddSdS)z Need MooncakeTransferEngine when: 1) PD disaggregation uses mooncake for KV transfer (prefill/decode) 2) HiCache uses mooncake storage backend 3) Encoder disaggregation uses mooncake nullrr)init_mooncake_transfer_engine)hostnamer ib_deviceN)rdisaggregation_modedisaggregation_transfer_backendenable_hierarchical_cachehicache_storage_backend encoder_onlyencoder_transfer_backend language_onlyDsglang.srt.distributed.device_communicators.mooncake_transfer_enginerrnrdisaggregation_ib_devicer)ruse_mooncake_terrrrrWs*      z0ModelRunner.init_shared_mooncake_transfer_enginec CsRt}t|j|j}tdt|j|jdd|jdkr$td|jdkrMtj ddkrMtd d |j _ tj |j_ tj dd krMtd tdd lm}||j j|j j|j j|j j|j jd}t|j j|j j|j j|j|j j|j j|j j|j j |j!||j j"|j#d |_$|jdkrt%|j|j$|j&|_|j jt'j(kr|j j t)j*kr|jdkrt+,t+-}t.j/t0|j j|j j|j j|fd}|1t2|j j3p|j4o|j j5}|j6j7t8|d,t9|j$|jd|_:|j:j;|jt<|j|jd|_=t>|j:dr|j:j?|_?Wdn 1swYt2ddt@A|j jBr2tC|_D|jDjE|j=dd|j jFdkrg|j jGdurbtHtI|j=ddr[|j=J|j jGtd|j jGn td|j=jKtLdd|_Mt>|j=drx|j=N|_Mn'|jjOr|jjMdur|jjM|_Mn|jjPdur|jjP|_Mtd|jM|jj |_ t|j|j}|||_Qtd t|dd!tR|j=jSd"|j d#|dd$|jQdd% |j jTdurtU|j=|j jT|j jV|j&|j|jWtX|j=|j |jt|j jYd&krtZj[t\j]d'dSztZj^t\j]t_j`tad(dd)WdSty(tbd*|jd+dw),NzLoad weight begin. avail mem=rrrrr6rzKCompute capability below sm80. Use float16 due to lack of bfloat16 support.float16z$SGLang only supports sm75 and above.)ModelOptConfig)quantcheckpoint_restore_pathcheckpoint_save_path export_pathquantize_and_serve) load_format download_dirmodel_loader_extra_configr.remote_instance_weight_loader_seed_instance_ip8remote_instance_weight_loader_seed_instance_service_port6remote_instance_weight_loader_send_weights_group_ports%remote_instance_weight_loader_backend-remote_instance_weight_loader_transfer_enginemodelopt_configrl_quant_profiler)targetargs)enable_cpu_backup) load_configr)r device_configrT)reverser) module_prefixrload_kv_cache_scalesz'Loaded KV cache scaling factors from %szfUsing FP8 KV cache and scaling factors provided but model %s does not support loading scaling factors.zUsing FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0. This may lead to less accurate results!!get_attention_sliding_window_sizez8Setting sliding_window_size to be attention_chunk_size: zLoad weight end. elapsed=z s, type=z, dtype=z , avail mem=z GB, mem usage= GB.rgroup)seconds)rrwait_all_rankszTP rank z could finish the model loading, but there are other ranks that didn't finish loading. It is likely due to unexpected failures (e.g., OOM) or a slow node.)crrrlrrrrrset_num_threadsr6get_device_capabilityrdtyperrrry"sglang.srt.configs.modelopt_configrmodelopt_quant modelopt_checkpoint_restore_pathmodelopt_checkpoint_save_pathmodelopt_export_pathrrrrrrrrrrrrrrrrrREMOTE_INSTANCEr]NCCLsocket gethostbyname gethostname threadingThreadr_startr*enable_weights_cpu_backuprenable_draft_weights_cpu_backupr;regionrr\loaderrFrrr*rr} post_initenable_layerwise_nvtx_markerr{ pyt_hooksregister_hookskv_cache_dtypequantization_param_pathcallablerrrrvsliding_window_sizerrrweight_load_mem_usagetyperdebug_tensor_dump_output_folderr debug_tensor_dump_layersrrxrCdistbarrierr"rmonitored_barrierdatetime timedelta"UNBALANCED_MODEL_LOADING_TIMEOUT_SrU) r tic_totalrrr instance_iptrafter_avail_memoryrrrrFsB                        zModelRunner.load_modelnew_expert_location_metadataupdate_layer_idscCsntdur%t}|dusJ|j||d||jj|jjdddS|jj|j j |||jj |j ddS)N)r&cSsd|vod|vS)Nz mlp.expertszmlp.shared_expertsr)namerrrLsz4ModelRunner.update_expert_location..)r&nnodesr) r+instancer4updateupdate_weights_from_diskrrrrBrrouted_experts_weights_of_layerr)r)rr%r&old_expert_location_metadatarrrupdate_expert_location<s&   z"ModelRunner.update_expert_locationrrweight_name_filterrecapture_cuda_graphrc stdtjjddtj|j_t|d}t |jt t s3dd}d|fSfdd }fd d }t jj pz|j} Wn#tys} zd | d}d|fWYd } ~ Wd Sd } ~ wwz|j| } Wn4ty} z(d| d}~ t|j} |j| _d|fWYd } ~ Wd Sd } ~ wwWd n1swY| _|j_|j_|_|rވjdksڈjdkrވtddS)z-Update engine weights in-place from the disk.z8Update engine weights online from disk begin. avail mem=rr)rzFailed to get model loader: .Fcs4tj|j}durfdd|D}|S)Nc3s$|] \}}|r||fVqdSrr)r,r'weight)r0rr ss   zPModelRunner.update_weights_from_disk..get_weight_iter..)_get_weights_iteratorr[Sourcer?r)configiter)rrr0rrget_weight_iterns z=ModelRunner.update_weights_from_disk..get_weight_itercs|||Sr)load_weights_and_postprocess)rr8)r target_devicerrmodel_load_weightsysz@ModelRunner.update_weights_from_disk..model_load_weightsz Failed to get weights iterator: NzFailed to update weights: z#. Rolling back to original weights.r6r7zUpdate weights end.)Tz"Succeeded to update model weights.)rrrlrrrrrrr\ isinstancer[r`rrrgccollectrrrrc) rrrr0r1rmessager9r<r8ryrr)rrr;r0rr,Ws\          z$ModelRunner.update_weights_from_diskrc Csptjs Jd|dksJd|d}t||jks+Jd|jdt|d||j}|d|d|j}td |jd |j d |d |d |d|d|d|tj d} d} z.t |d|d||||t d|j d|j|<tj|j|dd} d|d|d} Wnty} zd| d} t| WYd} ~ nd} ~ wwtj | | fS)N/Default torch process group must be initializedrGroup name cannot be emptyr Expected  ports, but got  ports.rz#init custom process group: tp_rank=z , gpu_id=z, master_address=, master_port=z , group_rank= , world_size= , group_name= , backend=Frrsr6)r init_methodrr group_name device_idrTz Succeeded to init group through z group.zFailed to init group: r2)rris_initializedrlenrrrrrr6 empty_cacherorrrrrerror) rmaster_addressports group_rankrrKr ports_list group_portsuccessr@ryrrr+init_weights_send_group_for_remote_instancesZ        z7ModelRunner.init_weights_send_group_for_remote_instancec Csjtjs Jd|dksJd|d}t||jks+Jd|jdt|d||j}|d|d|j}|j|durH|j|}nd |d }t |d |fStj d }d}z#|j D] \} } tjj| d |d qfd}d|d|d|d}Wnty} zd| d}t |WYd} ~ nd} ~ ww|j|=tjj|tj ||fS)NrArrBrrCrDrErGroup zb not in _weights_send_group list. Please call `init_weights_send_group_for_remote_instance` first.FrsrcrTz"Succeeded to send weights through rs r2zFailed to send weights: )rrrMrrNrrrrrPr6rOrnamed_parameters broadcastrdistributed_c10ddestroy_process_group) rrQrRrKrTrU send_groupr@rVrweightsryrrrsend_weights_to_remote_instancesJ        z+ModelRunner.send_weights_to_remote_instancec Cstjs Jd|dksJd||j}td|d|d|d|d|d |d |zt|d |d ||||d |j|<WdStyf}zd|d} t | d| fWYd}~Sd}~ww)aInitialize the Torch process group for model parameter updates. `_model_update_group` is used in the RLHF workflow, where rank 0 is the actor model in the training engine, and the other ranks are the inference engine, which is used for rollout. In the RLHF workflow, the training engine updates the model weights/parameters online, and broadcasts them to the inference engine through the `_model_update_group` process group. rArrBz*init custom process group: master_address=rFz, rank_offset=z, rank=rGrHrIrrs)rrJrrrK)Tz-Succeeded to initialize custom process group.z+Failed to initialize custom process group: r2FN) rrrMrrrrorrrP) rrQ master_port rank_offsetrrKrrryr@rrrinit_weights_update_groupsD    z%ModelRunner.init_weights_update_groupc Csrz||jvr|j|}tj|WdSWdSty8}zd|d}t|d|fWYd}~Sd}~ww)N)Tz*Succeeded to destroy custom process group.)Fz)The group to be destroyed does not exist.z(Failed to destroy custom process group: r2F)rpoprrr_rrrP)rrKpgryr@rrrdestroy_weights_update_group(s     z(ModelRunner.destroy_weights_update_groupc Cs$||jvsJd|dt|jd|dkr!|||||SzOg}g}t|||D]2\}} } t| tjr9| ntt| } tj | | |j d} | tj j | d|j|dd| || fq,|D]} | qa|j|Wd Sty}zd |d }t|d |fWYd }~Sd }~ww)aD Update specific parameter in the model weights online through `_model_update_group` process group. Args: name: the name of the parameter to be updated. dtype: the data type of the parameter to be updated. shape: the shape of the parameter to be updated. rXz not in z0. Please call `init_weights_update_group` first.flattened_bucketrrrT)rZrasync_opT%Succeeded to update parameter online.#Failed to update parameter online: ^. The full weights of the ModelRunner are partially updated. Please discard the whole weights.FN)rlistkeys)_update_bucketed_weights_from_distributedzipr=rrremptyrrrr]waitr load_weightsrrrP)rnamesdtypesshapesrKrrahandlesr'rshape target_dtyper3handlery error_msgrrrupdate_weights_from_distributed5sD     z+ModelRunner.update_weights_from_distributedc CszKg}t|||D] \}}}t|tjr|ntt|} ||tj|| |jdfq t|d} | } tj j | d|j |d| } |j| d WStyl} zd| d}t|d |fWYd} ~ Sd} ~ ww) Nrj) named_tensorsrrYTrmrnroFrl)rsr=rrrrrtrrget_flattened_tensorrr]rreconstruct_tensorsrrvrrrP)rrwrxryrKrr'rr{r|bucketflattened_tensorreconstructed_tensorsryr~rrrrrps2    z5ModelRunner._update_bucketed_weights_from_distributedrLocalSerializedTensorcst|dkr j|dStj_jfdd|D}|dkr/tj|dS|j j vrAt |}|j|dS|durMj |dSt d|)Nri)flattened_tensor_bucket_dictcs$g|]\}}|t|jdfqS))rr)_unwrap_tensorr)r,r'tensorinfered_devicerrrr/sz:ModelRunner.update_weights_from_tensor..directzUnknown load_format=TSuccess)r%_update_weights_from_flattened_bucketrrr device_modulecurrent_device_model_load_weights_directrrcustom_weight_loaderrirvNotImplementedError)rrr custom_loaderrrrupdate_weights_from_tensors*      z&ModelRunner.update_weights_from_tensorc Csl|d}|d}g}|D]}t|j|j|j|j|j|jd}||q t||d}| }|j |dS)z1Handle flattened bucket format for weight updatesrmetadata)r'r{r start_idxend_idxnumel)rrr) rr'r{rrrrrrrrrv) rrrrconverted_metadatametaconverted_metarrrrrrs&  z1ModelRunner._update_weights_from_flattened_bucketdr' truncate_sizec CsTz |jj|||jdWSty)}ztd|d|WYd}~dSd}~ww)zGet the weights of the parameter by its name. Similar to `get_parameter` in Hugging Face. Only used for unit test with an unoptimized performance. For optimized performance, please use torch.save and torch.load. )rzError when getting parameter z: N)rget_weights_by_namerrrrP)rr'rryrrrrs zModelRunner.get_weights_by_namecCsJt|j|jj|jj|j|j|j|jj|j |j |jj |jj |jj d |_dS)N) base_modelbase_hf_configmax_loras_per_batchrrr lora_backendrr max_lora_ranktarget_modules lora_paths)rKrrrrrrrrrrrlora_target_modulesr lora_managerr.rrrrRs zModelRunner.init_lora_managerlora_refcCXtd|dt|j|jdd|j|}td|dt|j|jdd|S)z1Load a new lora adapter from disk or huggingface.zLoRA adapter loading starts: . avail mem=rrz LoRA adapter loading completes: )rrrlrrrload_lora_adapterrrresultrrrr zModelRunner.load_lora_adaptercCs:td|d|j||||}td|d|S)Nz*LoRA adapter loading from tensors starts: r2z-LoRA adapter loading from tensors completes: )rrrload_lora_adapter_from_tensors)rrtensors config_dictadded_tokens_configrrrrrs z*ModelRunner.load_lora_adapter_from_tensorscCr)zZUnload a lora adapter that was previously loaded during initialization or dynamic loading.zLoRA adapter unloading starts: rrrz"LoRA adapter unloading completes: )rrrlrrrunload_lora_adapterrrrrr rzModelRunner.unload_lora_adaptercC|jj}t|tr |SdSr)rrr=rrr7rrrqwen3_next_config zModelRunner.qwen3_next_configcCrr)rrr=r rrrrhybrid_lightning_config"rz#ModelRunner.hybrid_lightning_configcCs.|jj}t|ttBtBtBtBr|SdSr) rrget_text_configr=rrrrrrrrrhybrid_gdn_config)s zModelRunner.hybrid_gdn_configcCs|jj}t|tr|jrt|dd}|durd|vrdSt|ttBtBtBr)|St|t r1|j St|t rIt ddt|dgD}|sGdS|SdS)Nmtp_hybrid_override_patternMcss|]}|dkVqdS)mambaNr)r, layer_typerrrr4Hs  z,ModelRunner.mamba2_config.. layer_types) rrr=rrrr rrr llm_configr any)rr7pattern has_mambarrr mamba2_config7s&    zModelRunner.mamba2_configcCs|jr t|j|jS|jS)z?Return the max token pool size considering hybrid swa settings.)rr[swa_max_total_num_tokensr]r.rrrmax_token_pool_sizeSszModelRunner.max_token_pool_sizecCrr)rrr=rrrrrkimi_linear_config[rzModelRunner.kimi_linear_configcCs|jp |jp |jp |jSr)rrrrr.rrrmambaish_configbszModelRunner.mambaish_configcCs^|jrdS|jjrttddS|jdkrttddSts&tr-ttddSdS)NFzYDisable piecewise CUDA graph because piecewise_cuda_graph has conflict with torch compilerzMDisable piecewise CUDA graph because piecewise_cuda_graph does not support PPz?Disable piecewise CUDA graph due to existing compilation errorsT) rrenable_torch_compilersrrrF is_deepep is_mooncaker.rrrcan_run_piecewise_cuda_graphks* z(ModelRunner.can_run_piecewise_cuda_graphcCs:|jjdkr?dj@dddjAd| d| d| d | d!|d"|d#j/d$j2d%tBCd&d'jDd(jd)|d*|d+j'd,|d-||durjEFjGfd.d/}t%HjIjJKt%L+|p tM |Wdn1swYWddSWddS1s4wYdS)1z.Run a dummy forward pass for warmup/profiling.rzThis should not happenrF)rmax_bs max_num_token hidden_sizer rrris_encoder_decoderrwseq_len_fill_valueencoder_len_fill_valuenum_tokens_per_bscache_loc_dtypeenable_mamba_track.rjNcsi|] \}}||dqSrrr,kv)rmrr sz*ModelRunner._dummy_run..csd}js jr4ddlm}jrtd|djdddddjj jj jj t j ddd }|SjrOddlm}|djddddd}t j|_|S)Nr)EagleVerifyInputzThis should not happen.) draft_token custom_mask positions retrive_indexretrive_next_tokenretrive_next_siblingretrive_cum_len spec_stepstopkdraft_token_numcapture_hidden_mode seq_lens_sum seq_lens_cpu)NgramVerifyInput)r& tree_maskr(r)r*r+r/)rr r !sglang.srt.speculative.eagle_infor%rrr'rspeculative_num_stepsspeculative_eagle_topkspeculative_num_draft_tokensrSFULLr !sglang.srt.speculative.ngram_infor3NULLr0) spec_infor%r3)buffersrrrr get_spec_infosD   z-ModelRunner._dummy_run..get_spec_info forward_moder input_idsreq_pool_indicesseq_lensr2next_token_logits_buffer orig_seq_lensrtoken_to_kv_poolr out_cache_locr1 encoder_lensreturn_logprobr(extend_num_tokensextend_seq_lensextend_prefix_lensextend_start_locextend_prefix_lens_cpuextend_seq_lens_cpuglobal_num_tokens_gpu!global_num_tokens_for_logprob_gpudp_padding_modeglobal_dp_buffer_lenmrope_positionsrr<r0num_token_non_paddedglobal_forward_modelora_idscsd__tjtdi}jjdkr3dt j j j vr3t ddjD|d<js:d|d<j j jjfi|}|S)NFrrcSsi|] \}}||qSr)cloner!rrrr$5sz.run_once..T get_embedding)dp_local_start_posdp_local_num_tokensr@rQ is_max_lenrArrrrrrrrVritemsrr@r()kwargs!logits_output_or_pp_proxy_tensors)r= forward_batchrRrmrrrrrun_once%s2  z(ModelRunner._dummy_run..run_oncer)NrrUDECODEEXTENDrSr;rr r r rr TARGET_VERIFYrr8enable_return_hidden_statesr9r!get_cuda_graph_seq_len_fill_valuerrRrrrgrwrvrurXr9rrrr rrrrrint64rTfullint32zerosarangerVrr\rOcopy_rrPr0rQrTr@rArBr2rCrrErFsumitemrGr(r=get_default_mode_in_cuda_graphrSrprepare_lora_batchinit_forward_metadatar synchronizerrrrj)rrrcapture_forward_moder0rrequire_mlp_tp_gather_rMrNrIrJrKrLr>r<rVr`r)r=r_rRrmrrrrrTsp            (         ! $   TzModelRunner._dummy_runc Csd|_d|_|js dS|jjtjkrdS|jdkr!|jj r!dS|jdkr,|jj s,dSt }t |j|j}tddddd}td ||jd |d d td dttd}||j||_t |j|j}|||_td ||jdt |d d|jd d|d d dS)zCapture device graphs.NrrcSsdS)Nz cuda graphrrrrrr(\z0ModelRunner.init_device_graphs..z cpu graphz npu graph)rr8zCapture z7 begin. This can take up to several minutes. avail mem=rrcSstSr)rQrrrrr(frtz end. Time elapsed:  s. mem usage= GB. avail mem=r)rdrerrrrrrrr}rrrrlrrrrrPr7)rr before_mem graph_backend graph_runners after_memrrrrcGsJ zModelRunner.init_device_graphsc Cs<d|_|jjr |s dSt|j|j_t|jd|j}g|_g|_g|_ |jj D]}t |drNt |j dr?|j |j jnKt |j drM|j |j jnSGLANG_CPU_OMP_THREADS_BIND is not set, in this case, tp_size zG should be smaller than or equal to number of numa node on the machine aR. If you need tp_size to be larger than number of numa node, please set the CPU cores for each tp rank via SGLANG_CPU_OMP_THREADS_BIND explicitly. For example, on a machine with 2 numa nodes, where core 0-31 are on numa node 0 and core 32-63 are on numa node 1, it is suggested to use -tp 2 and bind tp rank 0 to core 0-31 and tp rank 1 to core 32-63. This is the default behavior if SGLANG_CPU_OMP_THREADS_BIND is not set and it is the same as setting SGLANG_CPU_OMP_THREADS_BIND=0-31|32-63. If you do need tp_size to be larger than the number of numa nodes, you could set SGLANG_CPU_OMP_THREADS_BIND explicitly for example SGLANG_CPU_OMP_THREADS_BIND=0-15|16-31|32-47|48-63 and run with -tp 4. If you don't want each tp rank to use all the cores on one numa node, you could set for example SGLANG_CPU_OMP_THREADS_BIND=0-15|32-47 and run with -tp 2.z!Detected the current machine has z- numa nodes available, but tp_size is set to z , so only z numa nodes are used.|zLSGLANG_CPU_OMP_THREADS_BIND setting must be aligned with TP size parameter (z%). Please double check your settings.z TP size (z")is larger than numa node number (z), in this case the available memory amount of each rank cannot be determined in prior. Please set proper `--max-total-tokens` to avoid the out-of-memory error.) rrrrmrNrrrvrrr)r omp_cpuidscpu_ids_by_node n_numa_nodethreads_bind_listrrrrs4      z ModelRunner.init_threads_bindingcCsDtd|jdddlm}tj|j|jf}||j |dS)Nz%Enabling torch tensor parallelism on z devices.r)tensor_parallel) rrr sglang.srt.layers.model_parallelrrrinit_device_meshrr)rr device_meshrrrrPs zModelRunner.apply_torch_tp stream_idxcCs|j||_dSr)rr)rrrrrupdate_decode_attn_backendz&ModelRunner.update_decode_attn_backendr_skip_attn_backend_initcCs\|s|jjr|j||j|_n|j|i}|jr ||d<|jj|j|j |fi|S)Nr) rrrrprrrrr@r()rr_rrr]rrrforward_decodes   zModelRunner.forward_decodecCsi}|jr ||d<|jdur|j|d<|jsd|d<|jduo&|j|}|r5|jj|fi||fS|s=|j||j j |j |j |fi||fS)Nr input_embedsTrX) rrrrrcan_runreplayrrprrr@r()rr_rrr]rrrrforward_extend s2    zModelRunner.forward_extendcCsD|jdkr |j|i}|jr||d<|jj|j|j|fi|S)Nrr)rrrprrrr@r()rr_rr]rrr forward_idle+ s  zModelRunner.forward_idlerreinit_attn_backend forward_countcCsT|jdks|r |j|t|j||jj}|j|j|j ||j|f}||_|S)Nr) split_indexrrpr[rrIrforward_split_prefillr@r()rr_rrnext_split_indexretrrrr> s z!ModelRunner.forward_split_prefillrsplit_forward_countc Cs|jd7_t|j|J}||||||}t}|durR|sR||t d|j } zt | Wn tyGYnwq8||||||}Wdn1s\wY|d|_tj||jt|jddd|j dur|j |S)NrzEPLB due to rank faultsTmetricsbs)r_rcuda_graph_batch)rr0with_forward_pass _forward_rawr+r*is_active_equal_lastsnapshot_active_to_lastsync_active_to_cpurrrA rebalancenext StopIterationrrrDon_forward_endrrrdon_forward_pass_end) rr_rrrrrecorder_outputsoutputelastic_ep_stategenrrrrS s\        zModelRunner.forwardc CsX|jdkr |jjn|jj}t|o|jo|j|}|r,|jj|||d}t||dS|j dur7| |n| ||j durU|j durUt|jrUtsU|j|jd|jrc|j|||d}n5|jrq|j|||d}n'|jjddr|j|||d\}}n|jr|j||d}ntd |j|j dur|jjr||t||dS) Nr)rr)rr)r)rrT)include_draft_extend_v2)rzInvalid forward mode: )rr? is_cpu_graph is_cuda_graphrrdrrrglobal_num_tokens_cpuprepare_mlp_sync_batchprepare_attn_tp_scatter_inputrTrOrvrr;'adjust_num_token_non_padded_for_attn_tp is_decoderis_split_prefillr is_extendris_idlerrUr is_last_rankpost_forward_mlp_sync_batch) rr_rrrr mode_checkrrrrrr sn              zModelRunner._forward_rawr sampling_infocCs|||jdSr)update_regex_vocab_maskapply_logits_biasnext_token_logits)rrrrrr_preprocess_logits szModelRunner._preprocess_logitsc srt|trtjfdd|DddS|j|jjjj j r1j }|Sj d}|S)aSample and compute logprobs and update logits_output. Args: logits_output: The logits output from the model forward forward_batch: The forward batch that generates logits_output Returns: A list of next_token_ids csg|]}|qSr)sample)r,valuesr_rrrr/ sz&ModelRunner.sample..r{)axisr)r=tuplerstackrrrErHtop_logprobs_numstoken_ids_logprobsr?rr(rB)rrr_next_token_idsrrrr s&   zModelRunner.samplecCs8|jsdS|||j|j||j|j|j|jdS)a Compute token_ids_logprobs without performing sampling. Optimized path for prefill-only requests that need token_ids_logprobs but don't require next token generation. Skips expensive sampling operations while still providing requested probability information. Args: logits_output: The logits output from the model forward forward_batch: The forward batch that generates logits_output N)rrrrEcompute_logprobs_onlyrHr)rrr_rrrr sz!ModelRunner.compute_logprobs_onlycCs8t|jjddpt|jjdi}|durdSd|v}|S)zDetect if the model has "mrope" rope_scaling type. mrope requires keep "rope_deltas" between prompt and decoding phases.rope_parametersN rope_scalingF mrope_section)rrr)rris_mrope_enabledrrrmodel_is_mrope& s zModelRunner.model_is_mropeurlcCs4ddlm}td|||j|jj|dS)Nr)RemoteModelLoaderzSaving model to )sglang.srt.model_loader.loaderrrr save_modelrrr)rrrrrrsave_remote_model2 s zModelRunner.save_remote_modelpathrmax_sizecCs>ddlm}td|d|d|||j|||dS)Nr)ShardedStateLoaderzSave sharded model to z with pattern z and max_size )rrrrrr)rrrrrrrrsave_sharded_model8 s zModelRunner.save_sharded_modelactioncCs|jj|ddS)N)r)rr})rrrrr check_weightsB szModelRunner.check_weightsc Cszddlm}||}||jWdSty,}z dd|fWYd}~Sd}~wtyK}ztd|dt|fWYd}~Sd}~ww)z:Update weights from IPC for checkpoint-engine integration.r))SGLangCheckpointEngineWorkerExtensionImpl)Tz(IPC weight update completed successfullyFz&IPC weight update failed: ImportError NzIPC weight update failed: ) 5sglang.srt.checkpoint_engine.checkpoint_engine_workerrupdate_weights_from_ipc zmq_handlesrurrrPr)rrecv_reqrworkerryrrrrE s  z#ModelRunner.update_weights_from_ipcc Cs|js|jjrtjdkrdSt|j |j Ft dtjdt ttjtjdddftj|jdWdn1sKwYWddSWddS1scwYdS)Nrz*Pre-allocating symmetric memory pool with z GiBirj)rrenable_symm_memr, SGLANG_SYMM_MEM_PREALLOC_GB_SIZErrrrrrrrr)r"rtuint8r.rrrriV s( "z*ModelRunner.prealloc_symmetric_memory_pool)NNNFNNN)NF)rr)r)F)FN)Fr)FNFr)NN)\rrrrrfloatintrcrrrOrNrrr r_r=rr rrrFr2rr/rrrr,rWrbrerhrrrrr rTensorrrrrRrLrrrpropertyrrrrrrrrrYr`rarrrVrbrrrrcrhrrPrrTrBrVrrGrrrrrrrbrrrrrrrrrirrrrrs      9"~(>   J 26 - ;" !          0  6   t.K"   )    = K   &      rrcCs,t|}|D] \}}t|||qdSr)dictr\ra)rr params_dictr'rrrrrl s  rcCst|tr ||}||Sr)r=rrto)rrrrrrrr s   rc@s,eZdZUdZeeed<defddZdS)rztorch.Tensor that gets serialized by MultiprocessingSerializer (which only serializes a pointer and not the data). The i-th element in the list corresponds to i-th rank's GPU.rrcCst|j|Sr)rg deserializer)rrrrrr rzLocalSerializedTensor.getN) rrrrrbytesrrrrrrrrx s  r)rrr>rrrrrrr collectionsr dataclassesrtypingrrrrr rtorch.distributedrrr sglang.srt.configsr r r rrrrrrrrrr sglang.srt.configs.device_configrsglang.srt.configs.load_configrrsglang.srt.configs.model_configrrr sglang.srt.configs.update_configrsglang.srt.constantsr/sglang.srt.debug_utils.tensor_dump_forward_hookr sglang.srt.distributedr!r"r#r$r%r&r'r(r?r@rA"sglang.srt.layers.logits_processorrB-sglang.srt.layers.moe.routed_experts_capturerrCrDrEsglang.srt.layers.moe.utilsrFsglang.srt.layers.poolerrG)sglang.srt.layers.quantization.fp8_kernelrHsglang.srt.layers.samplerrIsglang.srt.layers.torchao_utilsrJsglang.srt.lora.lora_managerrKsglang.srt.lora.lora_registryrL"sglang.srt.managers.schedule_batchrMsglang.srt.mem_cache.allocatorrN sglang.srt.mem_cache.memory_poolrO*sglang.srt.model_executor.cpu_graph_runnerrP+sglang.srt.model_executor.cuda_graph_runnerrQrR,sglang.srt.model_executor.forward_batch_inforSrTrUrV&sglang.srt.model_executor.hook_managerrW'sglang.srt.model_executor.input_buffersrX5sglang.srt.model_executor.model_runner_kv_cache_mixinrY5sglang.srt.model_executor.piecewise_cuda_graph_runnerrZrr[r\;sglang.srt.model_loader.remote_instance_weight_loader_utilsr]r^r_sglang.srt.model_loader.utilsr`$sglang.srt.model_loader.weight_utilsra'sglang.srt.sampling.sampling_batch_inforbsglang.srt.server_argsrcrdre sglang.srt.speculative.spec_inforfsglang.srt.utilsrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxryrz#sglang.srt.utils.nvtx_pytorch_hooksr{sglang.srt.utils.offloaderr|r}r~sglang.srt.utils.patch_torchrr+sglang.srt.utils.torch_memory_saver_adapterrsglang.srt.utils.weight_checkerr$sglang.srt.weight_sync.tensor_bucketrrrr rr%sglang.srt.hardware_backend.npu.utilsrrrrfloat8_e4m3fnuzrrrrrr  getLoggerrrModulerFilterrrrrrrrrrrrrs     <    (                             X        h