o .icQ@s8dZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z ddl Z ddl Z ddlmZddlmZddlmZddlmZeeZejdd Zejd efd d Zejd efd dZde de d e fddZde d e dBfddZ!efde de dede ffddZ"e"ddZ#e"ddZ$e"ddZ%e"dd Z&e"d!d"Z'e"d!d#Z(e"d!d$Z)e"d!d%Z*e"d!d&Z+e"d!d'Z,e"d(d)d*d+d,Z-ejd efd-d.Z.ejd efd/d0Z/ejd efd1d2Z0ejd efd3d4Z1ejd efd5d6Z2ejd efd7d8Z3ejd efd9d:Z4ejd efd;d<Z5ejd efd=d>Z6d edBfd?d@Z7dAe8dBe8d efdCdDZ9  E EddAe8dBe8dFe8dGe8dHe8dIe dJe j:dKedLedBdMedNed efdOdPZ;er e jdUe j>dVe j>dWe j>dXe j>dYe j:dZed[e d e j>fd\d]Z?e j<@dQdTe j>dUe j>dVe j>dWe j>dXe j>dYe j:dZed[e d e j>fd^d_ZAe jdUe j>dVe j>dWe j>dYe j:d[e d e j>fdadbZBe j<@d`dTe j>dUe j>dVe j>dWe j>dYe j:d[e d e j>fdcddZCe jdge j>d eDe j>e j>ffdhdiZEe j<@dedfe j>dge j>d eDe j>e j>ffdjdkZFdfe j>dle j>dme j>dne j>doe j>dpe j:d[e d e j>fdqdrZG ddfe j>dle j>dse j>dte j>dpe j:due j>dBd e j>fdvdwZHdfe j>dge j>d eDe j>e j>ffdxdyZIe"dzd{ZJejd efd|d}ZKejd efd~dZLdede j:de j>de j>fddZMgdZNdS)zoCompatibility wrapper for FlashInfer API changes. Users of vLLM should always import **only** these wrappers. N)Callable)AnyNoReturn) init_logger)vllm_is_batch_invariant)current_platformFLASHINFER_CUBINS_REPOSITORYzWhttps://edge.urm.nvidia.com/artifactory/sw-kernelinferencelibrary-public-generic-local/returncCs,tjrdStjddurdStddS)z7Return `True` if flashinfer-cubin package is available.Tflashinfer_cubinNz&flashinfer-cubin package was not foundF)envsVLLM_HAS_FLASHINFER_CUBIN importlibutil find_speclogger debug_oncerrR/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/utils/flashinfer.pyhas_flashinfer_cubin&s  rcCsDtjddurtddSts tddur tddSdS)z8Return `True` if flashinfer-python package is available. flashinferNz2FlashInfer unavailable since package was not foundFnvcczSFlashInfer unavailable since nvcc was not found and not using pre-downloaded cubinsT)r rrrrrshutilwhichrrrrhas_flashinfer1s r___cOstd)z/Placeholder for unavailable FlashInfer backend.zFlashInfer backend is not available. Please install the package to enable FlashInfer kernels: https://github.com/flashinfer-ai/flashinfer) RuntimeError)rrrrr_missingDsr module_namec Cs&zt|WSttfyYdSw)zBSafely import a submodule and return it, or None if not available.N)r import_module ImportErrorModuleNotFoundError)rrrr_get_submoduleMs  r" attr_name fallback_fn.cs&tjfddfdd}|S)z5Create a lazy import wrapper for a specific function.cs&tsdSt}|rt|dSdSN)rr"getattr)mod)r#rrr _get_impl[sz'_lazy_import_wrapper.._get_implcs*}|dur|i|S||i|Sr%r)argskwargsimpl)r(r$rrwrapperbsz%_lazy_import_wrapper..wrapper) functoolscache)rr#r$r,r)r(r#r$rr_lazy_import_wrapperVsr/flashinfer.fused_moetrtllm_fp8_block_scale_moetrtllm_fp8_per_tensor_scale_moecutlass_fused_moe$flashinfer.cute_dsl.blockscaled_gemmgrouped_gemm_nt_maskedr fp4_quantizenvfp4_batched_quantize*silu_and_mul_scaled_nvfp4_experts_quantizescaled_fp4_grouped_quantizenvfp4_block_scale_interleavetrtllm_fp4_block_scale_moezflashinfer.autotunerautotunecOstSr%) contextlib nullcontext)r)r*rrrsr?)r$cCto tjdduS)z5Return `True` if FlashInfer comm module is available.flashinfer.commNrr rrrrrrhas_flashinfer_commsrCcC@tsdSgd}|D]\}}t|}|rt||sdSq dS)z7Return `True` if FlashInfer mnnvl all2all is available.F))rAMapping)zflashinfer.comm.mnnvl MnnvlMemory)flashinfer.comm.trtllm_alltoallMnnvlMoe)rGMoEAlltoallInfoT)rCr"hasattrrequired_functionsrr#r'rrrhas_flashinfer_all2all rMcCr@)z4Return `True` if FlashInfer MoE module is available.r0NrBrrrrhas_flashinfer_moesrOcCr@)z:Return ``True`` if FlashInfer cutedsl module is available.zflashinfer.cute_dslNrBrrrrhas_flashinfer_cutedslsrPcCrD)z:Return `True` if FlashInfer TRTLLM fused MoE is available.F))r0r1)r0r2r0r;TrOr"rJrKrrrhas_flashinfer_trtllm_fused_moes rScCrD)z;Return `True` if FlashInfer CUTLASS fused MoE is available.F))r0r3)rr6)rr:rQTrRrKrrr has_flashinfer_cutlass_fused_moerNrTcCrD)z=Return ``True`` if FlashInfer CUTLASS fused MoE is available.F))r4r5)rr9)r&silu_and_scaled_nvfp4_experts_quantizeT)rPr"rJrKrrr-has_flashinfer_cutedsl_grouped_gemm_nt_maskeds rVc Cs~trdSz tjtdd}|jdk}|rtd|WStd|j|WSty>}z td|WYd}~d Sd}~ww) zReturn `True` if NVIDIA's artifactory is accessible. This checks connectivity to the kernel inference library artifactory which is required for downloading certain cubin kernels like TRTLLM FHMA. T)timeoutz NVIDIA artifactory is accessiblez2NVIDIA artifactory returned failed status code: %dz+Failed to connect to NVIDIA artifactory: %sNF) rrequestsgetr status_coderr warning_once Exception)response accessibleerrrhas_nvidia_artifactorys$   rbcCstrdStdo tS)z TRTLLM attention is supported if the platform is SM100, NVIDIA artifactory is accessible, and batch-invariant mode is not enabled. Fd)rris_device_capability_familyrbrrrrsupports_trtllm_attentionsrecCsddlm}|}|jjS)a, This function should only be called during initialization stage when vllm config is set. Return `None` if --attention-config.use_trtllm_attention is not set, return `True` if TRTLLM attention is forced to be used, return `False` if TRTLLM attention is forced to be not used. r)get_current_vllm_config) vllm.configrfattention_configuse_trtllm_attention)rf vllm_configrrrforce_use_trtllm_attention!s rk num_qo_heads num_kv_headscCs$tdurdSt}|o||dkS)z=Check if the current configuration supports TRTLLM attention.Fr)rkre)rlrm has_trtllmrrrcan_use_trtllm_attention/s roF num_tokens max_seq_lendcp_world_sizekv_cache_dtypeq_dtype is_prefillforce_use_trtllm has_sinkshas_specc Cs|dur|sdS|dkrtddSts|rtddS||dkr.|r,tddS| r9|s9tdd S|tkrFtd d S| rOtd d S|durs|rb|d k} | r`td | S|dkoi|d k} | rqtd| Stdd S)z*Return `True` if TRTLLM attention is used.NFzcTrtllm does not support returning LSE and as a result does not support DCP, reverting to FlashInferzkTRTLLM attention is not supported on this platform, but --attention-config.use_trtllm_attention is set to 1rzTRTLLM attention is not supported for this combination of query and key heads, but --attention-config.use_trtllm_attention is set to 1z:Using TRTLLM attention (enabled for speculative decoding).Tz,Using TRTLLM attention (query is quantized).z6Using TRTLLM attention (required for attention sinks).autoz/Using TRTLLM prefill attention (auto-detected).z.Using TRTLLM decode attention (auto-detected).zLUsing TRTLLM attention (--attention-config.use_trtllm_attention is set to 1))rr]re info_oncer fp8_dtype) rlrmrprqrrrsrtrurvrwrx use_trtllmrrrri7sT        rizvllm::flashinfer_mm_fp4cuda) mutates_args device_typesABA_scaleB_scaleg_scaledtypeuse_8x4_sf_layoutbackendc Cs&ddlm}|||||||d||d S)Nr)mm_fp4) block_sizerr)rr) rrrrrrrrflashinfer_mm_fp4_rrrflashinfer_mm_fp4s rcCs tj|jd|jd||jdS)Nrryrdevicetorchemptyshaper)rrrrrrrrrrrflashinfer_mm_fp4_fakes rz vllm::bmm_fp8cCs ddlm}||||||d|S)Nr)bmm_fp8)rr)rrrrrrbmm_fp8_rrrrs rcCs(tj|jd|jd|jd||jdS)Nrryrr)rrrrrrrrr bmm_fp8_fakes rzvllm::flashinfer_nvfp4_quantizea a_global_sfcCs*ddlm}ddlm}||||jddS)Nr)SfLayout)nvfp4_quantizeF)sfLayout do_shuffle)rrr layout_8x4)rrrnvfp4_quantize_rrrflashinfer_nvfp4_quantizes   rcCs^|j\}}dd}||d}|d}||d}tj||dtj|jdtj||tj|jdfS)NcSs||d||S)Nryr)xyrrrr?sz0flashinfer_nvfp4_quantize_fake..rrr)rrruint8r)rrmnround_up rounded_mscale_n rounded_nrrrflashinfer_nvfp4_quantize_fakes    rb block_scale_a block_scale_balpha out_dtypec Cs|jdkr |jdks J|jdkr|jdksJ|ddkr&|ddks(J|jd|jdks4J|dkrD|tj}|tj}|dkrQ|jddkrQdnd }t||||||||d S) Nrrycutlasstrtllmr TF)rr)ndimstriderviewrrrt)rrrrrrrrrrrflashinfer_scaled_fp4_mms$    rscale_ascale_bbiascCs|jdkr |jdks J|jd|jdksJ|dkr$|dks&J|jtjkr2|jtjks4J|jjdkr@|jjdksBJ|jtjkrN|jtjksPJ|jjdkr\|jjdks^Jt | d| d|||d |jd|jd}|dur||}|S)Nrryrrrz) rrnumelrr float8_e4m3fnrtypefloat32r unsqueezer)rrrrrroutputrrrflashinfer_scaled_fp8_mms( rcCs t||Sr%)r)rrrrr$flashinfer_quant_nvfp4_8x4_sf_layout9s rflashinfer.gemmfp8_blockscale_gemm_sm90cCstotdottddS)z>Return `True` if FlashInfer block-scale FP8 GEMM is available.Zrr)rris_device_capabilityrJr"rrrr"has_flashinfer_fp8_blockscale_gemmDs  rcCs tjotS)z>Return `True` if FlashInfer block-scale FP8 GEMM is supported.)r #VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFERrrrrr+is_flashinfer_fp8_blockscale_gemm_supportedNsris_flashinfer_supported output_dtypeinputweightc Csb|sdSd}d}|j}|j}|tjko.|tjko.|tjko.|jd|dko.|jd|dk}|S)NF@rry)rrbfloat16rr) rrrr N_MULTIPLE K_MULTIPLE weight_dtype input_dtypeshould_use_flashinferrrr-should_use_flashinfer_for_blockscale_fp8_gemmWs  r)r%flashinfer_trtllm_fp8_block_scale_moeflashinfer_cutlass_fused_moe)flashinfer_cutedsl_grouped_gemm_nt_maskedflashinfer_fp4_quantizer8r9r:r;r<rOrCrMrTrVrrbrerorirrrflashinfer_fp8_blockscale_gemmrr)NFFr%)O__doc__r=r-r importlib.utilosrcollections.abcrtypingrrrZr vllm.envsr vllm.loggerr*vllm.model_executor.layers.batch_invariantrvllm.platformsr__name__renvironr[rr.boolrrrstrr"r/r*flashinfer_trtllm_fp8_per_tensor_scale_moerrrr7r8r9r:r;r<rCrMrOrPrSrTrVrbrerkintrorrilibrary custom_opTensorr register_fakerrrtuplerrrrrrrrr__all__rrrrs*                T           &