o ۾ivX@sdZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z ddl Z ddl Z ddlmZddlmZddlmZddlmZeeZejdd Zejd efd d Zejd efd dZde de d e fddZde d e dBfddZ!efde de dede ffddZ"e"ddZ#e"ddZ$e"ddZ%e"ddZ&e"d d!Z'e"d"d#Z(e"d"d$Z)e"d"d%Z*e"d"d&Z+e"d'd(Z,e"d"d)Z-e"d*d+d,d-d.Z.ejd efd/d0Z/ejd efd1d2Z0ejd efd3d4Z1ejd efd5d6Z2ejd efd7d8Z3ejd efd9d:Z4ejd efd;d<Z5ejd efd=d>Z6ejd efd?d@Z7d edBfdAdBZ8dCe9dDe9d efdEdFZ:  G GddCe9dDe9dHe9dIe9dJe9dKe dLe j;dMedNedBdOedPed efdQdRZZ>dTe j?dUe j?dVe j?d dfdWdXZ@dTe j?dUe j?dVe j?d dfdYdZZAe>d[e@dTgeAd\e jBjCd]gd^d_d`e j?dae j?dbe j?dce j?dde j?dee j;dfedge d e j?fdhdiZDe jBEd]d`e j?dae j?dbe j?dce j?dde j?dee j;dfedge d e j?fdjdkZFe jBjCdlgd^d_d`e j?dae j?dbe j?dce j?dee j;dge d e j?fdmdnZGe jBEdld`e j?dae j?dbe j?dce j?dee j;dge d e j?fdodpZHe jBjCdqgd^d_dre j?dse j?d eIe j?e j?ffdtduZJe jBEdqdre j?dse j?d eIe j?e j?ffdvdwZKdre j?dxe j?dye j?dze j?d{e j?d|e j;dge d e j?fd}d~ZL ddre j?dxe j?de j?de j?d|e j;de j?dBd e j?fddZMdre j?dse j?d eIe j?e j?ffddZNe"ddZOejd efddZPejd efddZQdede j;de j?de j?fddZRgdZSdS)zoCompatibility wrapper for FlashInfer API changes. Users of vLLM should always import **only** these wrappers. N)Callable)AnyNoReturn) init_logger)vllm_is_batch_invariant)current_platformFLASHINFER_CUBINS_REPOSITORYzWhttps://edge.urm.nvidia.com/artifactory/sw-kernelinferencelibrary-public-generic-local/returncCs,tjrdStjddurdStddS)z7Return `True` if flashinfer-cubin package is available.Tflashinfer_cubinNz&flashinfer-cubin package was not foundF)envsVLLM_HAS_FLASHINFER_CUBIN importlibutil find_speclogger debug_oncerrI/home/ubuntu/.local/lib/python3.10/site-packages/vllm/utils/flashinfer.pyhas_flashinfer_cubin&s  rcCsDtjddurtddSts tddur tddSdS)z8Return `True` if flashinfer-python package is available. flashinferNz2FlashInfer unavailable since package was not foundFnvcczSFlashInfer unavailable since nvcc was not found and not using pre-downloaded cubinsT)r rrrrrshutilwhichrrrrhas_flashinfer1s r___cOstd)z/Placeholder for unavailable FlashInfer backend.zFlashInfer backend is not available. Please install the package to enable FlashInfer kernels: https://github.com/flashinfer-ai/flashinfer) RuntimeError)rrrrr_missingDsr module_namec Cs&zt|WSttfyYdSw)zBSafely import a submodule and return it, or None if not available.N)r import_module ImportErrorModuleNotFoundError)rrrr_get_submoduleMs  r" attr_name fallback_fn.cs&tjfddfdd}|S)z5Create a lazy import wrapper for a specific function.cs&tsdSt}|rt|dSdSN)rr"getattr)mod)r#rrr _get_impl[sz'_lazy_import_wrapper.._get_implcs*}|dur|i|S||i|Sr%r)argskwargsimpl)r(r$rrwrapperbsz%_lazy_import_wrapper..wrapper) functoolscache)rr#r$r,r)r(r#r$rr_lazy_import_wrapperVsr/flashinfer.fused_moetrtllm_bf16_moetrtllm_fp8_block_scale_moetrtllm_fp8_per_tensor_scale_moecutlass_fused_moe$flashinfer.cute_dsl.blockscaled_gemmgrouped_gemm_nt_maskedr fp4_quantizenvfp4_batched_quantize*silu_and_mul_scaled_nvfp4_experts_quantizescaled_fp4_grouped_quantizezflashinfer.fp4_quantizationblock_scale_interleavetrtllm_fp4_block_scale_moezflashinfer.autotunerautotunecOstSr%) contextlib nullcontext)r)r*rrrsr@)r$cCto tjdduS)z5Return `True` if FlashInfer comm module is available.flashinfer.commNrr rrrrrrhas_flashinfer_commsrDcC@tsdSgd}|D]\}}t|}|rt||sdSq dS)z7Return `True` if FlashInfer mnnvl all2all is available.F))rBMapping)zflashinfer.comm.mnnvl MnnvlMemory)flashinfer.comm.trtllm_alltoallMnnvlMoe)rHMoEAlltoallInfoT)rDr"hasattrrequired_functionsrr#r'rrrhas_flashinfer_all2all rNcCrA)z4Return `True` if FlashInfer MoE module is available.r0NrCrrrrhas_flashinfer_moesrPcCrA)z:Return ``True`` if FlashInfer cutedsl module is available.zflashinfer.cute_dslNrCrrrrhas_flashinfer_cutedslsrQcCrE)z:Return `True` if FlashInfer TRTLLM fused MoE is available.F))r0r2)r0r3r0r<)r0trtllm_mxint4_block_scale_moeTrPr"rKrLrrrhas_flashinfer_trtllm_fused_moes rUcCrE)z;Return `True` if FlashInfer CUTLASS fused MoE is available.F))r0r4)rr7)rnvfp4_block_scale_interleaverRTrTrLrrr has_flashinfer_cutlass_fused_moerOrWcCrE)z=Return ``True`` if FlashInfer CUTLASS fused MoE is available.F))r5r6)rr:)r&silu_and_scaled_nvfp4_experts_quantizeT)rQr"rKrLrrr-has_flashinfer_cutedsl_grouped_gemm_nt_maskeds rYc Cs~trdSz tjtdd}|jdk}|rtd|WStd|j|WSty>}z td|WYd}~d Sd}~ww) zReturn `True` if NVIDIA's artifactory is accessible. This checks connectivity to the kernel inference library artifactory which is required for downloading certain cubin kernels like TRTLLM FHMA. T)timeoutz NVIDIA artifactory is accessiblez2NVIDIA artifactory returned failed status code: %dz+Failed to connect to NVIDIA artifactory: %sNF) rrequestsgetr status_coderr warning_once Exception)response accessibleerrrhas_nvidia_artifactorys$   recCstrdStdo tS)z TRTLLM attention is supported if the platform is SM100, NVIDIA artifactory is accessible, and batch-invariant mode is not enabled. Fd)rris_device_capability_familyrerrrrsupports_trtllm_attentionsrhcCsddlm}|}|jjS)a, This function should only be called during initialization stage when vllm config is set. Return `None` if --attention-config.use_trtllm_attention is not set, return `True` if TRTLLM attention is forced to be used, return `False` if TRTLLM attention is forced to be not used. r)get_current_vllm_config) vllm.configriattention_configuse_trtllm_attention)ri vllm_configrrrforce_use_trtllm_attention$s rn num_qo_heads num_kv_headscCs$tdurdSt}|o||dkS)z=Check if the current configuration supports TRTLLM attention.Fr)rnrh)rorp has_trtllmrrrcan_use_trtllm_attention2s rrF num_tokens max_seq_lendcp_world_sizekv_cache_dtypeq_dtype is_prefillforce_use_trtllm has_sinkshas_specc Cs|dur|sdS|dkrtddSts|rtddS||dkr.|r,tddS| r9|s9tdd S|tkrFtd d S| rOtd d S|durs|rb|d k} | r`td | S|dkoi|d k} | rqtd| Stdd S)z*Return `True` if TRTLLM attention is used.NFzcTrtllm does not support returning LSE and as a result does not support DCP, reverting to FlashInferzkTRTLLM attention is not supported on this platform, but --attention-config.use_trtllm_attention is set to 1rzTRTLLM attention is not supported for this combination of query and key heads, but --attention-config.use_trtllm_attention is set to 1z:Using TRTLLM attention (enabled for speculative decoding).Tz,Using TRTLLM attention (query is quantized).z6Using TRTLLM attention (required for attention sinks).autoz/Using TRTLLM prefill attention (auto-detected).z.Using TRTLLM decode attention (auto-detected).zLUsing TRTLLM attention (--attention-config.use_trtllm_attention is set to 1))rr`rh info_oncer fp8_dtype) rorprsrtrurvrwrxryrzr{ use_trtllmrrrrl:sT        rl)direct_register_custom_opkk_nopek_pecCsddlm}||||dS)arCustom op wrapper for flashinfer's concat_mla_k. This is an in-place operation that concatenates k_nope and k_pe into k. The kernel is optimized for DeepSeek V3 dimensions: - num_heads=128 - nope_dim=128 - rope_dim=64 Key optimizations: - Warp-based processing with software pipelining - Vectorized memory access (int2 for nope, int for rope) - L2 prefetching for next row while processing current - Register reuse for rope values across all heads Args: k: Output tensor, shape [num_tokens, num_heads, nope_dim + rope_dim]. Modified in-place. k_nope: The nope part of k, shape [num_tokens, num_heads, nope_dim]. k_pe: The rope part of k (shared), shape [num_tokens, 1, rope_dim]. This is broadcast to all heads. r) concat_mla_kN)flashinfer.concat_opsr)rrrrrrr_flashinfer_concat_mla_ks rcCsdSr%r)rrrrrr_flashinfer_concat_mla_k_fakesrflashinfer_concat_mla_k)op_nameop_func mutates_args fake_implzvllm::flashinfer_mm_fp4cuda)r device_typesABA_scaleB_scaleg_scaledtypeuse_8x4_sf_layoutbackendc Cs&ddlm}|||||||d||d S)Nr)mm_fp4) block_sizerr)rr) rrrrrrrrflashinfer_mm_fp4_rrrflashinfer_mm_fp4s rcCs tj|jd|jd||jdS)Nrr|rdevicetorchemptyshaper)rrrrrrrrrrrflashinfer_mm_fp4_fakes rz vllm::bmm_fp8cCs ddlm}||||||d|S)Nr)bmm_fp8)rr)rrrrrrbmm_fp8_rrrrs rcCs(tj|jd|jd|jd||jdS)Nrr|rr)rrrrrrrrr bmm_fp8_fakes rzvllm::flashinfer_nvfp4_quantizea a_global_sfcCs*ddlm}ddlm}||||jddS)Nr)SfLayout)nvfp4_quantizeF)sfLayout do_shuffle)rrr layout_8x4)rrrnvfp4_quantize_rrrflashinfer_nvfp4_quantize s   rcCs^|j\}}dd}||d}|d}||d}tj||dtj|jdtj||tj|jdfS)NcSs||d||S)Nr|r)xyrrrr@"sz0flashinfer_nvfp4_quantize_fake..rrr)rrruint8r)rrmnround_up rounded_mscale_n rounded_nrrrflashinfer_nvfp4_quantize_fakes    rb block_scale_a block_scale_balpha out_dtypec Cs|jdkr |jdks J|jdkr|jdksJ|ddkr&|ddks(J|jd|jdks4J|dvrD|tj}|tj}|dkrQ|jddkrQdnd }t||||||||d S) Nrr|)cutlasscudnntrtllmr TF)rr)ndimstriderviewrrrt)rrrrrrrrrrrflashinfer_scaled_fp4_mm-s$    rscale_ascale_bbiascCs|jdkr |jdks J|jd|jdksJ|dkr$|dks&J|jtjkr2|jtjks4J|jjdkr@|jjdksBJ|jtjkrN|jtjksPJ|jjdkr\|jjdks^Jt | d| d|||d |jd|jd}|dur||}|S)Nrr|rrr}) rrnumelrr float8_e4m3fnrtypefloat32r unsqueezer)rrrrrroutputrrrflashinfer_scaled_fp8_mmMs( rcCs t||Sr%)r)rrrrr$flashinfer_quant_nvfp4_8x4_sf_layoutks rflashinfer.gemmfp8_blockscale_gemm_sm90cCstotdottddS)z>Return `True` if FlashInfer block-scale FP8 GEMM is available.Zrr)rris_device_capabilityrKr"rrrr"has_flashinfer_fp8_blockscale_gemmvs  rcCs tjotS)z>Return `True` if FlashInfer block-scale FP8 GEMM is supported.)r #VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFERrrrrr+is_flashinfer_fp8_blockscale_gemm_supportedsris_flashinfer_supported output_dtypeinputweightc Csb|sdSd}d}|j}|j}|tjko.|tjko.|tjko.|jd|dko.|jd|dk}|S)NF@rr|)rrbfloat16rr) rrrr N_MULTIPLE K_MULTIPLE weight_dtype input_dtypeshould_use_flashinferrrr-should_use_flashinfer_for_blockscale_fp8_gemms  r)r%flashinfer_trtllm_fp8_block_scale_moeflashinfer_cutlass_fused_moe)flashinfer_cutedsl_grouped_gemm_nt_maskedflashinfer_fp4_quantizer9r:rVr<r=rPrDrNrWrYrrerhrrrlrrrflashinfer_fp8_blockscale_gemmrr)NFFr%)T__doc__r>r-r importlib.utilosrcollections.abcrtypingrrr]r vllm.envsr vllm.loggerr*vllm.model_executor.layers.batch_invariantrvllm.platformsr__name__renvironr^rr.boolrrrstrr"r/flashinfer_trtllm_bf16_moer*flashinfer_trtllm_fp8_per_tensor_scale_moerrrr8r9r:rVr<r=rDrNrPrQrUrWrYrerhrnintrrrrlvllm.utils.torch_utilsrTensorrrlibrary custom_opr register_fakerrrtuplerrrrrrrrr__all__rrrrsb                T              &