o ۾iF:@s~UdZddlZddlZddlZddlmZddlmZddlm Z m Z ddl Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZGd d d eZejd efddZejd efddZde de d e fddZdaede fdBed<da ede fdBed<da!ede fdBed<da"ede fdBed<da#ede fdBed<da$ede fdBed<da%ede fdBed<da&ede fdBed<da'ede fdBed<d\d d!Z(d e)fd"d#Z*ejd e+e)fd$d%Z,d&e j-d e j-fd'd(Z.d)d*Z/d+d,Z0d-d.Z1d/d0Z2d1e j-d2e3e j-e j-fd3e j-d4e j-d5e j-d6ed e j-fd7d8Z4d9e j-d:e)d;e)d e j-fde j-d?e j-d3e j-d9e j-d@e j-dAe j-dBe)d6ed e j-fdCdDZ6d&e j-fdEdFZ7d&e)dGe)d e)fdHdIZ8d&e)dJe)fdKdLZ9dMdMgZ:e j;dNej d]dVe j?dWe j-dXedBfdYdZZ@gd[ZAdS)^zmCompatibility wrapper for DeepGEMM API changes. Users of vLLM should always import **only** these wrappers. N)Callable)Enum)AnyNoReturn)logger)get_fp8_min_max)current_platform) has_deep_gemmcdivc@s4eZdZdZdZdZed ddZed dd ZdS) DeepGemmQuantScaleFMTrreturnNcCs^t|dd}|dur dStjototdu}|s|j|_dStdr)|j |_dS|j |_dS)z>Initialize the oracle decision and store it in the class cache _oracle_cacheNd) getattrenvsVLLM_USE_DEEP_GEMM_E8M0is_deep_gemm_supported_fp8_gemm_nt_implFLOAT32rris_device_capability_familyUE8M0FLOAT32_CEIL_UE8M0)clscacheduse_e8m0rH/home/ubuntu/.local/lib/python3.10/site-packages/vllm/utils/deep_gemm.pyinit_oracle_cache&s z'DeepGemmQuantScaleFMT.init_oracle_cachecCs t|dd}|dusJd|S)z*Return the pre-initialized oracle decisionrNz2DeepGemmQuantScaleFMT oracle cache not initialized)r)rrrrr from_oracle<s z!DeepGemmQuantScaleFMT.from_oraclerN)rr ) __name__ __module__ __qualname__rrr classmethodr r!rrrrr s r rcCs,to tdp td}tjoto|S)zReturn `True` if DeepGEMM is supported on the current platform. Currently, only Hopper and Blackwell GPUs are supported. Zr)ris_cudais_device_capabilityrrVLLM_USE_DEEP_GEMMr )is_supported_archrrrrDs  rcCs^ts tddSttdurtjddddStjr&tjddddStjd dddS) znReturn `True` if vLLM is configured to use DeepGEMM " "E8M0 scale on a Hopper or Blackwell-class GPU. z>DeepGEMM E8M0 disabled: DeepGEMM not supported on this system.FNz3DeepGEMM E8M0 disabled: _fp8_gemm_nt_impl not foundlocal)scopez*DeepGEMM E8M0 enabled on current platform.Tz0DeepGEMM E8M0 disabled on current configuration.)rr debug_once _lazy_initr info_oncerrrrrris_deep_gemm_e8m0_usedPs r1___cOstd)z-Placeholder for unavailable DeepGEMM backend.zDeepGEMM backend is not available or outdated. Please install or update the `deep_gemm` to a newer version to enable FP8 kernels.) RuntimeError)r2r3rrr_missingksr5.r _grouped_impl_grouped_masked_impl_fp8_mqa_logits_impl_fp8_paged_mqa_logits_impl#_get_paged_mqa_logits_metadata_impl%_get_mn_major_tma_aligned_tensor_impl,_get_mk_alignment_for_contiguous_layout_impl'_transform_sf_into_required_layout_implcCstdus tdus tdus tdus tdus tdus tdus tdur"dSts'dSd}t j |ds;t j tjdt j |<td}t|ddat|ddat|ddat|ddat|ddat|d dat|d dat|d dat|d datdS) z2Import deep_gemm and resolve symbols on first use.NDG_JIT_CACHE_DIR deep_gemm fp8_gemm_nt m_grouped_fp8_gemm_nt_contiguousfp8_m_grouped_gemm_nt_maskedfp8_mqa_logitsfp8_paged_mqa_logitsget_paged_mqa_logits_metadataget_mn_major_tma_aligned_tensor&get_mk_alignment_for_contiguous_layout!transform_sf_into_required_layout)rr6r7r8r9r:r<r=r osenvirongetpathjoinrVLLM_CACHE_ROOT importlib import_modulerr;r r )DEEP_GEMM_JIT_CACHE_ENV_NAME_dgrrrr/~sF        r/cCsttd}t|S)Nr?)r/rOrPint get_num_sms)rRrrrrTs  rTcCs"ttdur tSt}||gSN)r/r<r5) mk_align_sizerrrrGs rGxcCsttdur tSt|S)z6Wrapper for DeepGEMM's get_mn_major_tma_aligned_tensorN)r/r;r5rWrrr get_col_major_tma_aligned_tensorsrYcOsNttdurt|i|Sd|vr|d}|d=nt}t|d| i|S)Nr1disable_ue8m0_cast)r/rr5r1)argskwargs use_ue8m0rrrr@sr@cO2ttdurt|i|St|dt i|SNrZ)r/r6r5r1r[r\rrrrArAcOr^r_)r/r7r5r1r`rrrrBrarBcOr^r_)r/r=r5r1r`rrrrHrarHqkvweights cu_seqlen_ks cu_seqlen_ke clean_logitscCs(ttdur tSt||||||dS)a%Compute FP8 MQA logits for a single sequence without KV paging. Args: q: Query tensor of shape [M, H, D]. Casted to `torch.float8_e4m3fn` by caller. kv: Tuple `(k_fp8, k_scales)` where `k_fp8` has shape [N, D] with dtype `torch.float8_e4m3fn` and `k_scales` has shape [N]) with dtype `torch.float32`. weights: weights of shape [M, H], dtype `torch.float32`. cu_seqlen_ks: Start indices (inclusive) for valid K per query position, shape [M], dtype int32. cu_seqlen_ke: End indices (exclusive) for valid K per query position, shape [M], dtype int32. clean_logits: Whether to clean the unfilled logits into `-inf`. Returns: Logits tensor of shape [M, N], dtype `torch.float32`. Nrg)r/r8r5)rbrcrdrerfrgrrrrCs  rC context_lens block_sizenum_smscCs ttdur tSt|||S)aBuild scheduling metadata for paged MQA logits. Args: context_lens: Tensor of shape [B], dtype int32; effective context length per batch element. block_size: KV-cache block size in tokens (e.g., 64). num_sms: Number of SMs available. 132 for Hopper Returns: Backend-specific tensor consumed by `fp8_paged_mqa_logits` to schedule work across SMs. N)r/r:r5)rirjrkrrrrEs rEq_fp8 kv_cache_fp8 block_tablesschedule_metadata max_model_lenc Cs,ttdur tSt||||||||dS)aOCompute FP8 MQA logits using paged KV-cache. Args: q_fp8: Query tensor of shape [B, next_n, H, D]. Casted to `torch.float8_e4m3fn` by caller. kv_cache_fp8: Paged KV-cache in packed FP8+scale layout with shape [num_blocks, block_size, 1, D+4], dtype `torch.uint8`. The last 4 bytes per (block,pos) store the `float` dequant scale. weights: Tensor of shape [B * next_n, H], dtype `torch.float32`. context_lens: Tensor of shape [B], dtype int32; effective context length for each batch element. block_tables: Tensor of shape [B, max_blocks], dtype int32; maps logical block indices to physical blocks in the paged cache. schedule_metadata: Returned by `get_paged_mqa_logits_metadata`; used to distribute work across SMs. max_model_len: Maximum sequence length used to size the logits output. clean_logits: Whether to clean the unfilled logits into `-inf`. Returns: Logits tensor of shape [B * next_n, max_model_len], dtype `torch.float32`. Nrh)r/r9r5)rlrmrdrirnrorprgrrrrD(s rDc Cstdtt|S)Ng@)torchpowceillog2absrXrrr_ceil_to_ue8m0WsrvycCst|||SrUr )rWrwrrr_align[rx element_sizecCst|d|S)N)rx)rWrzrrrget_tma_aligned_size`ryr|T)dynamicbackendFr]cCst}|dks J|j\}}|\}}tjt||t||f|j|jd}||d|d|f<| d|| d||} | j dddd} t\} } | | } |r[t| n| } | d | |}||d|d|f| | d | dfS) Nr)dtypedevicer )r T)dimkeepdimg-C6?g?r)r fp8_dtypershaperqzerosrxrrviewsizerufloatamaxclamprrvtoview_as contiguous)rWrjr]rmnblock_mblock_nx_paddedx_viewx_amaxr2fp8_maxsfx_scaledrrrper_block_cast_to_fp8hs"   rcCsB||}}||||}d|||}d|S)a|Return a global difference metric for unit tests. DeepGEMM kernels on Blackwell/B200 currently exhibit noticeable per-element error, causing `torch.testing.assert_close` to fail. Instead of checking every element, we compute a cosine-style similarity over the whole tensor and report `1 - sim`. Once kernel accuracy improves this helper can be removed. rr )doublesum)rWrw denominatorsimrrr calc_diffs r output_dtypeweightsupports_deep_gemmcCsH|durt}d}d}|o#|tjko#|jd|dko#|jd|dkS)N@r}rr )rrqbfloat16r)rrr N_MULTIPLE K_MULTIPLErrr"should_use_deepgemm_for_fp8_linearsr)rr r@rArBrCrDrErr1rrTrrYrGr"rU)B__doc__ functoolsrOrIcollections.abcrenumrtypingrrrq vllm.envsr vllm.loggerr9vllm.model_executor.layers.quantization.utils.quant_utilsrvllm.platformsrvllm.utils.import_utilsr vllm.utils.math_utilsr r cacheboolrr1r5r__annotations__r6r7r8r9r:r;r<r=r/rSrTlistrGTensorrYr@rArBrHtuplerCrErDrvrxr|DEFAULT_BLOCK_SIZEcompilesimple_compile_backendrrrr__all__rrrrs        )  5     "  /