o .i9@svUdZddlZddlZddlZddlmZddlmZddlm Z m Z ddl Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZGd d d eZejd efddZejd efddZde de d e fddZdaede fdBed<da ede fdBed<da!ede fdBed<da"ede fdBed<da#ede fdBed<da$ede fdBed<da%ede fdBed<da&ede fdBed<da'ede fdBed<d[d d!Z(d e)fd"d#Z*ejd e+e)fd$d%Z,d&e j-d e j-fd'd(Z.d)d*Z/d+d,Z0d-d.Z1d/d0Z2d1e j-d2e3e j-e j-fd3e j-d4e j-d5e j-d e j-f d6d7Z4d8e j-d9e)d:e)d e j-fd;d<Z5d=e j-d>e j-d3e j-d8e j-d?e j-d@e j-dAe)d e j-fdBdCZ6d&e j-fdDdEZ7d&e)dFe)d e)fdGdHZ8d&e)dIe)fdJdKZ9dLdLgZ:e j;dMej d\dUe j?dVe j-dWedBfdXdYZ@gdZZAdS)]zmCompatibility wrapper for DeepGEMM API changes. Users of vLLM should always import **only** these wrappers. N)Callable)Enum)AnyNoReturn)logger)get_fp8_min_max)current_platform) has_deep_gemmcdivc@s4eZdZdZdZdZed ddZed dd ZdS) DeepGemmQuantScaleFMTrreturnNcCs^t|dd}|dur dStjototdu}|s|j|_dStdr)|j |_dS|j |_dS)z>Initialize the oracle decision and store it in the class cache _oracle_cacheNd) getattrenvsVLLM_USE_DEEP_GEMM_E8M0is_deep_gemm_supported_fp8_gemm_nt_implFLOAT32rris_device_capability_familyUE8M0FLOAT32_CEIL_UE8M0)clscacheduse_e8m0rQ/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/utils/deep_gemm.pyinit_oracle_cache&s z'DeepGemmQuantScaleFMT.init_oracle_cachecCs t|dd}|dusJd|S)z*Return the pre-initialized oracle decisionrNz2DeepGemmQuantScaleFMT oracle cache not initialized)r)rrrrr from_oracle<s z!DeepGemmQuantScaleFMT.from_oraclerN)rr ) __name__ __module__ __qualname__rrr classmethodr r!rrrrr s r rcCs,to tdp td}tjoto|S)zReturn `True` if DeepGEMM is supported on the current platform. Currently, only Hopper and Blackwell GPUs are supported. Zr)ris_cudais_device_capabilityrrVLLM_USE_DEEP_GEMMr )is_supported_archrrrrDs  rcCsRts tddSttdurtddStjr"tddStddS)znReturn `True` if vLLM is configured to use DeepGEMM " "E8M0 scale on a Hopper or Blackwell-class GPU. z>DeepGEMM E8M0 disabled: DeepGEMM not supported on this system.FNz3DeepGEMM E8M0 disabled: _fp8_gemm_nt_impl not foundz*DeepGEMM E8M0 enabled on current platform.Tz0DeepGEMM E8M0 disabled on current configuration.)rr debug_once _lazy_initr info_oncerrrrrris_deep_gemm_e8m0_usedPs   r/___cOstd)z-Placeholder for unavailable DeepGEMM backend.zDeepGEMM backend is not available or outdated. Please install or update the `deep_gemm` to a newer version to enable FP8 kernels.) RuntimeError)r0r1rrr_missingisr3.r _grouped_impl_grouped_masked_impl_fp8_mqa_logits_impl_fp8_paged_mqa_logits_impl#_get_paged_mqa_logits_metadata_impl%_get_mn_major_tma_aligned_tensor_impl,_get_mk_alignment_for_contiguous_layout_impl'_transform_sf_into_required_layout_implcCstdus tdus tdus tdus tdus tdus tdus tdur"dSts'dSd}t j |ds;t j tjdt j |<td}t|ddat|ddat|ddat|ddat|ddat|d dat|d dat|d dat|d datdS) z2Import deep_gemm and resolve symbols on first use.NDG_JIT_CACHE_DIR deep_gemm fp8_gemm_nt m_grouped_fp8_gemm_nt_contiguousfp8_m_grouped_gemm_nt_maskedfp8_mqa_logitsfp8_paged_mqa_logitsget_paged_mqa_logits_metadataget_mn_major_tma_aligned_tensor&get_mk_alignment_for_contiguous_layout!transform_sf_into_required_layout)rr4r5r6r7r8r:r;r osenvirongetpathjoinrVLLM_CACHE_ROOT importlib import_modulerr9r r )DEEP_GEMM_JIT_CACHE_ENV_NAME_dgrrrr-|sF        r-cCsttd}t|S)Nr=)r-rMrNint get_num_sms)rPrrrrRs  rRcCs"ttdur tSt}||gSN)r-r:r3) mk_align_sizerrrrEs rExcCsttdur tSt|S)z6Wrapper for DeepGEMM's get_mn_major_tma_aligned_tensorN)r-r9r3rUrrr get_col_major_tma_aligned_tensorsrWcOsNttdurt|i|Sd|vr|d}|d=nt}t|d| i|S)Nr/disable_ue8m0_cast)r-rr3r/)argskwargs use_ue8m0rrrr>sr>cO2ttdurt|i|St|dt i|SNrX)r-r4r3r/rYrZrrrr?r?cOr\r])r-r5r3r/r^rrrr@r_r@cOr\r])r-r;r3r/r^rrrrFr_rFqkvweights cu_seqlen_ks cu_seqlen_kecCs$ttdur tSt|||||S)aCompute FP8 MQA logits for a single sequence without KV paging. Args: q: Query tensor of shape [M, H, D]. Casted to `torch.float8_e4m3fn` by caller. kv: Tuple `(k_fp8, k_scales)` where `k_fp8` has shape [N, D] with dtype `torch.float8_e4m3fn` and `k_scales` has shape [N]) with dtype `torch.float32`. weights: weights of shape [M, H], dtype `torch.float32`. cu_seqlen_ks: Start indices (inclusive) for valid K per query position, shape [M], dtype int32. cu_seqlen_ke: End indices (exclusive) for valid K per query position, shape [M], dtype int32. Returns: Logits tensor of shape [M, N], dtype `torch.float32`. N)r-r6r3)r`rarbrcrdrrrrAsrA context_lens block_sizenum_smscCs ttdur tSt|||S)aBuild scheduling metadata for paged MQA logits. Args: context_lens: Tensor of shape [B], dtype int32; effective context length per batch element. block_size: KV-cache block size in tokens (e.g., 64). num_sms: Number of SMs available. 132 for Hopper Returns: Backend-specific tensor consumed by `fp8_paged_mqa_logits` to schedule work across SMs. N)r-r8r3)rerfrgrrrrC s rCq_fp8 kv_cache_fp8 block_tablesschedule_metadata max_model_lenc Cs,ttdur tSt|||||||ddS)aCompute FP8 MQA logits using paged KV-cache. Args: q_fp8: Query tensor of shape [B, next_n, H, D]. Casted to `torch.float8_e4m3fn` by caller. kv_cache_fp8: Paged KV-cache in packed FP8+scale layout with shape [num_blocks, block_size, 1, D+4], dtype `torch.uint8`. The last 4 bytes per (block,pos) store the `float` dequant scale. weights: Tensor of shape [B * next_n, H], dtype `torch.float32`. context_lens: Tensor of shape [B], dtype int32; effective context length for each batch element. block_tables: Tensor of shape [B, max_blocks], dtype int32; maps logical block indices to physical blocks in the paged cache. schedule_metadata: Returned by `get_paged_mqa_logits_metadata`; used to distribute work across SMs. max_model_len: Maximum sequence length used to size the logits output. Returns: Logits tensor of shape [B * next_n, max_model_len], dtype `torch.float32`. NT) clean_logits)r-r7r3)rhrirbrerjrkrlrrrrB"srBc Cstdtt|S)Ng@)torchpowceillog2absrVrrr_ceil_to_ue8m0OsrsycCst|||SrSr )rUrtrrr_alignSru element_sizecCst|d|S)N)ru)rUrwrrrget_tma_aligned_sizeXrvryT)dynamicbackendFr[cCst}|dks J|j\}}|\}}tjt||t||f|j|jd}||d|d|f<| d|| d||} | j dddd} t\} } | | } |r[t| n| } | d | |}||d|d|f| | d | dfS) Nr)dtypedevicer )r T)dimkeepdimg-C6?g?r)r fp8_dtypershapernzerosrur}r~viewsizerrfloatamaxclamprrstoview_as contiguous)rUrfr[rmnblock_mblock_nx_paddedx_viewx_amaxr0fp8_maxsfx_scaledrrrper_block_cast_to_fp8`s"   rcCsB||}}||||}d|||}d|S)a|Return a global difference metric for unit tests. DeepGEMM kernels on Blackwell/B200 currently exhibit noticeable per-element error, causing `torch.testing.assert_close` to fail. Instead of checking every element, we compute a cosine-style similarity over the whole tensor and report `1 - sim`. Once kernel accuracy improves this helper can be removed. rr )doublesum)rUrt denominatorsimrrr calc_diffws r output_dtypeweightsupports_deep_gemmcCsH|durt}d}d}|o#|tjko#|jd|dko#|jd|dkS)N@rzrr )rrnbfloat16r)rrr N_MULTIPLE K_MULTIPLErrr"should_use_deepgemm_for_fp8_linearsr)rr r>r?r@rArBrCrr/rrRrrWrEr"rS)B__doc__ functoolsrMrGcollections.abcrenumrtypingrrrn vllm.envsr vllm.loggerr9vllm.model_executor.layers.quantization.utils.quant_utilsrvllm.platformsrvllm.utils.import_utilsr vllm.utils.math_utilsr r cacheboolrr/r3r__annotations__r4r5r6r7r8r9r:r;r-rQrRlistrETensorrWr>r?r@rFtuplerArCrBrsruryDEFAULT_BLOCK_SIZEcompilesimple_compile_backendrrr}r__all__rrrrs        )  5       -