o ۾i9 @sJddlZddlmZddlmZddlmZddlmZddl m Z ddl m Z ddl mZmZe eZGdd d ejZejd d d ejd ejfddZd ejdejdBdejdBd ejfddZd ejdejd ejfddZdejdeeejfd ejfddZd ejdejdBdejdBdeeejfd ejf ddZddZdS)N)version)envs)rocm_aiter_ops) LogprobsMode) init_logger) CpuArchEnumcurrent_platformc s|eZdZdZddeddffdd Zdejd ee ej fd ejdBd ejdBde ejejdBff d d Z dejd ee ej fd ejdBd ejdBde ejejdBff ddZ dejd ee ej fd ejdBd ejdBde ejejdBff ddZdejd ee ej fd ejdBd ejdBde ejejdBff ddZdejd ejdBd ejdBd ee ej fdejf ddZZS)TopKTopPSamplerz Module that performs optional top-k and top-p filtering followed by weighted random sampling of logits. Implementations may update the logits tensor in-place. raw_logprobs logprobs_modereturnNcs8t||_|dvrJtrJtjr@ddlm}t }|dus#J| |s4| }t d|dt jddd|j|_nWt d |j|_nMtrdt}|tjtjfvr_|j|_n8|j|_n3|dvrtrzddl}tjj|_t d |j |_Wnt!yt "d |j|_Ynw|j|_t#|_#dS) Nprocessed_logitsprocessed_logprobsr)FlashInferBackendz/FlashInfer does not support compute capability z&, unset VLLM_USE_FLASHINFER_SAMPLER=1.z,Using FlashInfer for top-p & top-k sampling.global)scopezFlashInfer top-p/top-k sampling is available but disabled by default. Set VLLM_USE_FLASHINFER_SAMPLER=1 to opt in after verifying accuracy for your workloads.z9Using aiter sampler on ROCm (lazy import, sampling-only).z[aiter.ops.sampling is not available on ROCm. Falling back to forward_native implementation.)$super__init__r ris_cudarVLLM_USE_FLASHINFER_SAMPLER%vllm.v1.attention.backends.flashinferrget_device_capabilitysupports_compute_capabilityas_version_str RuntimeErrorlogger info_once forward_cudaforward debug_onceforward_nativeis_cpuget_cpu_architecturerRISCVPOWERPC forward_cpur is_enabledaiter.ops.samplingtorchopsaiter aiter_ops forward_hip ImportError warning_onceapply_top_k_top_p)selfr r capabilitycapability_strarchr+ __class__X/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/sample/ops/topk_topp_sampler.pyrs`             zTopKTopPSampler.__init__logits generatorskpcCsZ||||}d}|jdkr|}n |jdkr|jdtjd}|jdtjd}t|||fS)z PyTorch-native implementation of top-k and top-p sampling. The logits tensor may be updated in-place. Nrrdimdtype)r0r log_softmaxr)float32softmax random_sample)r1r9r:r;r<logits_to_returnprobsr7r7r8r!\s   zTopKTopPSampler.forward_nativecCsZ|dur|dus |r|rtd|||||S|jdvs"Jdt||||dfS)z;More optimized implementation for top-k and top-p sampling.NziFlashInfer 0.2.3+ does not support per-request generators. Falling back to PyTorch-native implementation.r z5FlashInfer does not support returning logits/logprobs)rr r!r flashinfer_sample contiguous)r1r9r:r;r<r7r7r8rqs  zTopKTopPSampler.forward_cudac Cs||||}d}|jdkr|}n |jdkr|jdtjd}t||jdkr-t||fS|jdtjd}t |}| | D] \}} ||j | dqB| |j ddd|fS) z PyTorch-native implementation of top-k and top-p sampling for CPU. The logits tensor may be updated in-place. Nrrr=r>r generatorr?)r0r rAr)rBlenshapecompiled_random_samplerC empty_like exponential_itemsdiv_argmaxview) r1r9r:r;r<rErFqirJr7r7r8r&s     zTopKTopPSampler.forward_cpucCsrd} |dur |dus |r|rtd|||||S|jdvs%Jd|r/|||||S|||||dfS)NTzVaiter sampler does not support per-request generators; falling back to PyTorch-native.r z9aiter sampler does not support returning logits/logprobs.)rr/r!r aiter_sample)r1r9r:r;r<DISABLE_AITER_SAMPLERr7r7r8r-s zTopKTopPSampler.forward_hipc Cs|du}|du}|r/|r/|jdtjd}|jj|dgt|t|Rddi}|dS|rP|jdtjd}|jj|dgt|Rddi}|dS|rr|jdtjd}|jj |gt|R} tj | dddSt d) z#Sample from logits using aiter ops.Nr=r> deterministicT) num_samplesz6aiter_sample was called with no active top-k or top-p.) rCr)rBrHr,top_k_top_p_sampling_from_probs_to_tensor_scalar_tuplerTtop_p_sampling_from_probstop_k_renorm_probs multinomialr) r1r9r;r<r: use_top_k use_top_prFnext_token_ids renorm_probsr7r7r8rWsD  zTopKTopPSampler.aiter_sample)r )__name__ __module__ __qualname____doc__rrr)Tensordictint Generatortupler!rr&r-rW __classcell__r7r7r5r8r stB         r T)dynamicr9r cCs:|jdtjd}t|}|||jdddS)Nr=r>rK)rCr)rBrOrPdivrSrT)r9rFrUr7r7r8rNs rNr;r<c Cs|dur|dur |St||S|jddd\}}|dur>|d|tj}|d|jdd}||k}||t d |durk|j dd}tj |d|d}|d|jddk}d|dddf<||t d |j d||d }|S) zApply top-k and top-p masks to the logits. If a top-p is used, this function will sort the logits tensor, which can be slow for large batches. The logits tensor may be updated in-place. Nr=F)r? descendingrZrKinf)r?out)r?indexsrc) apply_top_k_onlysortsizetor)longgather unsqueeze masked_fill_floatrCcumsumscatter) r9r;r< logits_sort logits_idx top_k_mask probs_sort probs_sum top_p_maskr7r7r8r0s$   r0cCs||jdk}||d}|}|dd}|j|ddjd|}| |dt d | ||kt d |S)z Apply top-k mask to the logits. This implementation doesn't involve sorting the entire vocab. The logits tensor may be updated in-place. rZrKrr) rM masked_fillmaxsub_r|topkvaluesr{rzr}r~)r9r; no_top_k_mask max_top_kk_indexrr7r7r8rvs rvrFr:cCsbt|}t||jdkr||r%|D] \}}||j|dq||jdddS)zRandomly sample from the probabilities. We use this function instead of torch.multinomial because torch.multinomial causes CPU-GPU synchronization. rrIr=rK) r)rOrLrMrPrQrRrSrT)rFr:rUrVrJr7r7r8rD7s rDcCsddl}t|jtdkrtd|dur|durJ|dur3|jdtjd}|jj ||dd}n |durI|jdtjd}|jj ||dd}n |jj |||dd}| dS) abSample from the logits using FlashInfer. Statistically, this function is equivalent to the `random_sample` function. However, this function is faster because it avoids sorting the logits tensor via rejection sampling. NOTE: The outputs of this function do not necessarily match the outputs of the `random_sample` function. It only guarantees that the outputs are statistically equivalent. NOTE: This function includes CPU-GPU synchronization, while `random_sample` does not. Call this function at the end of the forward pass to minimize the synchronization overhead. rNz0.2.3zCFlashInfer version >= 0.2.3 required for top-k and top-p sampling. r=r>T)rY) flashinferrparse __version__r.rCr)rBsamplingr^top_k_sampling_from_probs top_k_top_p_sampling_from_logitsrT)r9r;r<r:rrFrcr7r7r8rGOs( rGcCst|tjr |dfSd|fS)Nr) isinstancer)ri)xr7r7r8r]s r])r)torch.nnnn packagingrvllmrvllm._aiter_opsrvllm.config.modelr vllm.loggerrvllm.platformsrrrerModuler compilerirNr0rvrjrkrlrDrGr]r7r7r7r8s`       Z +     1