o پiw@sUddlZddlmZmZmZmZmZddlZddlm Z ddlm Z ddl m Z ddlmZmZddlmZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZm Z m!Z!m"Z"e!rrdd l#m$Z$m%Z%m&Z&m'Z'e"ryddl(Z(e)e*Z+e dZ,e dZ-iZ.ee/egdffe0d<hdZ1Gddde j2Z3de/degdfddfddZ4d9dee/ddfddZ5dej6dej6dej6dej6de7d eej6d!ej6fd"d#Z8d$ej6dej6dej6dej6de7f d%d&Z9ej:d'd(d)ej6d*ej6d!ej6dej6fd+d,Z;  d:dej6d eej6d!eej6fd-d.Zdeeeffd2d3Z? 4d;d$ej6d5ed6e>fd7d8Z@dS)<N)CallableDictListOptionalTuple)nn) get_tp_group)get_attention_tp_groupis_dp_attention_enabled)LogitsProcessorOutput) murmur_hash32)get_token_ids_logprobsget_top_logprobs)SamplingBatchInfo) TOP_K_ALL)get_global_server_args)crash_on_warningsget_bool_env_varis_cudais_npu)min_p_sampling_from_probstop_k_renorm_probtop_k_top_p_sampling_from_probstop_p_renorm_probSYNC_TOKEN_IDS_ACROSS_TPSGLANG_RETURN_ORIGINAL_LOGPROBSampler_CUSTOM_SAMPLER_FACTORIES>ascendpytorch flashinferc sleZdZfddZdejdedejfddZdeded e d e e d e e e d ejf d dZ dejded ejde dejf ddZ dejded ejdejfddZdejdede dejfddZdejdede d e deejeejff ddZdedejd e e d e e e dedejf ddZdejdefddZdeded e d e e d e e e ddf d d!ZZS)"rcs`ttj|_tj|_trt j|_tj |_ tj |_ |j du|_ tjdk|_dS)Nr)super__init__renable_nan_detectionuse_nan_detectionr device_group tp_sync_groupr r rl_on_policy_targetenable_deterministic_inferenceenable_deterministicuse_log_softmax_logprobsampling_backenduse_ascend_backend)self __class__M/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/layers/sampler.pyr")s      zSampler.__init__logits sampling_inforeturncCs^|jrt|||jr-tt|r-tdtt|t |d|}t r-t d|S)z7Apply custom logit processors and handle NaN detection.z3Detected errors during sampling! NaN in the logits.gj) has_custom_logit_processorapply_custom_logit_processorr$torchanyisnanloggerwarningwhere full_liker ValueError)r-r2r3r0r0r1_preprocess_logits9s  zSampler._preprocess_logits logits_outputreturn_logprobtop_logprobs_numstoken_ids_logprobs positionscCsn|j}|||}|jr t|d}|rtjjj|dd} } n|j o+|j o+|j } |r7t r7tj|dd} d} |j durP| |j } tj| dd} ~ |jr^|||| |\}} nA|jrt|jrt| rt|| ||}|rst ss| } n+||jtj|dd|dd<|}||||| }|rt s| dur| nt|} ~|rt r| } ||| ||||||||S)aRun a sampler & compute logprobs and update logits_output accordingly. Args: logits_output: The logits from the model forward sampling_info: Metadata for sampling return_logprob: If set, store the output logprob information to logits_output top_logprobs_nums: Number of top lobprobs per sequence in a batch token_ids_logprobs: Per-sequence list of specific token IDs to retrieve logprobs for. Each element is a list of token IDs (or None) for one sequence in the batch. This is used in speculative decoding. positions: The positions of the tokens in the sequence. Used for deterministic sampling to get the unique seed for each position. dimN)next_token_logitsr? is_all_greedyr7argmaxr functional log_softmaxneed_top_p_samplingneed_top_k_samplingneed_min_p_samplingrr'bfloat16div temperaturesr,_forward_ascend_backendr*r)_sample_from_logprobsdiv_softmax_sample_from_probslog_attach_logprobs_to_output_sync_token_ids_across_tp)r-r@r3rArBrCrDr2batch_next_token_idsoriginal_logprobslogprobssimple_sampling_caselogprobs_via_logsoftmax_kernellogits_div_temperatureprobsr0r0r1forwardLs       zSampler.forwardrar^cCs|r t||j|d}|Stj}|dkrD|jdusJd|jr4t||j}t||j}t ||j }|St | |j|jd|j d}|S|dkrYt||j|j|j |j|j|}|Std|) zSample from probability distribution (after softmax). Used for standard sampling with flashinfer/pytorch backends. Handles both simple (direct multinomial) and complex (top-k/top-p/min-p) cases. ) sampling_seedrDr Nz5Sampling seed is not supported for flashinfer backendjoint)filter_apply_order check_nanrzInvalid sampling backend: )sampling_from_probs_torchrcrr+rOrtop_ksrtop_psrmin_psr contiguousr$+top_k_top_p_min_p_sampling_from_probs_torchr>)r-rar3rDr^r[backendr0r0r1rWsN &    zSampler._sample_from_probsr]cCs2|jdus Jdt||j|}|dtjS)zSample from log-probabilities using the Gumbel trick. Used for deterministic sampling with simple cases (no top-k/top-p/min-p). Requires sampling_seed to be set in sampling_info. Nz4sampling_seed is required for sampling from logprobsrE)rcmultinomial_with_seedviewtor7int32)r-r]r3rD sampled_indexr0r0r1rTs zSampler._sample_from_logprobscCsd|rtj|dd}tj|ddd}|tjS|js Jdt||j|j |j |j }|tjS)zSample from temperature-scaled logits without softmax. Used for the Ascend NPU backend which handles softmax internally. rErF num_samplesz1Only ascend backend supports sampling from logits) r7rV multinomialrorprqr,-top_k_top_p_min_p_sampling_from_logits_ascendrhrirjrO)r-r2r3r^rar[r0r0r1_sample_from_logitss    zSampler._sample_from_logitscCs<||j||||}d}|rtstj|dd}||fS)amHandle the full Ascend backend sampling path. Ascend backend has fused kernels that handle softmax internally, so we sample directly from temperature-scaled logits. Returns: A tuple of (batch_next_token_ids, logprobs). logprobs is None when return_logprob is False or SGLANG_RETURN_ORIGINAL_LOGPROB is set. NrErF)rUrRrxrr7rL)r-r2r3r^rAr[r]r0r0r1rSs zSampler._forward_ascend_backendr[cCs|jt|jjdtdd|Drt||\|_|_tdd|Dr/t ||\|_ |_ |tj t ||jd|f|_dS)Nmincs|]}|dkVqdSrNr0.0xr0r0r1 Ez5Sampler._attach_logprobs_to_output..css|]}|duVqdSNr0r}r0r0r1rKrdevice)clamp_r7finfodtyperzr8rnext_token_top_logprobs_valnext_token_top_logprobs_idxr !next_token_token_ids_logprobs_val!next_token_token_ids_logprobs_idxarangelenrnext_token_logprobs)r-r@r]rBrCr3r[r0r0r1rY8s   z"Sampler._attach_logprobs_to_outputcCs*ts|jrtjj|tjj|jddSdS)N)opgroup) rgrammarsr7 distributed all_reducedistReduceOpMINr&)r-r[r3r0r0r1rZVs  z!Sampler._sync_token_ids_across_tpNc Cs|jdur tddStdd|D}tdd|D}|s$|s$dS||j|}tjjj|dd} |r?t | |\|_ |_ |rLt | |\|_ |_dSdS)z Compute logprobs for requested token IDs without performing sampling. Optimized for prefill-only scoring requests that need token probabilities but don't require next token generation. Nz+No logits available for logprob computationcss$|] }|duo t|dkVqdS)Nrrr~ token_idsr0r0r1r{s  z0Sampler.compute_logprobs_only..csr{r|r0r}r0r0r1rrrErF)rHr:r;r8r?r7rrKrLrrr&get_token_ids_logprobs_batch_optimizedrr) r-r@r3rArBrCneeds_token_ids_logprobsneeds_top_logprobsr2r]r0r0r1compute_logprobs_onlygs,  zSampler.compute_logprobs_only)__name__ __module__ __qualname__r"r7Tensorrr?r boolrintrbrWrTrxrrrSrYrZr __classcell__r0r0r.r1r(s    p 5       rmfactoryr4cCsB|stdddlm}|tvrtd||||t|<dS)z7Register a custom sampler factory for a backend string.z"backend must be a non-empty stringr)SAMPLING_BACKEND_CHOICESz4Overriding existing sampler factory for backend '%s'N)r>sglang.srt.server_argsrrr:r;add)rmrrr0r0r1register_sampler_backends    rcCsnt}|p |r |jnd}|tvr$t|}t|ts"td|d|S|dus,|tvr/tStd|d)z7Create a sampler honoring custom backend registrations.Nz$Custom sampler factory for backend 'z' must return a SamplerzUnknown sampling backend 'z.'. Register it via register_sampler_backend().)rr+r isinstancer TypeError_BUILT_IN_SAMPLING_BACKENDSr>)rm server_argssamplerr0r0r1create_samplers    rrarhrirjrOrcrDcCs|jddd\}}tj|dd} d|tjd|jd|jddd|ddk<d|| ||ddk<|rR|d us>Jd |d d df|} d||| ddk<|d ur^tj|dd } n|tj } ~| t | ||} |tj }tj |d| d d} | S) z A top-k, top-p and min-p sampling implementation with native pytorch operations. When sampling_seed is not None, deterministic inference will be enabled, it will sample with the sampling_seed of each request. rETrG descendingrFrrrsNzDWith sampling seed, multinomial_with_seed will provide wrong resultsrtrGindex)sortr7cumsumrshaperrorvrpfloat64log_rnrqgather)rarhrirjrOrcrD probs_sort probs_idx probs_summin_p_thresholdsrrr]r[r0r0r1rls0     rlr2cCsttdr=t|dk|dk@r=t|||}|jdd}|r5|jdd|}||ddk}||dtj |dd} ntj|dd} | j ddd \} } |t k} || | j dtj d | j d| jd dd|ddk}| |dtj| dd}|| |ddk}| |d|r| d d d f|}| |ddk}| |dtj | dd}| tj} tj| d|d } | dS)zA top-k, top-p and min-p sampling implementation for ascend npu with torch_npu interface. Takes temperature-scaled logits as input (softmax is applied internally). npu_top_k_top_pirsrErFrrtTrrrNr)hasattr torch_npur7allrrVmaxro masked_fill_rvrrrrrrrprqr)r2rhrirjrOlogits_top_k_top_pprobs_top_k_top_pr min_p_maskr[rarr topk_all_mask top_k_maskr top_p_maskrrr0r0r1rws>         rwT)dynamicr]seedcCs|j\}}|tj}tj||jd}t|||}|tjttj j }| j t |jjd| ||tjtj|dddS)aL Samples n elements from an input tensor `inputs` of shape (n, m) using a unique random seed for each row. This is a deterministic batched alternative to `torch.multinomial`. Args: inputs: A float tensor of shape (n, m) representing n categorical distributions with m categories each. The values are treated as weights and do not need to sum to 1. seed: An integer tensor of shape (n,) containing the random seed for each corresponding row in `inputs`. positions: The positions of the tokens in the sequence. Used for deterministic sampling to get the unique seed for each position. Returns: A tensor of shape (n,) where the i-th element is an index sampled from the distribution in `inputs[i]` using `seed[i]`. rryrsTrGkeepdim)rrpr7uint64rrr riinfouint32rrrrrrzneg_add_rJ)r]rrDnm col_indiceshashedrr0r0r1rn!s    rncCs@|dur tj|dd}n tt|||}|dtj}|S)zA sampling implementation with native pytorch operations, without top-k, top-p, or min-p filtering. Note: For deterministic sampling from logprobs, use Sampler._sample_from_logprobs instead. NrsrtrE)r7rvrnrXrorprq)rarcrDrrr[r0r0r1rgLs  rgcCs`|jddd\}}tj|dd}d||||ddk<||jdddt|d||S)NrETrrFrrsr)rr7rrorUsum zeros_likescatter_)rarirrrr0r0r1top_p_normalize_probs_torchas rrCcst|}j}tjdd|D|d}t|}|dkr/fdd|Ddd|DfSttj||d|}tjdd|D|tj d}||f}tj || dd } g} g} t |D](\} } | d ur{t| dkr{| | | | | q`| d| gq`| | fS) aK Vectorized batch processing for token ID logprobs extraction. Uses a single GPU kernel call for the entire batch instead of multiple separate calls, significantly improving performance for large batches. Args: logprobs: Log probabilities tensor [batch_size, vocab_size] token_ids_logprobs: List of token IDs to extract logprobs for Example: # Input: batch_size=3, vocab_size=5 logprobs = torch.tensor([ [-1.2, -2.1, -0.8, -3.0, -1.5], # batch 0 [-0.5, -1.8, -2.2, -1.1, -2.7], # batch 1 [-2.0, -0.9, -1.4, -2.8, -1.6], # batch 2 ]) token_ids_logprobs = [[1, 3], [2], [0, 2, 4]] # Output: # values = [tensor([-2.1, -3.0]), tensor([-2.2]), tensor([-2.0, -1.4, -1.6])] # indices = [[1, 3], [2], [0, 2, 4]] cSsg|]}t|pgqSr0rrr0r0r1 sz:get_token_ids_logprobs_batch_optimized..rrcsg|]}dqS)r) new_emptyr~_r]r0r1rcSsg|]}gqSr0r0rr0r0r1rscSsg|] }|pgD]}|qqSr0r0)r~rtoken_idr0r0r1rs)rrrFN)rrr7tensorrritemrepeat_interleaverlongsplit_with_sizestolist enumerateappendr)r]rC batch_sizer token_lengths total_tokens row_indicesrgathered_logprobssplit_logprobsoutput_token_ids_logprobs_valoutput_token_ids_logprobs_idxirr0rr1rmsB     rrssampling_batch_infonum_tokens_in_batchcs|jdt|ksJd|jddtd|djD]H\}\}}|jddd}|jdtksJJd|jdd tdt||}|||fd d |D||<td |j j d q#dS)a Apply custom logit processors to the logits. This function will modify the logits in-place. num_tokens_in_batch is needed to support spec decoding, where each batch can contain multiple tokens. By default, we assume each batch contains only 1 token. rzThe batch size of logits (z8) does not match the batch size of sampling_batch_info (z) x num_tokens_in_batch ()T)as_tuplezThe number of batch mask (z4) does not match the number of sampling_batch_info (csg|]}j|qSr0) custom_params)r~rrr0r1rrz0apply_custom_logit_processor..zCustom logit processor z is applied.N) rrcustom_logit_processoritemsnonzeror7rr:debugr/r)r2rrr processor batch_mask batch_indicesr0rr1r6s4   r6r)NN)rs)Aloggingtypingrrrrrr7torch.distributedrrrsglang.srt.distributedrsglang.srt.layers.dp_attentionr r "sglang.srt.layers.logits_processorr sglang.srt.layers.utils.hashr sglang.srt.layers.utils.logprobr r'sglang.srt.sampling.sampling_batch_infor#sglang.srt.sampling.sampling_paramsrrrsglang.srt.utils.commonrrrr sgl_kernelrrrrr getLoggerrr:rrrstr__annotations__rModulerrrrrrlrwcompilernrgrrrr6r0r0r0r1s          s 0 4 ,      [