o i; @sddlZddlmZmZddlmZmZddlmZdZ ej ddZ ej dej fd d Z d ed ejd ejdededef ddZdedefddZdededejdejdef ddZej dej fddZdS)N) VllmConfigreplace)tltriton)CommonAttentionMetadatacCstjdd}||kr dSt||}d}|dkr|}n t||d} || }t||} |d| } t|dk| d} t||dd} | | } t||| t||| dS)a Fused kernel for Eagle prepare_input_padded. This kernel computes the token index to sample for each request, taking into account the number of draft tokens and the number of valid sampled tokens (which is one more than the number of accepted tokens). raxisN)r program_idloadwherestore)cu_num_draft_tokens_ptrvalid_sampled_tokens_count_ptrquery_start_loc_gpu_ptrtoken_indices_to_sample_ptrnum_rejected_tokens_gpu_ptrnum_reqsreq_idx cu_draft_currnum_draft_tokens cu_draft_prev valid_countnum_rejected_tokensq_last_tok_idxindex_to_samplerO/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm/v1/spec_decode/utils.py"eagle_prepare_inputs_padded_kernels  rBLOCK_SIZE_TOKENSc Cs8tjdd} | |kr dSt|| } | r7t|| } tjddtjd} t|| | t|| | dStd| }||k}|| |}tj|||dd}|dk||k@|@}t|} | dkrtt ||d}tt ||k|d}t|| |nt|| } t|| | t|| | dS)a Fused kernel for Eagle prepare_next_token_ids_padded. This kernel computes the number of valid (1 + accepted) tokens for each request, and the corresponding "next" token id to sample from during speculative decoding. This is the "last accepted token" from the sampled tokens, or the backup token if no tokens were accepted or if the request is marked as discarded. rrNrdtypermaskother) rr r fulluint32rarangesummaxr )sampled_token_ids_ptrdiscard_request_mask_ptrbackup_next_token_ids_ptrnext_token_ids_ptrr vocab_sizenum_sampled_tokens_per_reqrstride_sampled_token_idsr r is_discarded backup_tokenr token_offs token_maskrow_ptr token_ids is_valid_masklast_valid_indexlast_valid_tokenrrr&eagle_prepare_next_token_padded_kernel9s0    r;cad new_positionsis_rejected_token_mask block_sizenum_new_tokens max_model_lencCs|jj\}}tj||jjd}tj|||t|d}tj ||dd} ||| |} |j d| } | |} | || } ||k}| |t | |t | S)N)device) output_sizer )r*r) block_table_tensorshapetorchr(query_start_locrBrepeat_interleavenaive_query_lenslenclampview masked_fill_PADDING_SLOT_ID)r<r=r>r?r@rA batch_sizen_blocks_per_req req_indicesclamped_positionsblock_table_indices block_nums block_offsetsnew_slot_mappingexceeds_max_model_lenrrrcompute_new_slot_mappingxs"     rXtarget_model_vllm_configreturncCsD|}|jdus Jd|j}t|j|jjd}t|d||jd}|S)anThe vllm_config is configured for the target model, e.g. its quant_config and parallel_config. But the draft model is potentially quantized differently, and has potentially different tensor_parallel_size. This function creates a new vllm_config configured for the drafter. The vllm_config is useful when loading the draft model with get_model(). Nzspeculative_config is not set)rank) quant_configparallel_config model_config)speculative_configrdraft_parallel_configr]r[draft_model_config)rYoldold_spec_confignew_parallel_confignewrrr"create_vllm_config_for_draft_models  rfcommon_attn_metadataNr(rVc Cs||}|j||dt|j}|j|tjt|jtjd}|j|||j||j| ||j ||j ||d}|S)ah Creates a new CommonAttentionMetadata with all query lengths increased by N. Also all seq lens are increased by N. This is useful e.g. in speculative decoding with parallel drafting, where we extend each sequence by N tokens and predict all tokens in one pass. The slot mapping is computed externally, as it requires more information. Nr!)rGquery_start_loc_cpuseq_lensnum_actual_tokens max_query_len max_seq_len slot_mapping) rGrJrirFr(int32rrjrkrOrlrm)rgrhr(rVr<new_query_start_locnew_query_start_loc_cpunew_cadrrrextend_all_queries_by_Ns     rsc0CsTtjdd}tjdd}t| |}t| |d}t| |}|r4||}d}|||d}n||d}d}|||}||d}|||}||td|}||k}||k}||k}||k|||k@}|||k} ||}!|||}"t|"| d}#tj||#||@dd}$t||}%t||}&t||&|$}$t|| |$}$t| | |$}$|%|}'t| d|'}'| |@}(||@})||k|||k@}*||}+|||+},|r||}-||-k}.||}/tj||/|!|.dtj||!|$|dtj||!|'|dtj||!|(|dtj||!|)|dtj||,|!|*|@ddS)z Copy and expand inputs from the target model to the drafting buffers for Eagle speculative decoding. This kernel handles padding slots and parallel drafting tokens, if enabled. rrr r#)r$N)rr r r(minimumr r)0target_token_ids_ptrtarget_positions_ptrr.out_input_ids_ptrout_positions_ptrout_is_rejected_token_mask_ptrout_is_masked_token_mask_ptrout_new_token_indices_ptrout_hidden_state_mapping_ptrquery_start_loc_ptrquery_end_loc_ptrpadding_token_idparallel_drafting_token_idtotal_input_tokensnum_padding_slots_per_requestshift_input_idsr request_idxtoken_batch_idxrGnext_query_start_loc query_end_locnum_valid_tokens input_offset output_start num_rejectedtotal_output_tokensj in_boundsis_valid_regionis_bonus_regionis_parallel_draft_regionis_rejected_regionout_idxin_idxin_idx_clampedr7 start_pos bonus_token positionsis_rejected_out is_masked_outis_new_token_regionnew_token_local_idxnew_token_out_idxnum_input_tokens_this_requestis_input_regionsrc_idxrrr#copy_and_expand_eagle_inputs_kernels~             r)rF vllm.configrrvllm.triton_utilsrr vllm.v1.attention.backends.utilsrrNjitr constexprr;TensorintrXrfrsrrrrrsX  * >    "