o ۾iS4@sZddlmZddlmZmZddlmZddlmZddl m Z ee Z GdddZ dS) )product) CUDAGraphMode VllmConfig)BatchDescriptor) init_logger)get_captured_lora_countsc@seZdZdZdefddZd!ddZdeefd d Z d"d ed e de dede f ddZ de de fddZ d#de defddZ    d$d ed e de de dedee e ff ddZdeee ee ffdd ZdS)%CudagraphDispatchera Runtime cudagraph dispatcher to dispatch keys for multiple set of cudagraphs. The dispatcher stores two sets of dispatch keys, one for PIECEWISE and one for FULL cudagraph runtime mode. The keys are initialized depending on attention support and what cudagraph mode is set in CompilationConfig. The keys stored in dispatcher are the only source of truth for valid cudagraphs that can be dispatched at runtime. At runtime, the dispatch method generates the runtime cudagraph mode (FULL, PIECEWISE, or NONE for no cudagraph) and the valid key (batch descriptor) based on the input key. After dispatching (communicated via forward context), the cudagraph wrappers will trust the dispatch key to either capture or replay (if the mode matches), or pass through to the underlying runnable without cudagraph (if the mode does not match or mode is NONE). vllm_configcCs||_|j|_|jjs dnd|jjj|_tjttjti|_ |jj r>|j s>Jd|jj d|jj d|jjd|_|jjdurL|jjjnd|_tj|_ dS)NzCompilation mode should be CompilationMode.VLLM_COMPILE when cudagraph_mode piecewise cudagraphs is used, and attention should be in splitting_ops or inductor splitting should be used. cudagraph_mode=z, compilation_mode=z, splitting_ops=F)r compilation_configspeculative_confignum_speculative_tokensuniform_decode_query_lenr PIECEWISEsetFULLcudagraph_keyscudagraph_moderequires_piecewise_compilationis_attention_compiled_piecewisemode splitting_opskeys_initialized lora_configspecialize_active_loraspecialize_lora_countNONE)selfr rP/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/cudagraph_dispatcher.py__init__ s6     zCudagraphDispatcher.__init__returnNcCs|jj}|jj}dg|d|_t||dgdg|D]\}}t||D]}||kr2||j|<q&||j|<q&q|jjrd|jtj krf|jjD] }||jjkrc|j|}||krct d|d|dqGdSdSdS)z=Pre-compute the mapping from batch size to padded graph size.rr zcompile_sizes contains z which would be padded to zw. All compile_sizes must be values that won't be changed by cudagraph padding. Use values from cudagraph_capture_sizes.N) r max_cudagraph_capture_sizecudagraph_capture_sizes_bs_to_padded_graph_sizeziprange compile_sizesrrr ValueError)rmax_size capture_sizesendstartbssizepaddedrrr _compute_bs_to_padded_graph_sizeEs8         z4CudagraphDispatcher._compute_bs_to_padded_graph_sizecCsB|jj}|dur dgS|jjrt|j|j}dg|S|jdgS)z Returns list of has_lora values for CUDA graph capture. This is the single source of truth for LoRA capture cases. Nrr )r rr cudagraph_specialize_lorar max_lorasr)rrcaptured_countsrrr_get_lora_caseses  z#CudagraphDispatcher._get_lora_casesr num_tokensuniform_decodehas_loranum_active_lorasc Csf|jjj}|j}|j|}|r#|jtjr#||}||dks"Jnd}t ||}t |||||dS)NrF)r5num_reqsuniformr7r8) r scheduler_config max_num_seqsrr$rhas_moderrminr) rr5r6r7r8r<rnum_tokens_paddedr9rrr_create_padded_batch_descriptorzs   z3CudagraphDispatcher._create_padded_batch_descriptor runtime_modebatch_descriptorcCs2|tjtjfvsJd||j||dS)Nz)Invalid cudagraph runtime mode for keys: )rrrradd)rrArBrrradd_cudagraph_keysz%CudagraphDispatcher.add_cudagraph_keyr rrc s||_|tjkr d|_dS||}dd|D|_|tjkrBt|j j |D]\}}| || |d|dk| q+|tjkry|ry|jjjfdd|j j D}t||D]\}}| tj| |d|dk|qed|_dS)NTcSsg|]}|r|qSrr).0 lora_countrrr s zACudagraphDispatcher.initialize_cudagraph_keys..Frcs g|] }|kr|kr|qSrr)rExmax_num_tokensrrrrGs )rrrrr0r4captured_lora_counts mixed_moderr r#rDr@ relax_for_mixed_batch_cudagraphs decode_moderseparate_routiner r;r<)rrr lora_casesr-r8"cudagraph_capture_sizes_for_decoderrIrinitialize_cudagraph_keyssP        z-CudagraphDispatcher.initialize_cudagraph_keysF disable_fullc Cs|jr|jtjks||jjkrtjt|fS|}|r@|dkr@|jr9ddl}| |j |}|t |j kr8|j |}n|j j jd}|||||} | } |sh| |jtjvr[tj| fS| |jtjvrhtj| fS| |jtjvrutj| fStjt|fS)ah Given conditions(e.g.,batch descriptor and if using piecewise only), dispatch to a cudagraph runtime mode and the valid batch descriptor. A new batch descriptor is returned as we might dispatch a uniform batch to a graph that supports a more general batch (uniform to non-uniform). Args: num_tokens: Number of tokens in the batch. uniform_decode: Whether the batch is uniform decode (i.e. uniform and query length is uniform_decode_query_len). has_lora: Whether LoRA is active. disable_full: If True, skip FULL cudagraph checks and return PIECEWISE or NONE only. (can be used for features like cascade attention that are not supported by full cudagraphs) num_active_loras: Number of distinct active LoRA adapters. rNr )rrrrr r"rrbisect bisect_leftrKlenr rr2r@rMrrr) rr5r6r7rSr8effective_num_active_lorasrTidx batch_descrelaxed_batch_descrrrdispatchs4       zCudagraphDispatcher.dispatchcCsb|jr |jtjkr gSg}tjtjfD]}t|j|}|r.|jdddd| ||fq|S)a Returns capture descriptors for cudagraph capturing. Returns: List of (runtime_mode, batch_descriptors) tuples, ordered PIECEWISE first then FULL. Batch descriptors are sorted largest-first for memory efficiency. cSs|jS)N)r5)drrr0sz7CudagraphDispatcher.get_capture_descs..T)keyreverse) rrrrrrlistrsortappend)rresultrdescsrrrget_capture_descss z%CudagraphDispatcher.get_capture_descs)r!N)r)r )FFFr)__name__ __module__ __qualname____doc__rr r0r`intr4boolrr@rrDrRtupler[rerrrrr s\ %     @  "ErN) itertoolsr vllm.configrrvllm.forward_contextr vllm.loggerrvllm.lora.utilsrrfloggerrrrrrs