o پi 5@sdddlZddlmZddlmZmZddlmZmZddl m Z ddl m Z dZ Gdd d eZdS) N)BaseLoRABackend) chunked_sgmv_lora_expand_forward chunked_sgmv_lora_shrink_forward) LoRABatchInfogenerate_sequence_lengths) ForwardBatch) ServerArgscsXeZdZdZdZdedejdeffdd Z dej d ej d ej fd d Z d.dej d ej dej dej d ej f ddZ d.dej dej dej dej dedej d ej fddZ d.dej dej dej dej dej d ej f ddZded efddZdedefd d!Zded"eed#eed$eed%ef d&d'Zedefd(d)Zd*ej d+efd,d-ZZS)/ChunkedSgmvLoRABackenda Chunked LoRA backend using segmented matrix-vector multiplication. This backend is largely based on the SGMV (Segmented Gather Matrix-Vector multiplication) algorithm introduced in the Punica paper (https://arxiv.org/pdf/2310.18547). One main variation made here is to segment the input sequences into fixed-size chunks, which reduces excessive kernel launches especially when the LoRA distribution is skewed. csgmvmax_loras_per_batchdevice server_argscst|||j|_dSN)super__init__max_lora_chunk_sizemax_chunk_size)selfr r r __class__[/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/lora/backend/chunked_backend.pyrs zChunkedSgmvLoRABackend.__init__xweightsreturncOst|||jddS)Nrr batch_info num_slices)rr)rrrargskwargsrrrrun_lora_a_sgemm$s z'ChunkedSgmvLoRABackend.run_lora_a_sgemmN output_offset base_outputc Os$|jd}|}t|||j|||dS)Nrrr slice_offsetsmax_slice_sizer$)shaperr) rrrr#r$r r! output_dimr(rrrrun_lora_b_sgemm.s z'ChunkedSgmvLoRABackend.run_lora_b_sgemm qkv_lora_a qkv_lora_bmax_qkv_out_dimc Os<t|tjsJt|||jdd} t| ||j|||d} | S)Nrr&) isinstancetorchTensorrrr) rrr,r-r#r.r$r r! lora_a_output lora_outputrrr run_qkv_loraCs z#ChunkedSgmvLoRABackend.run_qkv_loragate_up_lora_agate_up_lora_bc OsJt|tjsJ|jdd}t|||jdd} t| ||j|||d} | S)Nr%rr&)r0r1r2r)rrr) rrr6r7r#r$r r!r*r3r4rrrrun_gate_up_lorads"z'ChunkedSgmvLoRABackend.run_gate_up_lora forward_batchcCsP|jtkrtS|jr|jn|j}|dkrd}n |dkr d}nd}t|j|S)a Heuristically determine the chunk size based on token token number in a batch. Args: forward_batch (ForwardBatch): The batch information containing sequence lengths. Returns: The determined chunk size @ r )rMIN_CHUNK_SIZE forward_mode is_extendextend_num_tokens batch_sizemin)rr: num_tokens chunk_sizerrr_determine_chunk_sizes  z,ChunkedSgmvLoRABackend._determine_chunk_sizemax_bs_in_cuda_graphnum_tokens_per_bsc Cs|tdt|}||}td@t|dtj|tjdtj|dtjdtj|tjdtj|tjdtj|jtjdtj|jtjdddd |_WddS1sVwYdS)NrcudaTdtype) bsuse_cuda_graphseg_lens seg_indptrweight_indices permutation lora_ranksscalings num_segmentsmax_len) r?r1r rzerosint32r floatcuda_graph_batch_info)rrHrImax_num_segmentsmax_num_tokensrrrinit_cuda_graph_batch_infos$  "z1ChunkedSgmvLoRABackend.init_cuda_graph_batch_inforQrSrTrNcCs||}tj||d\}}|j||d\} } t| } tj|tjddd} tj|tjddd} |stt |j | |dtj | dftj|j dtj | ftj|j dtj |j ftj|j dtj |j ftj|j dtj t|ftj|j ddd }n |j}|j |_| |_||_|jd|j j| dd |jd|j j| dd |jd| j| dd |jd| dj| dd |jdt|j|dd ||_dS) N)seq_weight_indicesr:)weights_reorderedrFTcpu)rL pin_memoryr Fr)rLr ) rMrUrVrNrPrQrSrTrRrO) non_blocking)rGr _get_permutation_get_segments_infolenr1tensorrXrYrrCemptyr r rZrMrUrVrScopy_rTrQrPrRr)rr:rQrSrTrNrFrRweight_indices_reorderedseg_weight_indicesrPrUlora_ranks_tensorscalings_tensorrrrrprepare_lora_batchsn       z)ChunkedSgmvLoRABackend.prepare_lora_batchcCstd6tj|tjd}t|}t||}tjt|ftjdd}tj |d|d||}||fWdS1s>wYdS)a Computes permutation indices for reordering tokens by their LoRA adapter assignments. This function implements the "gather" step in Chunked Segmented Gather Matrix Vector multiplication by creating a permutation that groups tokens by their LoRA adapter. Tokens using the same LoRA adapter are placed together to enable efficient batched computation. Example: seq_weight_indices = [0, 1, 0] # 3 sequences using adapters [0, 1, 0] extend_seq_lens = [2, 1, 3] # sequence lengths [2, 1, 3 tokens] # Creates row_weight_indices: [0, 0, 1, 0, 0, 0] (6 tokens total) # Returns permutation: [0, 1, 3, 4, 5, 2] (groups adapter 0 tokens together) # weights_reordered: [0, 0, 0, 0, 0, 1] (sorted by adapter) Args: seq_weight_indices: List of LoRA adapter indices for each sequence forward_batch (ForwardBatch): Batch information containing sequence lengths Returns: tuple: (permutation, weights_reordered) where: - permutation: Token reordering indices to group by adapter - weights_reordered: Sorted adapter indices for each token r`rKTrLra)stableoutN) r1r rfrXrrepeat_interleavergrelongargsort)r^r: seg_lens_cpurow_weight_indicesrRr_rrrrcs $z'ChunkedSgmvLoRABackend._get_permutationr_rFc Cstd|tj|dd\}}g}g}t||D]/\}}|}||d|} ||g| ||g| d||| d|qtj|tjd} tj|tjdd}tj t | dftjdd} d| d<tj | dd| dd <|| fWd S1swYd S) a Computes segment information for chunked SGMV operations. This function takes the reordered weight indices and creates segments of fixed size (self.segment_size) for efficient kernel execution. Each segment contains tokens that use the same LoRA adapter, enabling vectorized computation. The segmentation is necessary because: 1. GPU kernels work efficiently on fixed-size blocks 2. Large groups of tokens using the same adapter are split into manageable chunks 3. Each segment can be processed independently in parallel Example: weights_reordered = [0, 0, 0, 0, 0, 1] # 5 tokens with adapter 0, 1 with adapter 1 segment_size = 3 # Creates segments: # Segment 0: tokens 0-2 (adapter 0), length=3 # Segment 1: tokens 3-4 (adapter 0), length=2 # Segment 2: token 5 (adapter 1), length=1 # Returns: # weight_indices_list: [0, 0, 1] (adapter for each segment) # seg_indptr: [0, 3, 5, 6] (cumulative segment boundaries) Args: weights_reordered (torch.Tensor): Sorted adapter indices for each token chunk_size (int): Fixed size for each segment Returns: tuple: (weight_indices_list, seg_indptr) where: - weight_indices_list: LoRA adapter index for each segment - seg_indptr: Cumulative segment boundaries (CSR-style indptr) r`T) return_countsrrKrnr)dimN) r1r unique_consecutivezipitemextendappendrfrXrgrecumsum) rr_rFunique_weightscountsweight_indices_list seg_lens_list weight_idx group_lennum_segsrOrPrrrrd+s. # $z)ChunkedSgmvLoRABackend._get_segments_infor)__name__ __module__ __qualname____doc__nameintr1r rrr2r"r+r5r9rrGr]listrYboolrm staticmethodrcrd __classcell__rrrrr s     ' "  I)r )r1$sglang.srt.lora.backend.base_backendrsglang.srt.lora.triton_opsrrsglang.srt.lora.utilsrr,sglang.srt.model_executor.forward_batch_inforsglang.srt.server_argsrr?r rrrrs