o ۾i=@sldZddlmZmZddlmZddlZddlmZm Z er$ddl m Z Gdd d eZ Gd d d e Z dS) z Based on: Chen, L., Ye, Z., Wu, Y., Zhuo, D., Ceze, L., & Krishnamurthy, A. (2023). Punica: Multi-Tenant LoRA Serving. https://arxiv.org/abs/2310.18547 )ABCabstractmethod) TYPE_CHECKINGN) compute_metaconvert_mapping) LoRAMappingc@seZdZdZedddeedBdededdf d d Zed ee j d fe j Bd e j dee j d fde de j dBf ddZ e  d"d e j d ee j d fe j Bdee j d fdeed fdede j dBf ddZ e d#d e j d e j de j dede j dBf ddZeddd e j d e j dee j d fdee j d fde deed fdee j d fdBde j dBfddZeddd e j d e j de j de j de j dBde j dBf d d!ZdS)$PunicaWrapperABCz PunicaWrapper ABC. mappingrlora_index_to_idN max_loras vocab_sizereturncKt)z2 Update the lora-related metadata NotImplementedErrorselfr r r r kwargsrX/home/ubuntu/.local/lib/python3.10/site-packages/vllm/lora/punica_wrapper/punica_base.pyupdate_metadatas z PunicaWrapperABC.update_metadatay.xlora_a_stackedscalecKr)z? Performs GEMM for multiple slices of lora_a. rrrrrrrrrr add_shrink) zPunicaWrapperABC.add_shrinkrTlora_b_stacked output_slices offset_startcKr)z> Performs GEMM for multiple slices of lora_b. rrrrrr r! add_inputsrrrr add_expand8szPunicaWrapperABC.add_expandr#cKr)z Applies lora specifically for VocabParallelEmbeddingWithLoRA, and this layer only requires the expand operation. rrrrrr#rrrradd_lora_embeddingHrz#PunicaWrapperABC.add_lora_embeddingbufferr(c Kr)z4 Applicable to linear-related lora. r rrrrrrr r(rrrradd_lora_linearWsz PunicaWrapperABC.add_lora_linearcKr)zI Applies lora specifically for LogitsProcessorWithLoRA. rrrrrrrr(rrrradd_lora_logitsjsz PunicaWrapperABC.add_lora_logitsrTT)__name__ __module__ __qualname____doc__rlistintrtupletorchTensorfloatrr$boolr&r*r,rrrrr s                 r c@s>eZdZdZdededejeBfddZddd e ed Bd ed efd dZ dej dd fddZ e deej ej ej eeeffddZe dej fddZe dej fddZe dej fddZe dej fddZddd e ed Bd ed efddZedeej d fej Bd!ej d"eej d fd#edej d Bf d$d%Ze & 'dKdej d!eej d fej Bd(eej d fd)eed fd*edej d Bf d+d,Ze 'dLdej d!ej d(ej d-edej d Bf d.d/Zed d0dej d!ej d"eej d fd(eej d fd#ed)eed fd1eej d fd Bdej d Bfd2d3Zed d0dej d!ej d"ej d(ej d1ej d Bdej d Bf d4d5Z 6dMd7ej d8ed9ed:ed ed;ej dd?Z 6 6 & dNdej d!ej d"eej d fd(eej d fd@ej dAej d BdBej dCej d BdDedEed;ej dFedGedHej d BfdIdJZd S)OPunicaWrapperBasez PunicaWrapperBase is designed to manage and provide metadata for the punica kernel. The main function is to maintain the state information for Multi-LoRA, and to provide the interface for the punica. max_num_batched_tokens max_batchesdevicecKstj|tj|d|_tj|tj|d|_tj|tj|d|_tjd|tj|d|_dgd|_tj|tj|d|_tj|tj|d|_ tj|tj|d|_ ||_ d|_ d|_ d|_d|_d|_dS)N)dtyper=rF)r6emptylong_token_lora_indices_sampler_indices_sampler_indices_padded_embeddings_indices indices_len_seq_start_locs _seq_lengths_lora_indices_per_batchr= max_length token_nums batch_size is_prefillno_lora)rr;r<r=rrrr__init__s0   zPunicaWrapperBase.__init__r rr Nr r c Csd}t||||||j\}}}} } |jd|jd||jd|jd||jd|jd||jd| jdd| jdf| | |jdd<dS)Nrr) rr=rDshapecopy_rErFrGrH) rr r r r extra_vocab_size base_indicessampler_indicessampler_indices_paddedembeddings_indicesrHrrr_update_base_metadatas2  z'PunicaWrapperBase._update_base_metadatatoken_lora_tensorrc Cst|\}}}}}}}|jd|jd||jd|jd||jd|jd|||_||_||_||_ dS)Nr) rrIrRrSrJrKrNrLrMrP) rrZb_seq_start_tensorseq_length_tensorlora_indices_tensorrNrLrMrPrrr_update_prefill_metadatas$  z*PunicaWrapperBase._update_prefill_metadatacCs:|jd|j|jd|j|jd|j|j|j|jfS)aY This property provides a convenient way to access the necessary metadata for prefill-related kernel computations. 1. seq_start_locs: Tensor of sequence start positions. 2. seq_lengths: Tensor of sequence lengths. 3. lora_indices_per_batch: Tensor of lora indices, and an index of -1 means no lora should be applied. 4. batch_size: Batch size after clustering identical lora indices. 5. max_length: The maximum sequence length in the batch. 6. token_nums: The token numbers in the batch. N)rIrNrJrKrLrM)rrrrprefill_metadatasz"PunicaWrapperBase.prefill_metadatacC|jd}|jd|S)z This property provides the lora indices corresponding to each token in the batch. An index of -1 means no lora should be applied. rN)rHrD)rtoken_lora_lenrrrtoken_lora_indices z$PunicaWrapperBase.token_lora_indicescCr`)zt This property is used to access the lora indices specifically for LogitsProcessorWithLoRA. rN)rHrE)rsampler_indices_lenrrrrVrcz!PunicaWrapperBase.sampler_indicescCr`)zJ This property provides access to padded sampler indices. r?N)rHrF)rindices_padded_lenrrrrW s z(PunicaWrapperBase.sampler_indices_paddedcCs |jd}|jddd|fS)z This property provides access to the indices used for lora embeddings, specifically for VocabParallelEmbeddingWithLoRA. N)rHrG)rembeddings_indices_lenrrrrXs z$PunicaWrapperBase.embeddings_indicescKs6||||||jr||jd|_dSd|_dS)NTF)rYrOr^rbrrrrrs    z!PunicaWrapperBase.update_metadatar.rrrcKr)a Performs GEMM for multiple slices of lora_a. Semantics: for i in range(len(lora_a_stacked)): y[i] += (x @ lora_a_stacked[i]) * scale Args: y (Union[tuple[torch.Tensor, ...], torch.Tensor]): Output tensors x (torch.Tensor): Input tensor lora_a_stacked (tuple[torch.Tensor, ...]): lora_a's weights scale (float): Scaling factor for the operation rrrrrr-szPunicaWrapperBase.add_shrinkrTrr r!cKr)a Performs GEMM for multiple slices of lora_b. Semantics: offset = offset_start for i in range(len(lora_b_stacked)): slice = output_slices[i] y[:, offset:offset+slice] += x[i] @ lora_b_stacked[i] offset += slice Args: y (torch.Tensor): Output tensor. x (Union[tuple[torch.Tensor, ...], torch.Tensor]): Input tensors lora_b_stacked (tuple[torch.Tensor, ...]): lora_b's weight output_slices (tuple[int, ...]): Every slice's size offset_start (int): The starting position of y, defaults to 0 add_inputs (bool): Defaults to True. rr"rrrr$GszPunicaWrapperBase.add_expandr#cKr)a Applies lora specifically for VocabParallelEmbeddingWithLoRA. and this layer only requires the expand operation. Semantics: y += x @ lora_b_stacked Args: y (torch.Tensor): Output tensor. x (torch.Tensor): Input tensor. lora_b_stacked (torch.Tensor): lora_b's weights. add_inputs (bool): Default to True. rr%rrrr&hsz$PunicaWrapperBase.add_lora_embeddingr'r(c Kr)aB Applicable to linear-related lora. Semantics: for i in range(len(lora_a_stacked)): y[i] += ( x[i].unsqueeze(0) @ lora_a_stacked[indices[i], layer_idx, :, :] @ lora_b_stacked[indices[i], layer_idx, :, :] * scale ).squeeze(0) Args: y (torch.Tensor): Output tensor. Will be changed in-place. x (torch.Tensor): Input tensor lora_a_stacked (tuple[torch.Tensor, ...]): lora_a's weight. lora_b_stacked (tuple[torch.Tensor, ...]): lora_b's weight. scale (float): Scaling factor. output_slices (tuple[int, ...]): Every slice's size. buffer (Optional[tuple[torch.Tensor, ...]]): Defaults to None. rr)rrrr*s#z!PunicaWrapperBase.add_lora_linearcKr)a Applies lora specifically for LogitsProcessorWithLoRA. Semantics: buffer = (x @ lora_a_stacked) * scale y += buffer @ lora_b_stacked Args: y (torch.Tensor): Output tensor. x (torch.Tensor): Input tensor. lora_a_stacked (torch.Tensor): lora_a's weights. lora_b_stacked (torch.Tensor):lora_b's weights. scale (float): Scaling factor. buffer (Optional[torch.Tensor]):Default to None. rr+rrrr,sz!PunicaWrapperBase.add_lora_logitsFtopk_ids num_tokens block_size num_expertsadapter_enabled expert_mappad_sorted_idsc Cr)z~ Aligns tokens and experts into block-sized chunks for LoRA-based mixture-of-experts (MoE) execution. r) rrhrirjrkr rlrmrnrrrmoe_lora_align_block_sizesz+PunicaWrapperBase.moe_lora_align_block_size topk_weightssorted_token_ids expert_idsnum_tokens_post_padded max_lora_rank top_k_num fully_shardedoffsettoken_lora_mappingcCr)zj Performs a fused forward computation for LoRA of Mixture-of-Experts (MoE) layer. r)rrrrrrprqrrrsrtru shrink_config expand_configrlmul_routed_weightrvrwrxrrradd_lora_fused_moesz$PunicaWrapperBase.add_lora_fused_moer-r.)NF)FFrN)r/r0r1r2r4r6r=strrQr3rYr7r^propertyr5r_rbrVrWrXrrr8rr$r9r&r*r,ror|rrrrr:|sR %  $                $  %  !     r:)r2abcrrtypingrr6utilsrrvllm.lora.layersrr r:rrrrs  f