o ۾i 9@sdZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z e r4ddl mZmZmZmZdd lmZd d lmZeGd d d eZdS)z Based on: Chen, L., Ye, Z., Wu, Y., Zhuo, D., Ceze, L., & Krishnamurthy, A. (2023). Punica: Multi-Tenant LoRA Serving. https://arxiv.org/abs/2310.18547 )finalN) LoRAMapping)get_captured_lora_counts) HAS_TRITONtriton)round_up)LoRAKernelMetafused_moe_lora lora_expand lora_shrink) _custom_ops)PunicaWrapperBasec@s<eZdZdZdededejeBfddZde de ed Bd ed efd d Z dej dej de ej dfdefddZ  dd'ej d(ed)ed*ed ed+ej d,ej d Bd-ed.ede ej ej ej ej ffd/d0Z & &  d?dej dej de ej dfde ej dfd1ej d2ej d Bd3ej d4ej d Bd5ed6ed+ej d7ed8ed9ej d Bfd:d;Zd S)@PunicaWrapperGPUz PunicaWrapperGPU is designed to manage and provide metadata for the punica kernel. The main function is to maintain the state information for Multi-LoRA, and to provide the interface for the punica triton kernel. max_num_batched_tokens max_batchesdevicecKsdt|||||d|_|jj|_t|j|jj}tj|j|||d|_tj|j|||d|_ dS)N lora_config)rcaptured_lora_counts) r__init__r max_lorasrspecialize_active_lorarmaketoken_mapping_metaprompt_mapping_meta)selfrrrkwargsrrW/home/ubuntu/.local/lib/python3.10/site-packages/vllm/lora/punica_wrapper/punica_gpu.pyr(s$     zPunicaWrapperGPU.__init__mappinglora_index_to_idNr vocab_sizecKs8|j|_||||||j|j|j|jdS)N) is_prefill_update_base_metadatarprepare_tensorstoken_lora_indicesrsampler_indices)rrr rr!rrrrupdate_metadataKsz PunicaWrapperGPU.update_metadatayxlora_a_stacked.scalecKsB|d|jd}t|||g|j|d|jj|RdS)a Performs GEMM for multiple slices of lora_a. Semantics: for i in range(len(lora_a_stacked)): y[i] += (x @ lora_a_stacked[i]) * scale Args: y (torch.Tensor): Output tensors x (torch.Tensor): Input tensor lora_a_stacked (tuple[torch.Tensor, ...]): lora_a's weights scale (float): Scaling factor for the operation rN)viewshaper r meta_argssizerr)rr(r)r*r+rrrr add_shrinkZs zPunicaWrapperGPU.add_shrinkrTlora_b_stacked output_slices offset_startreturnc Ks||}|d|jd}|jdksJ|dt|ksJ|d} t|||g|j| |jj R|dd| |}dS)a2 Performs GEMM for multiple slices of lora_b. Semantics: for i in range(len(lora_b_stacked)): slice = output_slices[i] y[:, offset:offset+slice] += x[i] @ lora_b_stacked[i] offset += slice Args: y (torch.Tensor): Output tensor. x (torch.Tensor): Input tensors lora_b_stacked (tuple[torch.Tensor, ...]): lora_b's weight output_slices (tuple[int, ...]): Every slice's size add_inputs (bool): Defaults to True. r,rr Tr4 add_inputsN) r-r.ndimr0lenr rr/rrview_as) rr(r)r2r3r4r8ry_org num_tokensrrr add_expand{s$  zPunicaWrapperGPU.add_expandr8cKs>t|jdd|f|g|j|d|jjRd|ddS)a] Applies lora specifically for VocabParallelEmbeddingWithLoRA. Semantics: y += x @ lora_b_stacked Args: y (torch.Tensor): Output tensor. x (torch.Tensor): Input tensor. lora_b_stacked (torch.Tensor): lora_b's weights. add_inputs (bool): Default to True. rdimr7N)r unsqueezerr/r0rr)rr(r)r2r8rrrradd_lora_embeddings  z#PunicaWrapperGPU.add_lora_embedding)bufferrCc Kst|t|krt|ksJJ|dusJd|dd} tjt||d| ftj|jd}|j||||fi||j||||fddi|dS)a5 Applicable to linear-related lora. Semantics: for i in range(len(lora_a_stacked)): y[i] += ( x[i].unsqueeze(0) @ lora_a_stacked[indices[i], layer_idx, :, :] @ lora_b_stacked[indices[i], layer_idx, :, :] * scale ).squeeze(0) Args: y (torch.Tensor): Output tensor. Will be changed in-place. x (torch.Tensor): Input tensor lora_a_stacked (tuple[torch.Tensor, ...]): lora_a's weight. lora_b_stacked (tuple[torch.Tensor, ...]): lora_b's weight. scale (float): Scaling factor. output_slices (tuple[int, ...]): Every slice's size. buffer (Optional[torch.Tensor]): Defaults to None. NdTo minimize overhead, the buffer should be created by .add_lora_linear() instead of being passed in.rr,dtyperr8T)r:r0torchemptyfloat32rr1r>) rr(r)r*r2r+r3rCrrrrradd_lora_linears4(!  z PunicaWrapperGPU.add_lora_linearc Ks|}|d|jd}|d|jd}|d} |dus!Jdtj|d| ftj|jd}t||g|jddg|j |d|j j |Rt |jdd|g|g|j |d|j j Rddi||}dS) a Applies lora specifically for LogitsProcessorWithLoRA. Semantics: buffer = (x @ lora_a_stacked) * scale y += buffer @ lora_b_stacked Args: y (torch.Tensor): Output tensor. x (torch.Tensor): Input tensor. lora_a_stacked (torch.Tensor): lora_a's weights. lora_b_stacked (torch.Tensor): lora_b's weights. scale (float): Scaling factor. buffer (Optional[torch.Tensor]): Default to None. r,NrDrrEr?r8T)r-r.r0rGrHrIrr rArr/rrr r;) rr(r)r*r2r+rCrr<rJrrradd_lora_logitss>     z PunicaWrapperGPU.add_lora_logitsFtopk_idsr= block_size num_expertsadapter_enabled expert_mappad_sorted_idsnaive_block_assignmentc Cs|j||jj\} } } } } } } | r|d} d}d}nS|||d}|r-t||}tj||ftj |j d}t ||}tj||ftj |j d} tj|tj |j d}t || ||||||| ||| |duro|| } d|| |fS)z~ Aligns tokens and experts into block-sized chunks for LoRA-based mixture-of-experts (MoE) execution. r,Nr rE)rr/rrreshapenumelrrGrHint32rrcdivopsmoe_lora_align_block_size)rrMr=rNrOrrPrQrRrStoken_lora_mapping_lora_ids expert_ids sorted_idsnum_tokens_post_padmax_num_tokens_paddedmax_num_m_blocksrrrrYEsT     z*PunicaWrapperGPU.moe_lora_align_block_size topk_weightssorted_token_idsr]num_tokens_post_padded max_lora_rank top_k_num fully_shardedoffsetrZcCs,|j|d|jj\}}}}}}}|dur|}tg|||||||||| | ||| | dd| dd| dd| dd | d d | d d | dd| dd| dd| dd| dd | d d | d d | dd|||RdS)zb Performs a fused forward computation for LoRA of Mixture-of-Experts (MoE) layer. rN BLOCK_SIZE_M@ BLOCK_SIZE_N BLOCK_SIZE_K GROUP_SIZE_M NUM_WARPS NUM_STAGESr6SPLIT_Kr )rr/r0rrr get)rr(r)r*r2rbrcr]rdrerf shrink_config expand_configrPmul_routed_weightrgrhrZtoken_lora_mapping_metar[r\num_active_lorasrrradd_lora_fused_moes                    z#PunicaWrapperGPU.add_lora_fused_moe)rT)T)NFF)FFrN)__name__ __module__ __qualname____doc__intrGrstrrrlistr'Tensortuplefloatr1r>boolrBrKrLrYrzrrrrr s #    '   3 )     G E   O     r)r~typingrrGvllm.lora.layersrvllm.lora.utilsrvllm.triton_utilsrrvllm.utils.math_utilsrvllm.lora.ops.triton_opsrr r r vllmr rX punica_baserrrrrrs