o ۾iB"@s\dZddlmZddlZddlmZddlmZmZm Z ddl m Z eGdd d e Z dS) z Based on: Chen, L., Ye, Z., Wu, Y., Zhuo, D., Ceze, L., & Krishnamurthy, A. (2023). Punica: Multi-Tenant LoRA Serving. https://arxiv.org/abs/2310.18547 )finalN) LoRAMapping) bgmv_expandbgmv_expand_slice bgmv_shrink)PunicaWrapperBasec@seZdZdZdededejeBfddZde de ed Bd ed efd d Z dej dej fddZdej dej dej defddZdej dej dej dededef ddZdej dej deej dfdefddZ !d1dej dej d"eej dfd#eedfd$edd f d%d&Z !d2dej dej d"ej dedd f d'd(Zd d)dej dej deej dfd"eej dfded#eedfd*ej d Bdd fd+d,Zedej fd-d.Zd d)dej dej dej d"ej d*ej d Bdd f d/d0Zd S)3PunicaWrapperXPUz PunicaWrapperXPU is designed to manage and provide metadata for the punica kernel. The main function is to maintain the state information for Multi-LoRA, and to provide the interface for the punica ipex kernel. max_num_batched_tokens max_batchesdevicecKsDt||||tj|jdtj|jdtj|jddS)Nrr)r__init__torch_dynamo mark_dynamic_token_lora_indices_embeddings_indices_sampler_indices_padded)selfr r r kwargsrW/home/ubuntu/.local/lib/python3.10/site-packages/vllm/lora/punica_wrapper/punica_xpu.pyr szPunicaWrapperXPU.__init__mappinglora_index_to_idN max_loras vocab_sizecKs|j|_|||||dSN) is_prefill_update_base_metadata)rrrrrrrrrupdate_metadata(sz PunicaWrapperXPU.update_metadataxreturncCst|jdd|dS)Nr)rnarrowrsize)rr rrr_get_token_lora_indices3sz(PunicaWrapperXPU._get_token_lora_indicesyw_t_allscalecCst||||||dSr)rr$)rr%r r&r'rrr _apply_shrink6szPunicaWrapperXPU._apply_shrinky_offset y_slice_size add_inputscCs"||}t|||||||dSr)r$r)rr%r r&r)r*r+token_lora_indicesrrr _apply_expand?s zPunicaWrapperXPU._apply_expandlora_a_stacked.cKs@|d|jd}tt|D]}|||||||qdS)a Performs GEMM for multiple slices of lora_a. Semantics: for i in range(len(lora_a_stacked)): y[i] += (x @ lora_a_stacked[i]) * scale Args: y (torch.Tensor): Output tensors x (torch.Tensor): Input tensor lora_a_stacked (tuple[torch.Tensor, ...]): lora_a's weights scale (float): Scaling factor for the operation N)viewshaperangelenr()rr%r r.r'r slice_idxrrr add_shrinkMszPunicaWrapperXPU.add_shrinkrTlora_b_stacked output_slices offset_startc Ks|}|d|jd}|jdksJ|dt|ksJtt|D]} |j||| || ||| |d||| 7}q#||dS)a2 Performs GEMM for multiple slices of lora_b. Semantics: for i in range(len(lora_b_stacked)): slice = output_slices[i] y[:, offset:offset+slice] += x[i] @ lora_b_stacked[i] offset += slice Args: y (torch.Tensor): Output tensor. x (torch.Tensor): Input tensors lora_b_stacked (tuple[torch.Tensor, ...]): lora_b's weight output_slices (tuple[int, ...]): Every slice's size add_inputs (bool): Defaults to True. r/rr+N)r0r1ndimr#r3r2r-view_as) rr%r r6r7r8r+ry_orgr4rrr add_expandgszPunicaWrapperXPU.add_expandcKs||}t|||||dS)a] Applies lora specifically for VocabParallelEmbeddingWithLoRA. Semantics: y += x @ lora_b_stacked Args: y (torch.Tensor): Output tensor. x (torch.Tensor): Input tensor. lora_b_stacked (torch.Tensor): lora_b's weights. add_inputs (bool): Default to True. N)r$r)rr%r r6r+rr,rrradd_lora_embeddings z#PunicaWrapperXPU.add_lora_embedding)bufferr@c Kst|t|krt|ksJJ|dur1|dd} tjt||d| ftj|jd}|j||||fi||j||||fddi|dS)a6 Applicable to linear-related lora. Semantics: for i in range(len(lora_a_stacked)): y[i] += ( x[i].unsqueeze(0) @ lora_a_stacked[indices[i], layer_idx, :, :] @ lora_b_stacked[indices[i], layer_idx, :, :] * scale ).squeeze(0) Args: y (torch.Tensor): Output tensor. Will be changed in-place. x (torch.Tensor): Input tensor lora_a_stacked (tuple[torch.Tensor, ...]): lora_a's weight. lora_b_stacked (tuple[torch.Tensor, ...]): lora_b's weight. scale (float): Scaling factor. output_slices (tuple[int, ...]): Every slice's size. buffer (Optional[torch.Tensor]): Defaults to None. Nrr/dtyper r+T)r3r#rzerosfloat32r r5r>) rr%r r.r6r'r7r@rrrrradd_lora_linears4(" z PunicaWrapperXPU.add_lora_linearcCs|jddS)zJ This property provides access to padded sampler indices. N)r)rrrrsampler_indices_paddedsz'PunicaWrapperXPU.sampler_indices_paddedc Ks|}|d|jd}|d|jd}|d} |dur,tj|d| ftj|jd}t|jdd|d} t |||| |t |||| dd| |S)a Applies lora specifically for LogitsProcessorWithLoRA. Semantics: buffer = (x @ lora_a_stacked) * scale y += buffer @ lora_b_stacked Args: y (torch.Tensor): Output tensor. x (torch.Tensor): Input tensor. lora_a_stacked (torch.Tensor): lora_a's weights. lora_b_stacked (torch.Tensor): lora_b's weights. scale (float): Scaling factor. buffer (Optional[torch.Tensor]): Default to None. r/NrrATr:) r0r1r#rrCrDr r"_sampler_indicesrrr<) rr%r r.r6r'r@rr=rEsampler_indicesrrradd_lora_logitss  z PunicaWrapperXPU.add_lora_logits)rT)T)__name__ __module__ __qualname____doc__intrr strr rlistrTensor IntTensorr$floatr(boolr-tupler5r>r?rFpropertyrGrJrrrrr s              2       = r ) rNtypingrrvllm.lora.layersrvllm.lora.ops.ipex_opsrrr punica_baserr rrrrs