o -i@sddlmZddlZerddlmZdejdeejejejeeeeffddZ dd d e edBd ed ed edej deejejejeje effddZ dS)) TYPE_CHECKINGN) LoRAMappingtoken_lora_tensorreturnc Cstj|dd\}}tj|dd}t|}|dd|dd|}|}|d}d}|dkr?|dkr?d}|||||||fS) at Get the information required for the sgmv kernel. With the features: 1. If consecutive requests in the batch use the same LoRA, this function will combine them into a single request, improving sgmv kernel inference performance. 2. At the beginning of each prefill stage inference, recalculations are needed based on the input, but only once. T) return_countsr)dimNF) torchunique_consecutivecumsum zeros_likecopy_maxitemsumsize) rlora_indices_tensorseq_length_tensor cum_resultb_seq_start_tensor max_length token_nums batch_sizeno_lorar[/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/lora/punica_wrapper/utils.py compute_meta s(      rmappingrlora_index_to_id max_loras vocab_sizeextra_vocab_sizedevicecslt|j}|}|}fdd|jD} d} tt|D]!} || dkr0|| nd} || dkr:| nd|| <| || <q!|||g} tj| tj |d} tj| tj |d}t | d|| d||g}t |dk|d|}| d}|}| }t |dk|d|}tj dt||tj d |t|}|jd|jd|jd|jdg}|||||fS) axConverts LoRAMapping to index tensors. Args: mapping: LoRAMapping mapping rows in a batch to LoRA ids. lora_index_to_id: List mapping LoRA ids to LoRA indices. max_loras: Maximum number of LoRAs. vocab_size: Model vocab size. extra_vocab_size: Extra vocab size each LoRA can have. Returns: A tuple of tensors: base_indices: Tensor of shape [batch_size] mapping batch rows to LoRA indices. sampler_indices: Tensor of shape [batch_size] mapping requests to LoRA indices for sampler. For generation, this will be the same as base_indices. For prefill, this will map requests to LoRA indices. sampler_indices_padded: Tensor of shape [batch_size] mapping requests to LoRA indices for sampler with padding. Same as sampler_indices, but -1 is replaced with max_loras. embeddings_indices: Tensor of shape [2, batch_size] mapping requests to embedding indices. First row is for embeddings added by the LoRAs, second row is for the LoRA.lora_a embeddings. indices_len: List of lengths of the above tensors. It contains (base_indices, sampler_indices, sampler_indices_padded, embeddings_indices). cs"g|] }|dkr |ndqS)rr )index).0xrrr ]sz#convert_mapping..Nrr )dtyper#r)r#r))list index_mappingcopyprompt_mappingrangelenr$r tensorlongstackwhereclonearangeshape)rrr r!r"r#index_mapping_indicesembedding_indices lora_indicesr.lora_idxi indices_listindicesprompt_mapping_tensorembeddings_indices base_indicessampler_indicessampler_indices_padded indices_lenrr'rconvert_mapping4sh%     rE) typingrr vllm.lora.layersrTensortupleintboolrr+r#rErrrrs0   '