o ۾i)@sUddlZddlZddlmZddlmZddlmZddlZddlm Z ddl m Z ddl m Z ddlmZdd lmZe eZe ZiZeeed feejd ffed <iZeeed feejd ffed <d eejdejfddZdeejdedejfddZ ejde!de"dBdedBfddZ#ej  d$de!dededededede"dBdedBdee!edBffd d!Z$ed%dejdBde"fd"d#Z%dS)&N) lru_cache)Path)Any)envs) init_logger)vllm_is_batch_invariant)current_platform)next_power_of_2._LORA_A_PTR_DICT_LORA_B_PTR_DICTlora_a_weightsdevicec CsFtdd|D}t|}r|Sg}g}g}g}|D]C}|jdkr3|ddks,J|jdd}n|jdks:J|s@J|||| d|| d|| dqt |dkrpt j ||t j d } n|d} t t|dkst t|dkst t|dkrtd | |d|d|dft|<t|S) a  `_LORA_A_PTR_DICT` collects the required information during `profile_run`, After this, it remains constant and subsequent usage is through LUT. Refer to: https://github.com/triton-lang/triton/blob/release/3.1.x/python/tutorials/08-grouped-gemm.py cs|]}|VqdSNdata_ptr.0 lora_weightrR/home/ubuntu/.local/lib/python3.10/site-packages/vllm/lora/ops/triton_ops/utils.py z"_get_lora_a_ptr..dimrr dtypez+All LoRA weights must have the same stride.)tupler getndimsizesqueeze is_contiguousappendrstridelentorchtensoruint64set ValueError) r r keyvalueslora_strides_d0lora_strides_d1lora_strides_d2 tensor_ptrs lora_a_weightlora_ptr_tensorrrr_get_lora_a_ptrs<    r7 lora_weights offset_startcCstdd|D}t|}r|Sg}g}g}g}g} g} |} |D]W} | jdkr9| ddks2J| jdd} n| jdks@J| sFJ|| || d|| d| | d|| | | d7} | | dq"t |dkrt j ||t j d } t j ||t j d }n|d}| d} t t|dkrt t|dkrt t| dkrt t| dkr|d}|d}| d}| d}d }nt j ||d }t j ||d }t j | |d }t j | |d }d }t| }|| ||||||ft|<t|S) a `_LORA_B_PTR_DICT` collects the required information during `profile_run`, After this, it remains constant and subsequent usage is through LUT. Refer to: https://github.com/triton-lang/triton/blob/release/3.1.x/python/tutorials/08-grouped-gemm.py csrrrrrrrrTrz"_get_lora_b_ptr..rrrrrrrTr F)r!r r"r#r$r%r&r'rr(r)r*r+r,r-max)r8r9r r/r0slice_offset_lstr4r1r2r3 hidden_sizes slice_offset lora_b_weightr6slice_start_tensorlora_strides_d0_tensorlora_strides_d1_tensorlora_strides_d2_tensorhidden_sizes_tensor same_strideMAX_Nrrr_get_lora_b_ptrIsn      rGop_type add_inputsreturncCstj}|durztsztj}|dd}|dd}d}|dkr2|d|dt|d}n |d|d}t |d|}| sSt d|dSt d|d tt|}t|}Wd|S1sswY|Sd}|S) N _-expandz.json/z No LoRA kernel configs found in z%Using tuned LoRA kernel configs from .)rVLLM_TUNED_CONFIG_FOLDERis_batch_invariantr*cudaget_device_namereplaceupperstrrexistslogger warning_once info_onceopenjsonload)rHrIuser_defined_config_foldergpu_name config_fname config_pathf config_datarrrload_lora_op_configs.       re max_lorasbatch hidden_sizerank num_slicesmoe_intermediate_sizec s|dvsJi}|dkr+|dkrdnd} trd} dd|dkr d nd| d ddd dd }n<|d vr?dtdt|dd dddd}n(|dvrVddtdtdt|d dddd}ndtdtd|dd dd dd}||dkrq||fn||f\t||} | std|S| tp| t| fddd} | t|} | tp| t| fddd} | tp| t| fddd} | tp| t| fddd} |dur|| tp| t| fddd} | dusJ| S)N)shrinkrNfused_moe_lora_w13_shrinkfused_moe_lora_w13_expandfused_moe_lora_w2_shrinkfused_moe_lora_w2_expandrl@r rr) block_mblock_nblock_ksplit_k num_warpsnum_ctas group_size_m num_stagesmax_nreg)rmror)rwrxryr{r~r}rz)rnrp)rwrxryr{r|r~rz!Using default LoRA kernel configsctt|Srabsintx)rfrrz%get_lora_op_configs..)r/crrrr)mrrrrcrrrr)krrrrcrrrr)nrrr$rcrrrr)irrr,r) rRminr r;rerYrZr"rWkeys) rHrfrgrhrirjrIrkdefaultrzrdr)rrrrfrrget_lora_op_configss          rcCsto tdo tj S)zz Refer to: https://github.com/triton-lang/triton/blob/v3.5.0/python/tutorials/11-programmatic-dependent-launch.py Z)ris_cudahas_device_capabilityrVLLM_LORA_DISABLE_PDLr:rrr supports_pdl3s r)NNr)& functoolsr]rpathlibrtypingrr*vllmr vllm.loggerr*vllm.model_executor.layers.batch_invariantrvllm.platformsrvllm.utils.math_utilsr __name__rYrRr dictr!rr+__annotations__r listTensorr r7rGrWboolrerrrrrrsb         &&0 P   x