o i#&@sddlZddlZddlZddlmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZddlmZeeZGd d d ZdS) N) init_logger)LoRALayerWeights) PEFTHelper) get_lora_idis_base_embeddding_weightsparse_fine_tuned_lora_name)TensorizerConfig) WeightsMapper)is_pin_memory_availablec@sVeZdZdZdededeeefddfddZdeddfd d Z d ededBfd d Z dede fddZ e d edeede fddZe     d#dedeeejfdededejdBdedBdedBdeedBddfddZedddddddddedeedededBdedejdBdedBdedBd edBdeedBddfd!d"ZdS)$ LoRAModelzA LoRA fine-tuned model. lora_model_idranklorasreturnNcCs.||_|dksJd|j||_||_dS)z Args: lora_model_id: The integer id for the lora model. rank: lora rank. loras: module name -> weights for lora-replaced layers. rz.a valid lora id should be greater than 0, got N)idr r)selfr r rrJ/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm/lora/lora_model.py__init__s    zLoRAModel.__init__cCs|j||j|jdS)z[Return a copy of the object with different ids. Will share the underlying tensors.)r r) __class__r rcopy)rr rrrclone0s zLoRAModel.clone module_namecCs|j|dS)z#Get LoRA for a given module by nameN)rget)rrrrrget_lora:szLoRAModel.get_lora lora_namecCs ||jvS)N)r)rrrrrcheck_lora_name>s zLoRAModel.check_lora_name skip_prefixescCs,|D]}d||vs||rdSqdS)z:Check if a module should be skipped based on skip prefixes.TF) startswith)rrprefixrrr_should_skip_moduleAs zLoRAModel._should_skip_modulecudatensors peft_helperdevicedtypemodel_vocab_sizeweights_mapperc Cst|dkot} i} |D]q\} } t| rq|r!|| |r!qt| |\} }| | vr4t| || | <|rjd| vrS|durS|| jdkrSt d| jdd|d| j ||d| | _ | ri| | j | | _ q| j ||d| | _ | r| | j | | _ q|||j| S) z0Create a LoRAModel from a dictionary of tensors.cpulora_embedding_ANzThe embedding LoRA size(z;) must be consistent with the base model's vocabulary size(z).)r%r&)strr itemsrr!rr from_configshape RuntimeErrortolora_a pin_memorylora_br)clsr r#r$r%r&r'r(rr3r tensor_nametensorr is_lora_arrrfrom_lora_tensorsIs@ zLoRAModel.from_lora_tensors)r r%r&r'r(tensorizer_config_dictrlora_direxpected_lora_modulesr;c stjd} tjd} tjd} i}gdtffdd }| rSddlm}tdi| }tj|jd } |}|| fd |j i|j }||n\tj | rgt j | d d }|||D] }||||<qkWd n1swYn*tj | stj | rtj | r| n| }tj||dd}||ntdj|d urtn||||||dS)a Create a LoRAModel from a local checkpoint. Args: lora_dir: The local path that has lora data. expected_lora_modules: Name of modules that are expected to be replaced by lora. peft_helper: Loaded lora configuration information. lora_model_id: LoRA model id. If not given, automatically set by a global counter. device: Device where the lora model is loaded. dtype: dtype of the lora model weights. skip_prefixes: List of module name prefixes to skip during loading. Models can define this to skip modules not used in inference (e.g., MTP layers). Format: ["mtp."] Returns: Loaded LoRA Model. zadapter_model.safetensorszadapter_model.binzadapter_model.ptmodulescs|D]F}t|r qd|vrqr|rqt|\}}d|vr;|d}||dd}|vr:|q|dddvrJ|qr[tdddd dS) N base_layerz.expertsr+rzWhile loading z, expected target modules in z but received z6. Please verify that the loaded LoRA module is correct)keysrr!rfindappendrsplit ValueError)r> lora_moduler_ expert_idx expert_suffixr6r=r<runexpected_modulesr(rrcheck_unexpected_moduless8    zALoRAModel.from_local_checkpoint..check_unexpected_modulesr)TensorDeserializerzadapter_model.tensorsr&pt) frameworkNT) map_location weights_onlyz doesn't contain tensors)r r#r$r%r&r'r(rr)ospathjoindict tensorizerrMrtensorizer_dir_construct_tensorizer_argsr&deserialization_kwargsisfile safetensors safe_openrA get_tensortorchloadrEr:r)r6r<r=r$r r%r&r'r(r;rlora_tensor_pathlora_bin_file_pathlora_pt_file_pathr#rLrMtensorizer_configtensorizer_argsfmodulelora_file_pathrrJrfrom_local_checkpoint{s`!      zLoRAModel.from_local_checkpoint)r"NNNN)__name__ __module__ __qualname____doc__intrUr,rrrrboolr staticmethodlistr! classmethodr^Tensorrr&r r:setrhrrrrr s      1     r )rRr[r^ vllm.loggerrvllm.lora.lora_weightsrvllm.lora.peft_helperrvllm.lora.utilsrrr+vllm.model_executor.model_loader.tensorizerr vllm.model_executor.models.utilsr vllm.utils.platform_utilsr riloggerr rrrrs