o ۾ih+@svUddlZddlmZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"dd l#m$Z$dd l%m&Z&dd l'm(Z(m)Z)erdd l*m+Z+ddl,m-Z-ddl.m/Z/ee0Z1de2de3de4e2fddZ5da6ddZ7e"eeeee eeeeeeee!eehZ8e9e:ee;d<dej d8dd!d"d#de2dede dBdef d$d%Z?dej=s&z,get_captured_lora_counts..)range)r"r#r&r)r*get_captured_lora_counts0s    r.cCs td7atS)Nr%)_GLOBAL_LORA_IDr&r&r&r* get_lora_idEsr0_all_lora_classesmodelcCs(tdd|DrtddSdS)z@Checks if the model contains FusedMoE layers and warns the user.css|]}t|tVqdSN) isinstancer)r'moduler&r&r* aszis_moe_model..z8MoE model detected. Using fused MoE LoRA implementation.TF)anymoduleslogger info_once)r2r&r&r* is_moe_model_s r;layer lora_configpacked_modules_list model_configcCs>tD]}|j||||dr||}|||||Sq|S)N) source_layerr=r>r?)r1can_replace_layercreate_lora_weights)r<r"r=r>r?lora_clsinstance_layerr&r&r* from_layergs rErlm_headr cCs0t||j|jj|jj|}|||||Sr3)r embedding_dimweightdtypedeviceget_sharded_to_full_mappingrB)r<rFr"r=r?retr&r&r*from_layer_logits_processor|srM module_name new_modulecCs<|d|ddd}|dd}t||||S)z1Replace a submodule in a model with a new module..N) get_submodulejoinsplitsetattr)r2rNrOparent target_namer&r&r*replace_submodules rXnameweights_mapperzWeightsMapper | NonecCs|dr|dd}|r||n|}d|}n |r ||n|}|dr)dnd}|d}|ddkrS|dd ksB|dd krSd||d}||dd kfS|dd ks_|dd krpd||d}||dd kfSt|d )aParse the name of lora weights. args: name: the name of the fine-tuned LoRA, e.g. base_model.model.dense1.weight weights_mapper: maps the name of weight, e.g. `model.` -> `language_model.model.`, return: tuple(module_name, is_lora_a): module_name: the name of the module, e.g. model.dense1, is_lora_a whether the tensor is lora_a or lora_b. zbase_model.model.r,rrPrQrHlora_Alora_Blora_embedding_Alora_embedding_Bz is unsupported LoRA weight) startswithreplace _map_namerTrS ValueError)rYrZ start_indexpartsnew_namer&r&r*parse_fine_tuned_lora_names    $rhcCsd}||S)N)z.embed_tokens.base_layer.weightz.lm_head.base_layer.weight)endswith)rYembedding_suffixesr&r&r*is_base_embeddding_weightss rkcCst}|D]8\}}t|dd}|dur|D]}||qt|tfr/||ddt|tfr?||ddqt|S)z2 In vLLM, all linear layers support LoRA. embedding_modulesNrPrQ) set named_modulesgetattraddr4rrTrlist)r2supported_lora_modulesrYr5rlr&r&r*get_supported_lora_moduless    rs lora_pathcstjrSdrtjStjrtjStjr>ddl m }m ddl m }fdd}||f}d}n fdd}ttf}d }z|}W|S|y`t|YSw) a' Resolves the given lora_path to an absolute local path. If the lora_path is identified as a Hugging Face model identifier, it will download the model and return the local snapshot path. Otherwise, it treats the lora_path as a local file path and converts it to an absolute path. Parameters: lora_path (str): The path to the lora model, which can be an absolute path, a relative path, or a Hugging Face model identifier. Returns: str: The resolved absolute local path to the lora model. ~r)InvalidParametersnapshot_download) HTTPErrorcs dS)N)model_idr&r&rtrwr&r*s z+get_adapter_absolute_path..z&Error downloading the ModelScope modelcs tjdS)N)repo_id)huggingface_hubrwr&)rtr&r*r{ s z'Error downloading the HuggingFace model)ospathisabsra expanduserexistsabspathrVLLM_USE_MODELSCOPE modelscope.hub.snapshot_downloadrvrwrequestsrxrrr9 exception)rtrvrx download_fndownload_exceptions error_loglocal_snapshot_pathr&rzr*get_adapter_absolute_paths,        rcCsHt|r t|}rt|}|jsdd|D|d<|Stdt|S)NcSs&g|]\}}}}d|vr|dqS)z..rP)rstrip)r'_ weight_namer&r&r*r+*s  z2process_packed_modules_mapping..expertszGTo support LoRA for MoE model, 'get_expert_mapping' must be implemented)r;rris_3d_moe_weightAttributeError)r2moe_packed_mappingpacked_modules_mappingr&r&r*process_packed_modules_mappings  rr3)Ir~typingrr}huggingface_hub.utilsrrtorchr transformersrvllmrvllm.config.lorar vllm.loggerr vllm.lora.layersr r r r rrrrrrrrrrrrr$vllm.model_executor.layers.fused_moer!vllm.model_executor.layers.linearrvllm.model_executor.utilsrr+vllm.model_executor.layers.logits_processorr3vllm.model_executor.layers.vocab_parallel_embeddingr vllm.model_executor.models.utilsr!__name__r9intboolrqr.r/r0r1rmtype__annotations__Moduler;rErMstrrXtuplerhrkrsrdictrr&r&r&r*s       L            , $7