o -iA)@sdUddlZddlmZmZddlZddlmZmZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)m*Z*erdd l+m,Z,ddl-m.Z.ddl/m0Z0ee1Z2da3ddZ4e#eeeee!e eeeeeee"eehZ5e6e7ee8d<de j9de:fddZ; d5de j9de d5dddd deNszis_moe_model..z8MoE model detected. Using fused MoE LoRA implementation.TF)anymoduleslogger info_once)r)r%r%r& is_moe_modelLs r4layer max_loras lora_configpacked_modules_list model_configcCs>tD]}|j||||dr||}|||||Sq|S)N) source_layerr7r8r9)r(can_replace_layercreate_lora_weights)r5r6r7r8r9lora_clsinstance_layerr%r%r& from_layerTs r?r lm_headr!cCs0t||j|jj|jj|}|||||Sr+)r embedding_dimweightdtypedeviceget_sharded_to_full_mappingr<)r5r@r6r7r9retr%r%r&from_layer_logits_processorisrG module_name new_modulecCs<|d|ddd}|dd}t||||S)z1Replace a submodule in a model with a new module..N) get_submodulejoinsplitsetattr)r)rHrIparent target_namer%r%r&replace_submodule{s rRnameweights_mapperr"cCs|dr|dd}|r||n|}d|}n |r ||n|}|dr)dnd}|d}|ddkrS|dd ksB|dd krSd||d}||dd kfS|dd ks_|dd krpd||d}||dd kfSt|d )aParse the name of lora weights. args: name: the name of the fine-tuned LoRA, e.g. base_model.model.dense1.weight weights_mapper: maps the name of weight, e.g. `model.` -> `language_model.model.`, return: tuple(module_name, is_lora_a): module_name: the name of the module, e.g. model.dense1, is_lora_a whether the tensor is lora_a or lora_b. zbase_model.model.rrJrKrBlora_Alora_Blora_embedding_Alora_embedding_Bz is unsupported LoRA weight) startswithreplace _map_namerNrM ValueError)rSrT start_indexpartsnew_namer%r%r&parse_fine_tuned_lora_names    $rccCsd}||S)N)z.embed_tokens.base_layer.weightz.lm_head.base_layer.weight)endswith)rSembedding_suffixesr%r%r&is_base_embeddding_weightss rfcCst}|D]8\}}t|dd}|dur|D]}||qt|tfr/||ddt|tfr?||ddqt|S)z2 In vLLM, all linear layers support LoRA. embedding_modulesNrJrK) set named_modulesgetattraddr,rrNrlist)r)supported_lora_modulesrSr.rgr%r%r&get_supported_lora_moduless    rn lora_pathcstjrSdrtjStjrtjStjr>ddl m }m ddl m }fdd}||f}d}n fdd}ttf}d }z|}W|S|y`t|YSw) a' Resolves the given lora_path to an absolute local path. If the lora_path is identified as a Hugging Face model identifier, it will download the model and return the local snapshot path. Otherwise, it treats the lora_path as a local file path and converts it to an absolute path. Parameters: lora_path (str): The path to the lora model, which can be an absolute path, a relative path, or a Hugging Face model identifier. Returns: str: The resolved absolute local path to the lora model. ~r)InvalidParametersnapshot_download) HTTPErrorcs dS)N)model_idr%r%rorrr%r&s z+get_adapter_absolute_path..z&Error downloading the ModelScope modelcs tjdS)N)repo_id)huggingface_hubrrr%)ror%r&rvs z'Error downloading the HuggingFace model)ospathisabsr\ expanduserexistsabspathrVLLM_USE_MODELSCOPE modelscope.hub.snapshot_downloadrqrrrequestsrsrrr2 exception)rorqrs download_fndownload_exceptions error_loglocal_snapshot_pathr%rur&get_adapter_absolute_paths,        rcCsHt|r t|}rt|}|jsdd|D|d<|Stdt|S)NcSs&g|]\}}}}d|vr|dqS)z..rJ)rstrip)r-_ weight_namer%r%r& s  z2process_packed_modules_mapping..expertszGTo support LoRA for MoE model, 'get_expert_mapping' must be implemented)r4rris_3d_moe_weightAttributeError)r)moe_packed_mappingpacked_modules_mappingr%r%r&process_packed_modules_mapping s  rr+)Irytypingrrrxhuggingface_hub.utilsrrtorchr transformersrvllmrvllm.config.lorar vllm.loggerr vllm.lora.layersr r r rrrrrrrrrrrrrr$vllm.model_executor.layers.fused_moer!vllm.model_executor.layers.linearrvllm.model_executor.utilsrr+vllm.model_executor.layers.logits_processorr 3vllm.model_executor.layers.vocab_parallel_embeddingr! vllm.model_executor.models.utilsr"__name__r2r$r'r(rhtype__annotations__Moduleboolr4intrlr?rGstrrRtuplercrfrnrdictrr%r%r%r&s      L            , $7