o ۾iW@sddlmZddlmZddlmZmZmZmZddl Z ddl m Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZd d lmZd d lmZmZerdddl mZm Z ddlmZede e j!dZ"ee#Z$gdZ%ddde j!dBfddZ&de j'de(ddde)fddZ*de(de(de(fddZ+d e"de"fd!d"Z,d#e"de"fd$d%Z-d#e"de"fd&d'Z.Gd(d)d)eZ/de j!fd*d+Z0ed,e)fd-d.Z1d/ee2e(e j3ffd0d1Z4d/ee2e(e j3ffd2d3Z5e4e5d4Z6d/ee2e(e j3ffd5d6Z7dS)7)Iterable)contextmanager) TYPE_CHECKINGAnyTypeVarcastN) VllmConfig) init_logger) get_act_fn)VerifyAndUpdateConfig)try_get_dense_modules)get_hf_file_bytes)supports_multimodal)VllmModelForPoolingis_pooling_model) ModelConfigr)Pooler_T)bound) ForCausalLMForConditionalGeneration ChatModel LMHeadModel model_configrreturncCst|j|jd}|durdSzBg}|D]2}|d}tj|d|d|dd|jd}t|||s2q|||d }rE|t |qtj |j |jd WSt y_t d YdSw) z3Load Sentence-Transformers Dense projection layers.)revisionNfolder in_features out_featuresbiasT)r dtypeactivation_function)r!zST projector loading failed)r modelrnnLinearget head_dtype_load_dense_weightsappendr Sequentialto Exceptionlogger exception)r dense_moduleslayers layer_configrlinearact_namer4W/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/adapters.py_load_st_projector's4     r6r2rc Csddlm}dD]}|r|d|n|}zht||j|j}|s"Wq|dr2ddlm}||}nddl} t j| |dd d }d D]8} | |vr{t |j d |} | |j || | d d} |jdurv| |vrvt |jd |} | |j|| Wd SqCWqtytd|YqwdS)z0Load weights using vLLM's weight_loader pattern.rdefault_weight_loader)zmodel.safetensorszpytorch_model.bin/z .safetensors)loadNcpuT) map_location weights_only)weightz linear.weightz dense.weight weight_loaderr>r zFailed to load %sF)-vllm.model_executor.model_loader.weight_utilsr8r r#rendswithsafetensors.torchr:iotorchBytesIOgetattrr>replacer r,r-r.)r2rrr8filename file_path file_bytesload_safetensors state_dictrC weight_keyr?bias_key bias_loaderr4r4r5r(GsH          r(orig_model_namepooling_suffixcCs |}tD]}||}q||SN)_GENERATE_SUFFIXES removesuffix)rPrQ model_namegenerate_suffixr4r4r5_get_pooling_model_namews rWorig_clscsNddlmddlmddlmmmGfddd|t}|S)Nr)LogitsProcessorParallelLMHeadr)AutoWeightsLoaderStageMissingLayerno_init_weightsc s|eZdZdZdddddededd ffd d Z ddddedd fd dZdee ee j fffdd Z Z S)z2_create_pooling_model_cls..ModelForPoolingTprefix vllm_configrrakwargsrNcs|fddfdtjd||d|Wdn1s$wY||_t|dd}|sDt|rD|}t|drD|j}|sM|j||d}||_dS)Ncs d|S)Noutputr4)mod)r]r4r5s zM_create_pooling_model_cls..ModelForPooling.__init__..)targets)rbrapoolerr`r4) super__init__rbrFrget_language_modelhasattrrh _init_pooler)selfrbrarcrhlanguage_model)rYr[r] __class__r^r4r5rjs      z;_create_pooling_model_cls..ModelForPooling.__init__rcSstrR)NotImplementedError)rnrbrar4r4r5rmsz?_create_pooling_model_cls..ModelForPooling._init_poolerweightsc stddg}dttttjf}|D]#\}||fztfdd|DWn t y;YqwrZ} dD] }|rNt |}qEt d|fddg||RD}fdd }t td |} | |S) Nr_zmodel.c3s |] }|vr|VqdSrRr4).0ra)name params_dictr4r5 s zR_create_pooling_model_cls..ModelForPooling.load_weights...zGMapping weights to %s as they are relative to this model instead of %s.c3s |] \}}||fVqdSrRr4)rsrtr>) target_prefixr4r5rvs  cs}||SrR) load_weights)rrloader)r\rnr4r5default_load_weightss z]_create_pooling_model_cls..ModelForPooling.load_weights..default_load_weightsry)dictnamed_parameterslisttuplestrrDTensorr)next StopIterationsplitrFr-info _get_nameri) rnrrcandidate_prefixes seen_weights loaded_weight target_modelattrmapped_weightsr{ry)r\rp)rtrurnrxr5rys>      z?_create_pooling_model_cls..ModelForPooling.load_weightsr_)__name__ __module__ __qualname__rrrrjrmrrrDrry __classcell__r4r\rYr[r]r^rpr5ModelForPoolings*! *r) +vllm.model_executor.layers.logits_processorrY3vllm.model_executor.layers.vocab_parallel_embeddingr[utilsr\r]r^r)rXrr4rr5_create_pooling_model_clss  XrclscsBt|r|SddlmGfdddt|}t|jd|_|S)a\ Subclass an existing vLLM model to support embeddings. By default, the embeddings of the whole prompt are extracted from the normalized hidden state corresponding to the last token. Note: We assume that no extra layers are added to the original model; please implement your own model if this is not the case. rDispatchPoolercs*eZdZ d dddeddffdd Zd S) z-as_embedding_model..ModelForEmbeddingr_rbrrarrcs|jj}|dus J|SrR)r pooler_config for_embedding)rnrbrarrr4r5rms  z:as_embedding_model..ModelForEmbedding._init_poolerNr)rrrrrmr4rr4r5ModelForEmbeddingsr ForEmbedding)r!vllm.model_executor.layers.poolerrrrWr)rrr4rr5as_embedding_models    rcslt|r|Sddlmddlmddlm}ddlmGfdddt ||}t |j d |_ |S) a Subclass an existing vLLM model to support classify and score tasks. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token. Note: We assume that the classification head is a single linear layer stored as the attribute `score` of the top-level model; please implement your own model if this is not the case. r)ReplicatedLinearr)SupportsCrossEncodingr) maybe_prefixcsReZdZ d dddeddffdd Zd eeeejfffd d Z Z S) z8as_seq_cls_model..ModelForSequenceClassificationr_rbrrarrc sd|jj}|j}|j}||jd|jj|d|dd|_|jj}|dus*Jj ||jdS)NFscore)r params_dtype quant_config return_biasra) classifier) r hf_configget_text_configrget_hidden_size num_labelsr'rr for_seq_cls)rnrbra text_configrrrrrrr4r5rms  zEas_seq_cls_model..ModelForSequenceClassification._init_poolerrrcspj}|}t|dt|dd}t|dt|dd}fdd}||}|dur3|dur3t|St|S)Nclassifier_from_tokenmethodc3sd|D],\}}|dkr*jjj}jjj}|||}tj|j_dj_ q||fVqdS)Nz score.biasF) rr>devicer!r+rDr$ Parameterr skip_bias_add)rrrtr>rr!r rnr4r5auto_set_score_bias@s     zbas_seq_cls_model..ModelForSequenceClassification.load_weights..auto_set_score_bias)configrrFriryseq_cls_model_loader)rnrrrrtokensrrrrr5ry6s     zEas_seq_cls_model..ModelForSequenceClassification.load_weightsr) rrrrrmrrrDrryrr4rrr5ModelForSequenceClassifications(rForSequenceClassification) r!vllm.model_executor.layers.linearrrr%vllm.model_executor.models.interfacesrrrrrWr)rrrr4rr5as_seq_cls_models     8rc@seZdZedddZdS)SequenceClassificationConfigrbrrNcCs|jj}|}t|dt|dd}t|dt|dd}|dur"dS|dus(J|tvs4Jd|d|dkrGt|dks@Jd|_d|_n t||_t||_t|dd }||_dS) Nrrmethod  not supportedfrom_2_way_softmaxr use_sep_tokenF)rrrrFSEQ_CLS_LOAD_METHODSlenrr)rbrrrrrr4r4r5verify_and_update_config[s(      z5SequenceClassificationConfig.verify_and_update_config)rbrrN)rrr staticmethodrr4r4r4r5rZsrcCst|rz |}||ur|WSWn tyYnwdD]}t||r. get_tokenizerrtokenizer_modetrust_remote_coderr?language_model.score.weight score.weightlm_head.weighthf_to_vllm_mapper)-rr[r@r8rbrrrrrFrr~intrrrl vocab_size hidden_sizelm_headtie_word_embeddingsr#rget_input_embeddings tie_weightsrrr__mro__ryvllm.tokenizersr tokenizertokenizer_revisionrrconvert_tokens_to_idsr>datar+rDfloat32radd _map_namediscard)r#rrr[r8rrrrrrorusing_vlm_head text_backbonerpooling_model_clsloaded_weightsrrfalse_idtrue_idlm_head_weight score_weight score_layerparamr?score_weight_name lm_head_namerr4r4r5%load_weights_using_from_2_way_softmaxsx             r csddlm}ddlm}|jj}|jj}|j}t |dg}t t t |}t |dks.Jt|}||u} | o.rrcsg|]}|qSr4)r)rstrr4r5 @sz3load_weights_no_post_processing..r?rrrr))rr[r@r8rbrrrrrFrr~rrrrlrrrrr#rrrrrrrryrrrrrrr>rrrrr)r#rrr[r8rrrrrorrrrrrr token_idsrrrr?r r rr4r r5load_weights_no_post_processingsb             r)rno_post_processingcCsL|jjj}|}t|dt|dd}|tvsJd|dt|||S)Nrrr)rbrrrrFr)r#rrrrrr4r4r5r\s r)8collections.abcr contextlibrtypingrrrrrDtorch.nnr$ vllm.configr vllm.loggerr %vllm.model_executor.layers.activationr !vllm.model_executor.models.configr vllm.transformers_utils.configr "vllm.transformers_utils.repo_utilsr interfacesrinterfaces_baserrrrrrrrrr-rSr6r%rboolr(rWrrrrrrrrr rrrr4r4r4r5sZ             0 b"V!$ RF