o ۾i) @sdZddlZddlZddlmZddlmZmZddlZddlm Z ddl m Z ddl m Z mZmZddlmZdd lmZmZdd lmZmZdd lmZmZdd lmZdd lmZddlm Z ee!Z"edddddddede#de$e j%dBde dBde j%f ddZ&de j%de dej'ddfddZ(edej j%dej'fdd Z)e*e+e,e$e j%e#ffZ- de de,e$e j%e#ffd!d"Z.de de,e$e j%e#ffd#d$Z/de de$e j%fd%d&Z0de de#fd'd(Z1eGd)d*d*Z2d+ede$e j%fd,d-Z3dS).z+Utilities for selecting and loading models.N)contextmanager) dataclassfield)nn) assert_never) ModelConfig VllmConfigset_current_vllm_config) init_logger) Attention MLAAttention)QuantizationConfigQuantizeMethodBase)record_metadata_for_reloadingset_torchao_reload_attrs) SupportsQuant) instrument)is_pin_memory_availablezInitialize model) span_name)prefix model_class model_config vllm_configrrrreturnc Cs|dur|j}|durt|\}}|jdurt|j|t|j}dd|jD}d|vrWd|vrWt |d|d|||d}t ||WdS1sRwYd }t j |t d d td |i} d|vrq|| d<d |vrz|j| d <d|vr|j| d<d|vr|j| d<d|vr|j| d<d|vr|j| d<t |d|d|di| }t |Wd|S1swY|S)z1Initialize a model with the given configurations.NcSsg|]}|jqS)name).0paramrrZ/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/utils.py 2sz$initialize_model..rrT) check_compiler)rra/vLLM model class should accept `vllm_config` and `prefix` as input arguments. Possibly you have an old-style model class registered from out of tree and it is used for new vLLM version. Check https://docs.vllm.ai/en/latest/design/arch_overview.html for the design and update the model class accordingly.) stacklevelz:Trying to guess the arguments for old-style model class %sconfig cache_config quant_config lora_configscheduler_configr)rget_model_architecturer&configure_quant_configinspect signature__init__ parametersvaluesr rwarningswarnDeprecationWarningloggerwarning hf_configr%r'r() rrrr_ signatures all_paramsmodelmsgkwargsrrrinitialize_model sR              r<r9 target_devicec Cs|D])\}}t|dd}t|tr-t|| ||Wdn1s(wYq|D]\}}t|ttfrHt|drH||j q2|j dkrUt ||dSdS)N quant_methodprocess_weights_after_loadingtorchao) named_modulesgetattr isinstancerdevice_loading_contextr?r r hasattrdtype quantizationr)r9rr=r6moduler>rrrr?\s       r?rHccs\|jdkr |VdSi}|D]\}}|jjdkr'|j||<|j||_qzE|VWt}|D]7\}}||vrk||}|jdkrdtj|j|j |jj |jj d|d}| |j||_q4|j||_q4dSt}|D]7\}}||vr||}|jdkrtj|j|j |jj |jj d|d}| |j||_qu|j||_quw)Ncpu)sizestriderFlayoutdevice pin_memory) typenamed_parametersrMdatatortorch empty_stridedrJrKrFrLcopy_)rHr=original_device_statesrprNoriginal_devicecpu_datarrrrDzs`       rDcCsddlm}m}t|jdg}|jj||d\}}||kr1|jdks&J|jdkr1t d||j }|dkr= ||fS|d krNt d ||}||fS|d kr_t d ||}||fSt |||fS) Nr)as_embedding_modelas_seq_cls_model architecturesrvllmautoz%s has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.noneembedzConverting to embedding model.classifyz,Converting to sequence classification model.)#vllm.model_executor.models.adaptersrZr[rBr5registryresolve_model_cls_get_transformers_backend_cls model_implr3 warning_once convert_type debug_oncer)rrZr[r\ model_clsarchrirrr_get_model_architectures6     rmc CsPt|j|j|j|j|jtt|jdgf}|t vrt |St |}|t |<|S)Nr\) hashr9ri runner_typetrust_remote_codergtuplerBr5_MODEL_ARCH_BY_HASHrm)rkey model_archrrrr)s r)cC t|dS)Nrr)r]rrr get_model_cls rwcCru)Nrvr]rrrget_architecture_class_namerxrzc@sreZdZUdZeeeefed<eedZ eee ee ffed<ddZ dede eeefd Bfd d Z d S) ParamMappingz A class to handle parameter mapping for model weight loading. It creates a bidirectional mapping between packed parameters and their constituent parts. packed_mapping)default_factoryinverse_packed_mappingcCsR|jD]!\}}t|dkr|d|krqt|D] \}}||f|j|<qqdS)Nryr)r|itemslen enumerater~)self packed_name sub_paramsindex param_namerrr __post_init__s zParamMapping.__post_init__ module_namerNcCs.|jD]\}}||r||fSqdS)N)r|rendswith)rrrsvaluerrrget_sub_moduless   zParamMapping.get_sub_modules)__name__ __module__ __qualname____doc__dictstrlist__annotations__rr~rqintrrrrrrr{s "& r{r&cCsNt|ts#t|dd}t|dd}|dur|||dur%||_dSdSdS)as Pass packed_modules_mapping by reference to quant_config so that quant_config can properly match fused modules Note that model attributes are passed by reference to quant_config, enabling them to be updated by model_class.__new__ (ex. chatglm, qwen) Once the `SupportsQuant` mixin has been added to all models, this function can be removed hf_to_vllm_mapperNpacked_modules_mapping) issubclassrrBapply_vllm_mapperr)r&rrr|rrrr*s    r*)4rr+r0 contextlibr dataclassesrrrSrtyping_extensionsr vllm.configrrr vllm.loggerr $vllm.model_executor.layers.attentionr r 3vllm.model_executor.layers.quantization.base_configr r'vllm.model_executor.model_loader.reloadrr%vllm.model_executor.models.interfacesr vllm.tracingrvllm.utils.platform_utilsrrr3rrOModuler<rMr?rDrrrqrrrmr)rwrzr{r*rrrrsp         ; )  #