o -it. @sddlmZddlmZddlmZddlmZddlm Z m Z m Z m Z m Z ddlmZmZddlmZmZddlmZdd lmZdd lmZdd lmZmZdd lmZeeZ Gd ddZ!dej"fde#de#dede fddZ$dS))Lock) defaultdict) HTTPStatus) EngineClient) ErrorInfo ErrorResponse ModelCard ModelListModelPermission) BaseModelPathLoRAModulePath)LoadLoRAAdapterRequestUnloadLoRAAdapterRequest)sanitize_message) init_logger) LoRARequest) LoRAResolverLoRAResolverRegistry) AtomicCountercseZdZdZdddedeedeedBffddZd d Z d e fd d Z d de dBd e fddZd efddZ d dede dBd ee BfddZded ee BfddZded edBfddZded edBfddZde d e eBfddZZS)!OpenAIServingModelszShared instance to hold data about the loaded base model(s) and adapters. Handles the routes: - /v1/models - /v1/load_lora_adapter - /v1/unload_lora_adapter N) lora_modules engine_clientbase_model_pathsrcst||_||_||_i|_td|_g|_t D] }|j t |qt t|_|jj|_|jj|_|jj|_|jj|_|jj|_dS)Nr)super__init__rrstatic_lora_modules lora_requestsrlora_id_counterlora_resolversrget_supported_resolversappend get_resolverrrlora_resolver_lockinput_processor io_processorrenderer model_config max_model_len)selfrrrlora_resolver_name __class__c/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/entrypoints/openai/models/serving.pyr's"        zOpenAIServingModels.__init__cs\|jdurdS|jD] }t|j|jd}|j||jdIdH}t|tr+t|j j q dS)zALoads all static LoRA modules. Raises if any fail to loadN) lora_path lora_name)requestbase_model_name) rr pathnameload_lora_adapterr1 isinstancer ValueErrorerrormessage)r(lora load_request load_resultr,r,r-init_static_lorasDs    z%OpenAIServingModels.init_static_lorasreturncstfdd|jDS)Nc3s|]}|jkVqdSN)r3).0model model_namer,r- Tsz4OpenAIServingModels.is_base_model..)anyr)r(rBr,rAr- is_base_modelSsz!OpenAIServingModels.is_base_model lora_requestcCs|dur|jS|jdjS)a&Returns the appropriate model name depending on the availability and support of the LoRA or base model. Parameters: - lora: LoRARequest that contain a base_model_name. Returns: - str: The name of the base model or the first available model path. Nr)r/rr3)r(rFr,r,r-rBVs zOpenAIServingModels.model_namecsBfddjD}fddjD}||t|dS)zLShow available models. This includes the base model and all adapterscs&g|]}t|jj|jtgdqS))idr'root permission)rr3r' model_pathr )r? base_modelr(r,r- esz=OpenAIServingModels.show_available_models..cs8g|]}t|j|j|jr|jnjdjtgdqS)r)rGrHparentrI)rr/r2r1rr3r )r?r9rLr,r-rMns  )data)rrvaluesextendr )r( model_cards lora_cardsr,rLr-show_available_modelsbs    z)OpenAIServingModels.show_available_modelsr0r1c sr|j}|j|4IdH||IdH}|dur&|WdIdHS|j}||jvr4|j|jn|jd}t||||j d}|durO| |rO||_ z |j |IdHWn5ty}z)d} tj} dt|vrrd} tj} tt|| | dWYd}~WdIdHSd}~ww||j|<td||d|d WdIdHS1IdHswYdS) N)r/ lora_int_idr. load_inplaceBadRequestErrorzNo adapter found NotFoundErrorr8err_type status_codez-Loaded new LoRA adapter: name '%s', path '%s'Success: LoRA adapter 'z' added successfully.)r/r" _check_load_lora_adapter_requestr.rrVrincrrWrEr1radd_lora Exceptionr BAD_REQUESTstr NOT_FOUNDcreate_error_responseloggerinfo) r(r0r1r/error_check_retr.rVrFe error_typer\r,r,r-r4|sP      0z%OpenAIServingModels.load_lora_adapterc s|j}|j|4IdH3||IdH}|dur&|WdIdHS|j|=td|d|dWdIdHS1IdHsFwYdS)NzRemoved LoRA adapter: name '%s'r]z' removed successfully.)r/r""_check_unload_lora_adapter_requestrrfrg)r(r0r/rhr,r,r-unload_lora_adapters  0z'OpenAIServingModels.unload_lora_adaptercsN|jr|jstddtjdS|js%|j|jvr%td|jddtjdSdS)Nz2Both 'lora_name' and 'lora_path' must be provided.InvalidUserInputrZThe lora adapter 'z`' has already been loaded. If you want to load the adapter in place, set 'load_inplace' to True.)r/r.rerrbrWrr(r0r,r,r-r^s  z4OpenAIServingModels._check_load_lora_adapter_requestcsB|js tddtjdS|j|jvrtd|jddtjdSdS)Nz:'lora_name' needs to be provided to unload a LoRA adapter.rmrZrnz' cannot be found.rY)r/rerrbrrdror,r,r-rks  z6OpenAIServingModels._check_unload_lora_adapter_requestr/c s|j|4IdH||jvr|j|WdIdHS|jj}|jd}d}|jD]W}|||IdH}|durd}||_z&|j |IdH||j|<t d||j j|WWdIdHSty}zt d||j j|WYd}~q.d}~wwq.|rtd|dd tjd WdIdHStd |d d tjd WdIdHS1IdHswYdS)adAttempt to resolve a LoRA adapter using available resolvers. Args: lora_name: Name/identifier of the LoRA adapter Returns: LoRARequest if found and loaded successfully. ErrorResponse (404) if no resolver finds the adapter. ErrorResponse (400) if adapter(s) are found but none load. NrUFTz.Resolved and loaded LoRA adapter '%s' using %szBFailed to load LoRA '%s' resolved by %s: %s. Trying next resolver.zLoRA adapter 'z$' was found but could not be loaded.rXrZz LoRA adapter z does not existrY)r"rr&r@rr_r resolve_lorarVrr`rfrgr+__name__ BaseExceptionwarningrerrbrd)r(r/r1 unique_id found_adapterresolverrFrir,r,r-rps^      / 0z OpenAIServingModels.resolve_lorar>)rq __module__ __qualname____doc__rlistr r rr<boolrErrcrBr rTr rr4rrlr^rkrp __classcell__r,r,r*r-rsL    /   rrXr8r[r\r=cCsttt|||jddS)N)r8typecode)r7)rrrvaluerZr,r,r-re)sreN)%asyncior collectionsrhttprvllm.engine.protocolr'vllm.entrypoints.openai.engine.protocolrrrr r 'vllm.entrypoints.openai.models.protocolr r $vllm.entrypoints.serve.lora.protocolr rvllm.entrypoints.utilsr vllm.loggerrvllm.lora.requestrvllm.lora.resolverrrvllm.utils.counterrrqrfrrbrcrer,r,r,r-s6