o ॵi @s<ddlmZmZddlmZddlmZGdddeZdS))ListUnion)InferFramework)is_vllm_availablec sneZdZ   ddedededeffdd Zd eeeeeefd eefd d ZdefddZ Z S)VllmautoNmodel_id_or_dirdtype quantizationtensor_parallel_sizecsVt|ts tdddlm}tds|dvrd}||j||d|d|_ d S) a Args: dtype: The dtype to use, support `auto`, `float16`, `bfloat16`, `float32` quantization: The quantization bit, default None means do not do any quantization. tensor_parallel_size: The tensor parallel size. zLInstall vllm by `pip install vllm` before using vllm to accelerate inferencer)LLM)bfloat16rfloat16T)r r trust_remote_coder N) super__init__r ImportErrorvllmr rcheck_gpu_compatibility model_dirmodel)selfr r r r r  __class__X/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/pipelines/accelerate/vllm.pyr s"    z Vllm.__init__promptsreturnc Ks|dd}|dd}|dd}|dd}|s"|dkr"d|d<|r.|t|d |d <|r4||d <d d lm}|di|}t|d trUd d |jj||dDSdd |jj||dDS)zGenerate tokens. Args: prompts(`Union[List[str], List[List[int]]]`): The string batch or the token list batch to input to the model. kwargs: Sampling parameters. do_sampleNnum_beamr max_lengthmax_new_tokensTuse_beam_searchr max_tokens)SamplingParamscSg|]}|jdjqSroutputstext.0outputrrr B z!Vllm.__call__..)sampling_paramscSr'r(r)r,rrrr/Gr0)prompt_token_idsr1r)poplenrr& isinstancestrrgenerate) rrkwargsr r!r"r#r&r1rrr__call__%s.     z Vllm.__call__ model_typecstfdddDS)Ncsg|]}|vqSr)lower)r-rr:rrr/Mr0z-Vllm.model_type_supported..) llamabaichuaninternlmmistralaquilabloomfalcongptmptoptqwenrA)any)rr:rr<rmodel_type_supportedLs zVllm.model_type_supported)rNr) __name__ __module__ __qualname__r6intrrrr9rI __classcell__rrrrrs" 'rN)typingrr$modelscope.pipelines.accelerate.basermodelscope.utils.import_utilsrrrrrrs