o ॵi @sZddlZddlmZddlmZmZddlZddl m Z m Z ddl m Z GdddZdS)N)abstractmethod)ListUnion) read_configsnapshot_download)Configc@sveZdZdefddZedeeeeeefdeefddZ defd d Z e d efd d Z e dddZdS)InferFrameworkmodel_id_or_dircKstj|r ||_nt||_||}t|j}|d}|dur)|p(||}tj|jd}tj |rLt |}|d}|durL|pK||}|sUt d|dS)z Args: model_id_or_dir(`str`): The model id of the modelhub or a local dir containing model files. z model.typeNz config.json model_typez"Model accelerating not supported: ) ospathexists model_dirrmodel_type_supportedrsafe_getjoinisfiler from_file ValueError)selfr kwargsmodel_supportedconfigr config_filerX/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/pipelines/accelerate/base.py__init__ s.        zInferFramework.__init__promptsreturncKdS)z Args: prompts(`Union[List[str], List[List[int]]]`): The string batch or the token list batch to input to the model. Returns: The answers in list according to the input prompt batch. Nr)rrrrrr__call__)s zInferFramework.__call__r cCr)NFr)rr rrrr5sz#InferFramework.model_type_supported major_versioncCstj\}}||kS)z%Check the GPU compatibility. )torchcudaget_device_capability)r!major_rrrcheck_gpu_compatibility8sz&InferFramework.check_gpu_compatibilityvllmcKs<|dkrddlm}||fi|}||_|Std|)aInstantiate the model wrapped by an accelerate framework. Args: model_id_or_dir(`str`): The model id of the modelhub or a local dir containing model files. framework(`str`): The framework to use. Returns: The wrapped model. r()VllmzFramework not supported: )r(r* llm_frameworkr)clsr frameworkrr*r(rrrfrom_pretrained?s  zInferFramework.from_pretrainedN)r()__name__ __module__ __qualname__strrrrrintr r staticmethodr' classmethodr.rrrrr s  r)os.pathr abcrtypingrr torch.cudar" modelscoperrmodelscope.utils.configrrrrrrs