o i@s<ddlZddlmZmZmZddlmZGdddeZdS)N)pipelineTurbomindEngineConfigGenerationConfig) BaseModelc@s<eZdZ   d ddZ   dd d Z   dd d ZdS) LMDeployModelcudadNcKsP|dksJd|dtjdj}t|d}|r|nd}t|d|d|_dS)Nrzclmdeploy only supports cuda devices, consider changing device or using a different backend instead.i)cache_max_entry_countzekwek/Soprano-1.1-80MERROR) log_levelbackend_config)torchrget_device_properties total_memoryrr)selfdevice cache_size_mb model_pathkwargscache_size_ratior model_name_or_pathrM/home/ubuntu/.local/lib/python3.10/site-packages/soprano/backends/lmdeploy.py__init__s   zLMDeployModel.__init__ffffff?333333?333333?c CsHtdd|||dd}|j||d}g}|D] }||j|jdq|SN generationTi)output_last_hidden_state do_sampletop_p temperaturerepetition_penaltymax_new_tokens) gen_config) finish_reason hidden_state)rrappendr'last_hidden_state) rpromptsr"r#r$r& responsesresresponserrrinfers zLMDeployModel.inferccsFtdd|||dd}|jj|g|d}|D] }|j|jdVqdSr)rr stream_inferr'r*)rpromptr"r#r$r&r,r.rrrr0+s zLMDeployModel.stream_infer)rr N)rrr)__name__ __module__ __qualname__rr/r0rrrrrs  r)rlmdeployrrrbaserrrrrrs