o }oi@sddlZddlmZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZGdd d e ZeZeZGd d d eZed d dZedddZeddefddZdS)N)Path)FastAPI HTTPException) BaseModel) BaseSettings) NemoQueryLLM)loggingcsreZdZUeed<eed<eed<fddZeddZedd Z ed d Z ed d Z eddZ Z S)TritonSettings_triton_service_port_triton_service_ip_triton_request_timeoutc stt|z7ttjdd|_tjdd|_ttjdd|_ tjdd d k|_ tjd d d k|_ WdSt yW}z td |WYd}~dSd}~ww) N TRITON_PORTiTRITON_HTTP_ADDRESSz0.0.0.0TRITON_REQUEST_TIMEOUT<OPENAI_FORMAT_RESPONSEFalsetrueOUTPUT_GENERATION_LOGITSzQAn exception occurred trying to retrieve set args in TritonSettings class. Error:)superr __init__intosenvirongetr r r lower_openai_format_response_output_generation_logits Exceptionrerror)selfr __class__V/home/ubuntu/.local/lib/python3.10/site-packages/nemo/deploy/service/rest_model_api.pyrs zTritonSettings.__init__cC|jSN)r r r#r#r$triton_service_port(z"TritonSettings.triton_service_portcCr%r&)r r'r#r#r$triton_service_ip,r)z TritonSettings.triton_service_ipcCr%r&)r r'r#r#r$triton_request_timeout0r)z%TritonSettings.triton_request_timeoutcCr%)zd Retuns the response from Triton server in OpenAI compatible format if set to True. )rr'r#r#r$openai_format_response4z%TritonSettings.openai_format_responsecCr%)zf Retuns the generation logits along with text in Triton server output if set to True. )rr'r#r#r$output_generation_logits;r-z'TritonSettings.output_generation_logits)__name__ __module__ __qualname__r__annotations__strrpropertyr(r*r+r,r. __classcell__r#r#r!r$r s      r c@sveZdZUeed<eed<dZeed<dZeed<dZ eed<d Z eed <d Z e ed <d Z ed Bed<dZeed<d S)CompletionRequestmodelprompti max_tokensg? temperaturegtop_ptop_kFstreamNstopfrequency_penalty)r/r0r1r3r2r9rr:floatr;r=r>boolr?r@r#r#r#r$r6Gs      r6z /v1/healthcCsddiS)Nstatusokr#r#r#r#r$ health_checkSsrEz/v1/triton_healthc sdtjdttjd}td|ztj|dd}|jdkr(dd iWSt d d d tj yE}z t d d t|d d}~ww)a: This method exposes endpoint "/triton_health" which can be used to verify if Triton server is accessible while running the REST or FastAPI application. Verify by running: curl http://service_http_address:service_port/v1/triton_health and the returned status should inform if the server is accessible. zhttp://:z/v2/health/readyz+Attempting to connect to Triton server at: )timeoutrCz$Triton server is reachable and readyizTriton server is not ready) status_codedetailzCannot reach Triton server: N) triton_settingsr*r3r(rinforequestsrrJrRequestException) triton_urlresponseer#r#r$check_triton_healthXs   rSz/v1/completions/requestc Csz6tjdttj}t||jd}|j|jg|j|j |j |j tj tj tjd}tj r.|WSd|ddiWStyR}ztd|ddiWYd}~Sd}~ww) NrF)url model_name)promptsmax_output_lenr=r;r: init_timeoutr,r.outputrzIAn exception occurred with the post request to /v1/completions/ endpoint:rzAn exception occurred)rLr*r3r(rr7 query_llmr8r9r=r;r:r+r,r.rrr)rTrUnqrZrr#r#r$completions_v1ls,   r])rpathlibrrNfastapirrpydanticrpydantic_settingsrnemo.deploy.nlpr nemo.utilsrr apprLr6rrErSpostr]r#r#r#r$s$     ,