o }oi(@s(ddlZddlZddlZddlZddlmZmZddlm Z m Z ddl m Z ddl mZddlmZGddde ZeZeZGd d d e ZGd d d eZGd ddeZedddZedddZddZddZddZeddefddZdd Zed!defd"d#Z dS)$N)FastAPI HTTPException) BaseModelmodel_validator) BaseSettings)NemoQueryLLMPyTorch)loggingcsJeZdZUdZeed<eed<fddZeddZ edd Z Z S) TritonSettingsz[ TritonSettings class that gets the values of TRITON_HTTP_ADDRESS and TRITON_PORT. _triton_service_port_triton_service_ipc sltt|zttjdd|_tjdd|_WdSt y5}z t d|WYd}~dSd}~ww)N TRITON_PORTi@TRITON_HTTP_ADDRESSz0.0.0.0zQAn exception occurred trying to retrieve set args in TritonSettings class. Error:) superr __init__intosenvirongetr r Exceptionrerror)selfr __class__e/home/ubuntu/.local/lib/python3.10/site-packages/nemo/deploy/service/fastapi_interface_to_pytriton.pyr s zTritonSettings.__init__cC|jS)zA Returns the port number for the Triton service. )r rrrrtriton_service_port)z"TritonSettings.triton_service_portcCr)z@ Returns the IP address for the Triton service. )r rrrrtriton_service_ip0rz TritonSettings.triton_service_ip) __name__ __module__ __qualname____doc__r__annotations__strrpropertyrr __classcell__rrrrr s   r c@s\eZdZUdZeed<dZeed<dZe ed<dZ e ed<d Z eed <e d d d dZ dS) BaseRequesta Common parameters for completions and chat requests for the server. Attributes: model (str): The name of the model to use for completion. max_tokens (int): The maximum number of tokens to generate in the response. temperature (float): Sampling temperature for randomness in generation. top_p (float): Cumulative probability for nucleus sampling. top_k (int): Number of highest-probability tokens to consider for sampling. modeli max_tokensg? temperaturegtop_prtop_kafter)modecCs(|jdkr|jdkrtdd|_|S)z(Validate parameters for greedy decoding.rzOBoth temperature and top_p are 0. Setting top_k to 1 to ensure greedy sampling.)r+r,rwarningr-rrrrset_greedy_paramsNs zBaseRequest.set_greedy_paramsN)r r!r"r#r%r$r*rr+floatr,r-rr2rrrrr(<s     r(c@s2eZdZUdZeed<dZeed<dZe ed<dS)CompletionRequesta4 Represents a request for text completion. Attributes: prompt (str): The input text to generate a response from. logprobs (int): Number of log probabilities to include in the response, if applicable. echo (bool): Whether to return the input text as part of the response. promptNlogprobsFecho) r r!r"r#r%r$r6rr7boolrrrrr4Ws  r4c@seZdZUdZeeed<dS)ChatCompletionRequesta Represents a request for chat completion. Attributes: messages (list[dict]): A list of message dictionaries for chat completion. logprobs (bool): Whether to return log probabilities for output tokens. top_logprobs (int): Number of log probabilities to include in the response, if applicable. logprobs must be set to true if this parameter is used. messagesN)r r!r"r#listdictr$rrrrr9fs  r9z /v1/healthcCsddiS)z Health check endpoint to verify that the API is running. Returns: dict: A dictionary indicating the status of the application. statusokrrrrr health_checktsr?z/v1/triton_healthc sdtjdttjd}td|ztj|dd}|jdkr(dd iWSt d d d tj yE}z t d d t|d d}~ww)aB This method exposes endpoint "/triton_health" which can be used to verify if Triton server is accessible while running the REST or FastAPI application. Verify by running: curl http://service_http_address:service_port/v1/triton_health and the returned status should inform if the server is accessible. http://:z/v2/health/readyz+Attempting to connect to Triton server at: )timeoutr=z$Triton server is reachable and readyizTriton server is not ready) status_codedetailzCannot reach Triton server: N) triton_settingsrr%rrinforequestsrrErRequestException) triton_urlresponseerrrcheck_triton_healths    rNcCsLt|tjr |St|trdd|DSt|tr$dd|DS|S)z1 Convert NumPy arrays in output to lists cSsi|] \}}|t|qSr convert_numpy).0kvrrr sz!convert_numpy..cSsg|]}t|qSrrO)rQirrr sz!convert_numpy..) isinstancenpndarraytolistr<itemsr;)objrrrrPs   rPc Cs.t||d} | j|||||||| d| d } | S)zn run_in_executor doesn't allow to pass kwargs, so we have this helper function to pass args as a list )url model_namei,) promptsr+r-r,compute_logprob max_lengthapply_chat_templaten_top_logprobs init_timeoutr7)r query_llm) r]r)r_r+r-r,r`rarbrcr7nqoutputrrr _helper_funs  rhc srddl} ddl} | } | j}| |t|||||||||| | IdH}Wd|S1s2wY|S)z Sends requests to `NemoQueryLLMPyTorch.query_llm` in a non-blocking way, allowing the server to process concurrent requests. This way enables batching of requests in the underlying Triton server. rN)asyncio concurrentget_event_loopfuturesThreadPoolExecutorrun_in_executorrh)r]r)r_r+r-r,r`rarbrcr7rirjlooppoolresultrrrquery_llm_asyncs0  rrz/v1/completions/requestc sZdtjdtj}td||j}t|jts |jg}t||j ||j |j |j |j duo4|j dk|jd|j |jd IdH}t|}|ddd dd|ddd <|j dur|j dkr|ddd d d|ddd d <|ddd d d|ddd d <|jr|ddd d ddnd|ddd <td ||S)z] Defines the completions endpoint and queries the model deployed on PyTriton server. r@rA Request: NrF r]r)r_r+r-r,r`rarbrcr7choicestextr6token_logprobs top_logprobsOutput: )rGrrrrHr5rWr;rrr)r+r-r,r6r*r7rPinsert)rsr]r_rgoutput_serializablerrrcompletions_v1sP  $ r}cCs t|S)z Serializes dict to str )jsondumps)r:rrr dict_to_strs rz/v1/chat/completions/c sdtjdtj}td||j}t|jts |jg}t|g}t ||j ||j |j |j d|jdddd IdH}d|d d d d |d d d <d|d<d|d d d<|d d d =t|}|d d d dd d |d d d d<td||S)zb Defines the chat completions endpoint and queries the model deployed on PyTriton server. r@rArtFTNru assistantrvrrw)rolecontentmessagezchat.completionobjectr6rrz)rGrrrrHr:rWr;rrrr)r+r-r,r*rP)rsr]r_ json_promptsrgr|rrrchat_completions_v1s@   "r)!r~rnumpyrXrIfastapirrpydanticrrpydantic_settingsrnemo.deploy.nlpr nemo.utilsrr apprGr(r4r9rr?rNrPrhrrpostr}rrrrrrs6        )*