o }oiG[@sddlZddlZddlmZddlmZmZddlZddl m Z dZ z ddl m Z mZWn ey7dZ YnwGdd d eZGd d d eZGd d d eZGdddeZdS)N)ABC)ListOptional)str_list2numpyT)DecoupledModelClient ModelClientFc@seZdZdZddZdS)NemoQueryLLMBasez Abstract base class for querying a Large Language Model (LLM). Args: url (str): The URL of the inference server. model_name (str): The name of the model to be queried. cCs||_||_dS)Nurl model_nameselfr r rM/home/ubuntu/.local/lib/python3.10/site-packages/nemo/deploy/nlp/query_llm.py__init__(s zNemoQueryLLMBase.__init__N)__name__ __module__ __qualname____doc__rrrrrrs rc"seZdZdZfddZ               ddeedeed ee d ee d ee d ee d eedeedeedeeedee dee dedee de deef ddZ Z S)NemoQueryLLMPyTorcha Sends a query to Triton for LLM inference Example: from nemo.deploy import NemoTritonQueryLLMPyTorch nq = NemoTritonQueryLLMPyTorch(url="localhost", model_name="GPT-2B") prompts = ["hello, testing GPT inference", "another GPT inference test?"] output = nq.query_llm( prompts=prompts, max_length=100, top_k=1, top_p=0.0, temperature=0.0, ) print("prompts: ", prompts) ctj||ddSNr superrr  __class__rrrA zNemoQueryLLMPyTorch.__init__NFN@prompts use_greedy temperaturetop_ktop_prepetition_penaltyadd_BOS all_probscompute_logprob end_strings min_length max_lengthapply_chat_templaten_top_logprobs init_timeoutechocCs<t|}d|i}|durtj|j|tjd|d<|dur(tj|j|tjd|d<|dur8tj|j|tjd|d<|durHtj|j|tjd|d<|durXtj|j|tjd|d<|durhtj|j|tjd|d <|durxtj|j|tjd|d <| durtj|j| tjd|d <| durt| |d <| durtj|j| tjd|d <| durtj|j| tjd|d<| durtj|j| tjd|d<|durtj|j|tjd|d<|durtj|j|tjd|d<t|j|j |dd}|j d$i|}|j j dj }d}d|vr |d}d}d|vr|d}|tjkrd|vr(|d}n WddStj|dd}dttdtt|j d|igd}|duri|ddd <||ddd d!<|durd"d#|D}||ddd d<|WdS|dWdS1swYdS)%a Query the Triton server synchronously and return a list of responses. Args: prompts (List(str)): list of sentences. use_greedy (bool): use greedy sampling, effectively the same as top_k=1 temperature (float): A parameter of the softmax function, which is the last layer in the network. top_k (int): limits us to a certain number (K) of the top tokens to consider. top_p (float): limits us to the top tokens within a certain probability mass (p). repetition_penalty (float): penalty applied to repeated sequences, 1.0 means no penalty. add_BOS (bool): whether or not to add a BOS (beginning of sentence) token. all_probs (bool): when using compute_logprob, returns probabilities for all tokens in vocabulary. compute_logprob (bool): get back probabilities of all tokens in the sequence. end_strings (List(str)): list of strings which will terminate generation when they appear in the output. min_length (int): min generated tokens. max_length (int): max generated tokens. apply_chat_template (bool): applies chat template if its a chat model. Default: False init_timeout (flat): timeout for the connection. rNdtyperr r!r"r#r$r%r&r'r(r)r*r+r-iX)init_timeout_sinference_timeout_sr log_probs top_logprobs sentencesUnknown output keyword.bytesutf-8cmpl-text_completiontextidobjectcreatedmodelchoicesr@logprobstoken_logprobscSsg|] }t|dqS)r)jsonloads).0 top_log_probrrr sz1NemoQueryLLMPyTorch.query_llm..r)rnpfullshapebool_singleint_rr r infer_batch model_configoutputsr/keysbytes_chardecodeastypeinttime)r rrr r!r"r#r$r%r&r'r(r)r*r+r,r-inputsclient result_dict output_typelog_probs_outputtop_log_probs_outputoutputr4openai_responsen_log_probs_outputrrr query_llmIs|&      &&zNemoQueryLLMPyTorch.query_llm)NNNNNNNNNNNFNrN rrrrrrstrrboolfloatrVra __classcell__rrrrr-sf       rcseZdZdZfddZ             ddeedeedee d ee d ee d ee d eed eedeedeedeeedee dee de fddZ Z S)NemoQueryLLMHFa Sends a query to Triton for LLM inference Example: from nemo.deploy import NemoQueryLLMHF nq = NemoQueryLLMHF(url="localhost", model_name="GPT-2B") prompts = ["hello, testing GPT inference", "another GPT inference test?"] output = nq.query_llm( prompts=prompts, max_length=100, top_k=1, top_p=0.0, temperature=0.0, ) print("prompts: ", prompts) crrrr rrrrrzNemoQueryLLMHF.__init__Nrrrr r!r"r#r$r% output_logits output_scoresr'r(r)r,cCst|}d|i}|durtj|j|tjd|d<|dur(tj|j|tjd|d<|dur8tj|j|tjd|d<|durHtj|j|tjd|d<|durXtj|j|tjd|d<|durhtj|j|tjd|d <|durxtj|j|tjd|d <| durtj|j| tjd|d <| durtj|j| tjd|d <| durt| |d <| durtj|j| tjd|d<| durtj|j| tjd|d<t|j|j |d{}|j di|}|j j dj }|tjkr;d|vr|d}n WddStj|dd}dttdtt|j d|igd}| r$d|vr$|d|d<| r2d|vr2|d|d<|WdS|dWdS1sJwYdS)a] Query the Triton server synchronously and return a list of responses. Args: prompts (List[str]): list of sentences. use_greedy (Optional[bool]): use greedy sampling, effectively the same as top_k=1 temperature (Optional[float]): A parameter of the softmax function, which is the last layer in the network. top_k (Optional[int]): limits us to a certain number (K) of the top tokens to consider. top_p (Optional[float]): limits us to the top tokens within a certain probability mass (p). repetition_penalty (Optional[float]): penalty applied to repeated sequences, 1.0 means no penalty. add_BOS (Optional[bool]): whether or not to add a BOS (beginning of sentence) token. all_probs (Optional[bool]): when using compute_logprob, returns probabilities for all tokens in vocabulary. output_logits (Optional[bool]): whether to return logits for each token output_scores (Optional[bool]): whether to return scores for each token end_strings (Optional[List[str]]): list of strs which will stop generation when they appear in the output. min_length (Optional[int]): min generated tokens. max_length (Optional[int]): max generated tokens. init_timeout (float): timeout for the connection. rNr.rr r!r"r#r$r%rhrir'r(r)r0rr4r5r6r7r8r9r:r;logitsscoresr)rrHrIrJrKrLrMrr r rNrOrPr/rRrQrSrTrUrVrW)r rrr r!r"r#r$r%rhrir'r(r)r,rXrYrZr[r^r4r_rrrrasd$        &zNemoQueryLLMHF.query_llm) NNNNNNNNNNNNrrbrrrrrgsZ       rgcseZdZdZfddZ                     ddeded ed ed ed ed edefddZ           dddZZ S) NemoQueryLLMa Sends a query to Triton for LLM inference Example: from nemo.deploy import NemoQueryLLM nq = NemoQueryLLM(url="localhost", model_name="GPT-2B") prompts = ["hello, testing GPT inference", "another GPT inference test?"] output = nq.query_llm( prompts=prompts, max_output_len=100, top_k=1, top_p=0.0, temperature=0.0, ) print("prompts: ", prompts) crrrr rrrrErzNemoQueryLLM.__init__NrFrr#r$r%r&openai_format_responseoutput_context_logitsoutput_generation_logitscCst|}d|i}|durtj|j|tjd|d<|dur(tj|j|tjd|d<|dur8tj|j|tjd|d<|durHtj|j|tjd|d<| durXtj|j| tjd|d<| durhtj|j| tjd|d <|durrt||d <|dur|t||d <|durtj|j|tjd|d <| durtj| d } t|jdt| gf| |d<| durtj| d } t|jdt| f| |d<| durtj|j| tj d|d<|durtj|j|tjd|d<|durtj|j|tj d|d<|durtj|j|tj d|d<|durtj|j|tj d|d<|durt||d<|dur/tj|j|tj d|d<|dur@tj|j|tj d|d<t |j |j |d}|j d%i|}|jjdj}|tjkrd|vrk|d}nd|vrw|d}n WddStj|dd }|rdttdtt|j d |igd!}|r|d"|d#dd"<|r|d$|d#dd$<|WdS|WdS|dWdS1swYdS)&a Query the Triton server synchronously and return a list of responses. Args: prompts (List(str)): list of sentences. max_output_len (int): max generated tokens. top_k (int): limits us to a certain number (K) of the top tokens to consider. top_p (float): limits us to the top tokens within a certain probability mass (p). temperature (float): A parameter of the softmax function, which is the last layer in the network. random_seed (int): Seed to condition sampling. stop_words_list (List(str)): list of stop words. bad_words_list (List(str)): list of bad words. no_repeat_ngram_size (int): no repeat ngram size. task_id (str): downstream task id if virtual tokens are used. init_timeout (flat): timeout for the connection. openai_format_response: return response similar to OpenAI API format output_generation_logits: return generation logits from model on PyTriton rNr.min_output_lenmax_output_lenr!r"r random_seedstop_words_listbad_words_listno_repeat_ngram_sizer7rtask_id lora_uidsrr#r$r%r&r'rorprjrPr4r5r6r8r9r:r;generation_logitsr@context_logitsr)rrHrIrJrMrLrSencodelenrKrr r rNrOrPr/rRrQrTrUrVrW)r rrtrurvrqrrr!r"r rsrwrxrr#r$r%r&r'r,rnrorprXrYrZr[r^r4r_rrrraKs,              &zNemoQueryLLM.query_llm?c cs8t|}d|i} |durtj|j|tjd| d<|dur)tj|j|tjd| d<|dur9tj|j|tjd| d<|durItj|j|tjd| d<| durYtj|j| tjd| d<|durstj|d }t|jd t|f|| d <|durtj|d }t|jd t|f|| d <|durtj|j|tjd| d <| durtj| d } t|jd t| gf| | d<| durtj| d } t|jd t| f| | d<t |j |j | d6}|j di| D]$}|j jd j}|tjkrtj|ddd }|Vq|dVqWddS1swYdS)aS Query the Triton server using streaming. Args: prompts (List(str)): list of sentences. max_output_len (int): max generated tokens. top_k (int): limits us to a certain number (K) of the top tokens to consider. top_p (float): limits us to the top tokens within a certain probability mass (p). temperature (float): A parameter of the softmax function, which is the last layer in the network. random_seed (int): Seed to condition sampling. stop_words_list (List(str)): list of stop words. bad_words_list (List(str)): list of bad words. no_repeat_ngram_size (int): no repeat ngram size. task_id (str): downstream task id if virtual tokens are used. init_timeout (flat): timeout for the connection. rNr.rrr!r"r rsr7rrtrurvrwrxrjrPr6r)rrHrIrJrMrLrSr{r|rr r rNrOrPr/rRrTrU)r rrtrurvrrr!r"r rsrwrxr,rXrYpartial_result_dictr[r4rrrquery_llm_streamingsH    $z NemoQueryLLM.query_llm_streaming)NNNNNNNNNNNNNNNNNrFFF) NNNr}r~rrNNNr) rrrrrrdrerarrfrrrrrm1sl    rm)rCrWabcrtypingrrnumpyrHnemo.deploy.utilsr use_pytritonpytriton.clientrr Exceptionrrrgrmrrrrs"    x