o ۾iS2@stddlmZddlmZddlmZddlmZmZddl Z ddl Z ddl m Z ddlmZddlmZmZdd lmZdd lmZdd lmZeeZeGd d d ZeGdddZGdddZeddddgddZe dedZGdddeeZ eGdddZ!Gddde e!Z"eGdddZ#Gdd d e e#Z$eGd!d"d"Z%Gd#d$d$e e%Z&dS)%)MutableSequence)Sequence) dataclass)AnyGenericN)TypeVar) init_logger)PromptLogprobsSampleLogprobs) LoRARequest)MultiModalPlaceholderDict)RequestStateStatsc@seZdZUdZeed<eed<eeed<edBed<e dBed<dZ e j dBed<dZ edBed <dZeeBdBed <dZedBed <d efd dZd efddZdS)CompletionOutputa!The output data of one completion output of a request. Args: index: The index of the output in the request. text: The generated output text. token_ids: The token IDs of the generated output text. cumulative_logprob: The cumulative log probability of the generated output text. logprobs: The log probabilities of the top probability words at each position if the logprobs are requested. finish_reason: The reason why the sequence is finished. stop_reason: The stop string or token id that caused the completion to stop, None if the completion finished for some other reason including encountering the EOS token. lora_request: The LoRA request that was used to generate the output. indextext token_idsNcumulative_logproblogprobsrouted_experts finish_reason stop_reason lora_requestreturncCs |jduSN)rselfr@/home/ubuntu/.local/lib/python3.10/site-packages/vllm/outputs.pyfinished3s zCompletionOutput.finishedcCsFd|jd|jd|jd|jd|jd|jd|jd|jd S) NzCompletionOutput(index=z, text=z , token_ids=z, routed_experts=z, cumulative_logprob=z , logprobs=z, finish_reason=z, stop_reason=))rrrrrrrrrrrr__repr__6s zCompletionOutput.__repr__)__name__ __module__ __qualname____doc__int__annotations__strGenericSequencefloatr rnpndarrayrrrr boolrr rrrrrs    rc@s<eZdZUdZejed<defddZde de fddZ d S) PoolingOutputznThe output data of one pooling output of a request. Args: data: The extracted hidden states. datarcCd|jdS)NzPoolingOutput(data=r)r.rrrrr MzPoolingOutput.__repr__othercCs t||jot|j|jkSr) isinstance __class__r,r.all)rr1rrr__eq__PszPoolingOutput.__eq__N) r!r"r#r$torchTensorr&r'r objectr,r5rrrrr-Cs  r-c @seZdZdZ     dddddededBdeedBdedBdeed e d e dBd e dBd edBd eedBdedBde dBde eefdBdeddfddZddde ddfddZdefddZdS) RequestOutputakThe output data of a completion request to the LLM. Args: request_id: The unique ID of the request. prompt: The prompt string of the request. For encoder/decoder models, this is the decoder input prompt. prompt_token_ids: The token IDs of the prompt. For encoder/decoder models, this is the decoder input prompt token ids. prompt_logprobs: The log probabilities to return per prompt token. outputs: The output sequences of the request. finished: Whether the whole request is finished. metrics: Metrics associated with the request. lora_request: The LoRA request that was used to generate the output. encoder_prompt: The encoder prompt string of the request. None if decoder-only. encoder_prompt_token_ids: The token IDs of the encoder prompt. None if decoder-only. num_cached_tokens: The number of tokens with prefix cache hit. kv_transfer_params: The params for remote K/V transfer. N)multi_modal_placeholderskv_transfer_params request_idpromptprompt_token_idsprompt_logprobsoutputsrmetricsrencoder_promptencoder_prompt_token_idsnum_cached_tokensr:r;kwargsrc Ksj|r tdt|||_||_||_| pi|_||_||_||_ ||_ ||_ | |_ | |_ | |_| |_dS)Nz+RequestOutput: Ignoring extra arguments: %s)logger warning_oncer'r<r=r>r:r?r@rrArrBrCrDr;)rr<r=r>r?r@rrArrBrCrDr:r;rErrr__init__ns"  zRequestOutput.__init__ next_output aggregatecCs|j|jO_|j|_|jD]Z}t|jD]L\}}|j|jkrb|r[|j|j7_t|jts6t |j|_|j |j|j rN|j dusGJ|j |j |j |_ |j |_ |j|_n||j|<nq|j|qdS)z,Merge subsequent RequestOutput into this oneN)rr;r@ enumeraterrr2rrlistextendrrrrappend)rrIrJnext_completioni completionrrradds.       zRequestOutput.addcCsfd|jd|jd|jd|jd|jd|jd|jd|jd |jd |j d |j d |j d S)NzRequestOutput(request_id=z , prompt=, prompt_token_ids=z, encoder_prompt=z, encoder_prompt_token_ids=z, prompt_logprobs= , outputs= , finished=z , metrics=z, lora_request=, num_cached_tokens=z, multi_modal_placeholders=r) r<r=r>rBrCr?r@rrArrDr:rrrrr s0    zRequestOutput.__repr__)NNNNN)r!r"r#r$r'rLr%r rr,r r r dictrrHrRr rrrrr9VsV      &r9T)r<r=r>r?r@r_O)defaultc @s:eZdZdZdededeededef ddZ d d Z d S) PoolingRequestOutputa The output data of a pooling request to the LLM. Args: request_id (str): A unique identifier for the pooling request. outputs (PoolingOutput): The pooling results for the given input. prompt_token_ids (list[int]): A list of token IDs used in the prompt. num_cached_tokens: The number of tokens with prefix cache hit. finished (bool): A flag indicating whether the pooling is completed. r<r@r>rDrcCs"||_||_||_||_||_dSr)r<r>rDrr@)rr<r@r>rDrrrrrHs  zPoolingRequestOutput.__init__c Cs8t|jd|jd|jd|jd|jd|jd S)Nz (request_id=rTrSrVrUr)typer!r<r@r>rDrrrrrr szPoolingRequestOutput.__repr__N) r!r"r#r$r'rYrLr%r,rHr rrrrr[s  r[c@PeZdZUdZeeed<edefddZ e de fddZ de fd d Zd S) EmbeddingOutputzThe output data of one embedding output of a request. Args: embedding: The embedding vector, which is a list of floats. Its length depends on the hidden dimension of the model. embeddingpooling_outputcC$|j}|jdkr tdt|S)Nz,pooled_data should be a 1-D embedding vector)r.ndim ValueErrorr^tolistr` pooled_datarrr from_bases  zEmbeddingOutput.from_basercC t|jSr)lenr_rrrr hidden_size zEmbeddingOutput.hidden_sizecCr/)NzEmbeddingOutput(hidden_size=r)rkrrrrr r0zEmbeddingOutput.__repr__N)r!r"r#r$rLr)r& staticmethodr-rhpropertyr%rkr'r rrrrr^s  r^c@eZdZedefddZdS)EmbeddingRequestOutputrequest_outputcC"t|jt|j|j|j|jdSN)r<r@r>rDr)rpr<r^rhr@r>rDrrqrrrrh z EmbeddingRequestOutput.from_baseNr!r"r#rmr[rhrrrrrprpc@r]) ClassificationOutputzThe output data of one classification output of a request. Args: probs: The probability vector, which is a list of floats. Its length depends on the number of classes. probsr`cCra)Nrbz.pooled_data should be a 1-D probability vector)r.rcrdrxrerfrrrrh&s  zClassificationOutput.from_basercCrir)rjryrrrr num_classes/rlz ClassificationOutput.num_classescCr/)Nz!ClassificationOutput(num_classes=r)rzrrrrr 3r0zClassificationOutput.__repr__N)r!r"r#r$rLr)r&rmr-rhrnr%rzr'r rrrrrxs  rxc@ro)ClassificationRequestOutputrqcCrrrs)r{r<rxrhr@r>rDrrtrrrrh8ruz%ClassificationRequestOutput.from_baseNrvrrrrr{7rwr{c@s:eZdZUdZeed<edefddZde fddZ d S) ScoringOutputzThe output data of one scoring output of a request. Args: score: The similarity score, which is a scalar value. scorer`cCs(|j}|jdkrtdt|S)Nrz$pooled_data should be a scalar score)r.squeezercrdr|itemrfrrrrhMs   zScoringOutput.from_basercCr/)NzScoringOutput(score=r)r}rrrrr Xr0zScoringOutput.__repr__N) r!r"r#r$r)r&rmr-rhr'r rrrrr|Cs  r|c@ro)ScoringRequestOutputrqcCrrrs)rr<r|rhr@r>rDrrtrrrrh]ruzScoringRequestOutput.from_baseNrvrrrrr\rwr)'collections.abcrrr( dataclassesrtypingrrnumpyr*r6typing_extensionsr vllm.loggerr vllm.logprobsr r vllm.lora.requestr vllm.multimodal.inputsr vllm.v1.metrics.statsr r!rFrr-r9STREAM_FINISHEDrYr[r^rprxr{r|rrrrrsJ        ,n $