o ۷iZ!@sddlZddlmZmZddlmZddlmZddlZddl m Z ddl m Z e dZd eejeeeffd d ZGd d d eZeGdddZeGdddZdS)N) dataclassfield)Enum)Optional)logging)tracedContinuousBatchingLoggerreturncCstjr(td}tjtjtj|j}tj|}tj |}n+tj j rHtj j rHtd}tj }|tj }d}n td}d}d}d}||||fS)Ncudampsrcpu)torchr is_availabledevice empty_cache synchronizeget_device_properties total_memorymemory_reservedmemory_allocatedbackendsr is_builtdriver_allocated_memoryrecommended_max_memory)rrreserved_memoryallocated_memoryrj/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/generation/continuous_batching/requests.pyget_device_and_memory_breakdowns"         rc@s,eZdZdZdZdZdZdZdZdZ dZ d S) RequestStatusz5Status of a generation request through its lifecycle.pending prefillingprefilling_splitsplit_pending_remainderdecodingfinishedfailedN) __name__ __module__ __qualname____doc__PENDING PREFILLINGPREFILLING_SPLITSPLIT_PENDING_REMAINDERDECODINGFINISHEDFAILEDrrrrr 5sr c@seZdZUdZeed<eedZee ed<eedZ ee ed<eedZ ee ed<dZ eeed<ejZeed <eejdZe ed <dS) GenerationOutputa5Tracks the output of a generation request. Attributes: request_id (str): The ID of the generation request. prompt_ids (list[int]): The IDs of the prompt tokens. generated_tokens (list[int]): The generated tokens. logprobs (list[float]): The log probabilities of the generated tokens. error (Optional[str]): Any error message associated with the request. When None, the request was successful. status (RequestStatus): The status of the request. created_time (float): The time the request was created. request_iddefault_factory prompt_idsgenerated_tokenslogprobsNerrorstatus created_time)r(r)r*r+str__annotations__rlistr7intr8r9floatr:rr r,r;timer<rrrrr3As  r3c@sTeZdZUdZeed<dZeee ed<dZ eee ed<e edZ ee ed<e edZ ee ed<d Ze ed <d Ze ed <ejZeed <d Ze ed<dZe ed<e ejdZeed<dZeeed<dZeeefed<edefddZejdefddZddZde fddZde fddZ e!d e de"fd!d"Z#d#d$Z$d%d&Z%dS)' RequestStateayTracks the state of a generation request through its lifecycle. Attributes: request_id (str): The ID of the generation request. full_prompt_ids (list[int] | None): The tokens IDs of the full prompt. prompt_ids (list[int] | None): The tokens IDs currently being processed. remaining_prompt_ids (list[int]): The tokens IDs remaining to be processed (for split requests). static_outputs (list[int]): The generated tokens. allocated_blocks (int): The number of blocks allocated to the request. position_offset (int): The current position in the sequence for position_ids. status (RequestStatus): The status of the request: can be one of PENDING, PREFILLING, PREFILLING_SPLIT, SPLIT_PENDING_REMAINDER, DECODING, FINISHED, FAILED max_new_tokens (int): The maximum number of new tokens to generate. eos_token_id (int): The ID of the end-of-sequence token. created_time (float): The time the request was created. error (Optional[str]): Any error message associated with the request. When None, has had no error yet. r4Nfull_prompt_idsr7r5remaining_prompt_idsstatic_outputsrallocated_blocksposition_offset_statusmax_new_tokens eos_token_idr<r:)rLrLlifespanr cC|jS)N)rIselfrrrr;{zRequestState.statusvaluecCsL|jtjkrtdf|_n|tjkr!|jdtf|_|||_dS)NrLr)rIr r,rBrNr1log_end_of_request)rQrSrrrr;s   c Cs`t|j}|}|jd|j}|jd|j}td|jd|d|d|d| dS)NrzRequest z finished: prefill_len = z decode_len = z start_time = z end_time = )lenrD generated_lenrNr<loggerinfor4)rQ prefill_len decode_len start_timeend_timerrrrTs "zRequestState.log_end_of_requestcCrO)zCGet the current length of the sequence (prompt + generated tokens).)rHrPrrr current_lenrRzRequestState.current_lencCs t|jS)z*Get the number of tokens generated so far.)rVrFrPrrrrWs zRequestState.generated_lentoken_idcCs`|jtjkrdS||jko|jdk}||jk}|r|r$|j|g|s(|r.tj|_dSdS)zUpdate the request with a newly generated token and check for completion. Args: token_id: The token ID to add to the output sequence Returns: bool: True if the request is now complete, False otherwise FrLT) r;r r0rMrWrKrFextendr1)rQr_is_eos is_max_lenrrrupdate_with_tokens zRequestState.update_with_tokenc Cs~d|jd|jd|dt|jdt|jd|jdt|jd|jd |j g }d d |d S) Nz request_id=zstatus=z out_tokens=z query_length=zremaining_tokens=z kv_length=zfull_prompt_length=zallocated_blocks=zgenerated_tokens=zRequestState( z, z )) r4rIrWrVr7rErHrDrGrFjoin)rQmsgrrr__repr__s       zRequestState.__repr__cCst|j|j|j|jg|jdS)z7Convert the request state to a GenerationOutput object.)r4r7r;r8r9r:)r3r4rDr;rFr:rPrrrto_generation_outputsz!RequestState.to_generation_output)&r(r)r*r+r=r>rDrr?r@r7rrErFrGrHr r,rIrKrMrBr<rAr:rNtuplepropertyr;setterrTr^rWrboolrcrfrgrrrrrCXs4       rC)rB dataclassesrrenumrtypingrr utils.loggingr utils.metricsr getLoggerrXrhrr@rr r3rCrrrrs