o ei8@sddlZddlmZmZddlmZddlZddlmZm Z ddl m Z ddl m Z er1ddlZdZe d Zd eejeeeffd d ZGd ddeZeGdddZeGdddZdS)N) dataclassfield)Enum)is_psutil_availableis_torch_xpu_available)logging)tracedContinuousBatchingLoggerreturncCs,tjr(td}tjtjtj|j}tj|}tj |}nht rNtd}tj tj tj |j}tj |}tj |}nBtj j rntj j rntd}tj }|tj }d}n"td}trtj}tj}|}n tdd}d}d}||||fS)NcudaxpumpsrcpuzCannot get memory breakdown on CPU without psutil: returning 0 for all memory values. Please install psutil to get an actual memory breakdown.)torchr is_availabledevice empty_cache synchronizeget_device_properties total_memorymemory_reservedmemory_allocatedrrbackendsris_builtdriver_allocated_memoryrecommended_max_memoryrpsutilvirtual_memorytotalProcess memory_inforssloggererror)rrreserved_memoryallocated_memoryr(r/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/generation/continuous_batching/requests.pyget_device_and_memory_breakdown$s>              r*c@s,eZdZdZdZdZdZdZdZdZ dZ d S) RequestStatusz5Status of a generation request through its lifecycle.pending prefillingprefilling_splitsplit_pending_remainderdecodingfinishedfailedN) __name__ __module__ __qualname____doc__PENDING PREFILLINGPREFILLING_SPLITSPLIT_PENDING_REMAINDERDECODINGFINISHEDFAILEDr(r(r(r)r+Ksr+c@seZdZUdZeed<eedZee ed<eedZ ee ed<eedZ ee ed<dZ edBed<ejZeed <eejdZe ed <d Zee e fed <dZee dBed <defddZdS)GenerationOutputaTracks the output of a generation request. Attributes: request_id (str): The ID of the generation request. prompt_ids (list[int]): The IDs of the prompt tokens. generated_tokens (list[int]): The generated tokens. logprobs (list[float]): The log probabilities of the generated tokens. error (Optional[str]): Any error message associated with the request. When None, the request was successful. status (RequestStatus): The status of the request. created_time (float): The time the request was created. lifespan (tuple[float, float]): The time the request was no longer pending and the time the request finished. request_iddefault_factory prompt_idsgenerated_tokenslogprobsNr%status created_timer r lifespan timestampsr cCs |jtjkSN)rEr+r<selfr(r(r) is_finishedps zGenerationOutput.is_finished)r3r4r5r6str__annotations__rlistrBintrCrDfloatr%r+r7rEtime perf_counterrFrHtuplerIboolrMr(r(r(r)r>Ws  r>c@seZdZUdZeed<eeed<dZe ed<dZ eed<e edZ eeed <e edZ eeed <e edZeeed <dZeed <dZeed <ejZeed<dZedBed<dZeed<dZe ed<e ejdZeed<dZedBed<dZeeefed<e edZeeed<dZeed<dZ eed<ddZ!e"defd d!Z#e#j$d"efd#d!Z#e"deedBfd$d%Z%d&d'Z&defd(d)Z'defd*d+Z(e)d,ede fd-d.Z*d/d0Z+d1d2Z,d3eddfd4d5Z-d8d6d7Z.dS)9 RequestStateaCTracks the state of a generation request through its lifecycle. Attributes: request_id (str): The ID of the generation request. initial_tokens (list[int]): The initial prompt tokens. num_children (int): The number of children requests full_prompt_ids (list[int] | None): The tokens IDs of the full prompt. prompt_ids (list[int] | None): The tokens IDs currently being processed. remaining_prompt_ids (list[int]): The tokens IDs remaining to be processed (for split requests). static_outputs (list[int]): The generated tokens. allocated_blocks (int): The number of blocks allocated to the request. position_offset (int): The current position in the sequence for position_ids. status (RequestStatus): The status of the request: can be one of PENDING, PREFILLING, PREFILLING_SPLIT, SPLIT_PENDING_REMAINDER, DECODING, FINISHED, FAILED max_new_tokens (int | None): The maximum number of new tokens to generate. eos_token_id (int): The ID of the end-of-sequence token. streaming (bool): Whether to stream tokens as they're generated created_time (float): The time the request was created. error (Optional[str]): Any error message associated with the request. When None, has had no error yet. r?initial_tokensFrecord_timestampsr num_childrenr@tokens_to_processremaining_prefill_tokensrCallocated_blocksposition_offset_statusNmax_new_tokensr eos_token_id streamingrFr%rGrH _timestamps_true_initial_tokens_new_tokens_limitcCs |jdur d|_dS|j|_dS)Nrf)rargrKr(r(r) __post_init__s zRequestState.__post_init__r cC|jSrJ)r_rKr(r(r)rEzRequestState.statusvaluecCsL|jtjkrtdf|_n|tjkr!|jdtf|_|||_dS)Nr r)r_r+r7rSrTrHr<log_end_of_request)rLrkr(r(r)rEs   cCs|jr|jSdSrJ)rYrdrKr(r(r)rIszRequestState.timestampsc Cs`t|j}|}|jd|j}|jd|j}td|jd|d|d|d| dS)NrRequest z finished: prefill_len = z decode_len = z start_time = z end_time = )lenrX generated_lenrHrFr$infor?)rL prefill_len decode_len start_timeend_timer(r(r)rls "zRequestState.log_end_of_requestcCri)zCGet the current length of the sequence (prompt + generated tokens).)r^rKr(r(r) current_lenrjzRequestState.current_lencCs t|jS)z*Get the number of tokens generated so far.)rorCrKr(r(r)rps zRequestState.generated_lentoken_idcCs|jtjkrdS|jr|jt||jko|jdk}| d}|s*||j kr4||j d<|d7}nt d|jd||j |sL||j krRtj|_dSdS)zUpdate the request with a newly generated token and check for completion. Args: token_id: The token ID to add to the output sequence Returns: bool: True if the request is now complete, False otherwise Fr rmrnz generated a useless token: T)rEr+r;rYrdappendrSrTrbrprgrCr$warningr?popr<)rLrwis_eosrvr(r(r)update_and_check_completions     z(RequestState.update_and_check_completionc Cs~d|jd|jd|dt|jdt|jd|jdt|jd|jd |j g }d d |d S) Nz request_id=zstatus=z out_tokens=z query_length=zremaining_tokens=z kv_length=zfull_prompt_length=zallocated_blocks=zgenerated_tokens=zRequestState( z, z )) r?r_rpror[r\r^rXr]rCjoin)rLmsgr(r(r)__repr__s       zRequestState.__repr__c Csx|jr|jdtkr|j|jr'|j|jd|j|_|jd|j|_t|j|j|jg|j|j|j |j |j d S)z7Convert the request state to a GenerationOutput object.r N) r?rBrCrDr%rErFrHrI) rC TMP_TOKEN_IDrzrerXr>r?r%rErFrHrIrKr(r(r)to_generation_outputs  z!RequestState.to_generation_outputnew_request_idcCst}tdid|d|jd|jd|jddd|jddd|jddd|jd |j d |j d |j d |j d |j d|d|dfdgd|jd|j}|S)ziFork the request into a new request with the same state expect for request_id, created_time and lifespan.r?rXrZr[Nr\rCr]r^r_rarbrcrFrHr rdr%rYr()rSrTrWrXrZr[r\rCr]r^rErarbrcr%rY)rLrt new_requestr(r(r)forksJ     zRequestState.forkc Cs|jr|jdtkr|j|jdurdn|jt|j}t|j|j|j|j|j |j|j||j |j d}|j t|j|_ |S)aCreates an equivalent new request by removing the generated tokens and adding them to the initial prompt. The created request has THE SAME request_id. Notably, we can retrieve the original request from the created one with the _true_initial_tokens attribute.r N)r?rXrZrYr[rarbrc) rCrrzrarorWr?rXrZrYrbrcre)rLra new_stater(r(r)!create_equivalent_initial_request's    z.RequestState.create_equivalent_initial_request)r rW)/r3r4r5r6rNrOrPrQrYrVrZrr[r\rCr]r^r+r7r_rarbrcrSrTrFrRr%rHrUrdrergrhpropertyrEsetterrIrlrvrpr r|rrrrr(r(r(r)rWtsJ           #rW)rS dataclassesrrenumrrutilsrr utils.loggingr utils.metricsr rr getLoggerr$rUrrQr*r+r>rWr(r(r(r)s"     '