o ۷i@sddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlmZmZddlmZmZmZddlmZddlmZmZddlmZddl mZddlmZmZmZdd l m!Z!dd l"m#Z#dd l$m%Z%ddl&Z&dd l'm(Z(m)Z)dd l*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3m4Z4m5Z5m6Z6ddl7m8Z8m9Z9ddl1m:Z:e8rddl;Z;ddl&mZ>m?Z?ddl@mAZAmBZBe,rddlCZCe0rddlDmEZEe.oe+oe/oe-ZFeFrddlGZGddlHmIZImJZJddlKmLZLddlMmNZNmOZOddlPmQZQddlRmSZSddlTmUZUddlVmWZWmXZXmYZYmZZZm[Z[ddl\m]Z]ddl^m_Z_m`Z`maZambZbmcZcmdZdmeZemfZfmgZgmhZhmiZimjZjmkZkmlZlmmZmddlnmoZodd lpmqZqmrZrmsZsGd!d"d"eod#d$ZtGd%d&d&e]d#d$ZuGd'd(d(eSd#d$ZveretZwereuZxerevZyhd)Zzhd*Z{hd+Z|e9}e~Zd,d-d.d/iZeeZd0ZGd1d2d2ejZd3efd4d5Zd6ed7d8d9d8fd:d;ZGdd?d?ZeGd@dAdAZGdBdCdCe:Ze~dDkreZedSdS)EN)ArgumentParser Namespace)AsyncGenerator GeneratorIterable)asynccontextmanager) dataclassfield)BytesIO)Thread)Optional TypedDictUnion model_infoHF_HUB_OFFLINE) DecodeStream)!MODEL_FOR_CAUSAL_LM_MAPPING_NAMES*MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES)is_fastapi_availableis_librosa_availableis_openai_availableis_pydantic_availableis_uvicorn_availableis_vision_available) AutoConfigLogitsProcessorListPreTrainedTokenizerFastProcessorMixinTextIteratorStreamer)is_torch_availablelogging)BaseTransformersCLICommand) AutoProcessorBitsAndBytesConfigGenerationConfigPreTrainedModel)ContinuousBatchingManager RequestStatus)Image)FastAPI HTTPException)CORSMiddleware) JSONResponseStreamingResponse) Transcription)TranscriptionCreateParamsBase)ChatCompletionMessageParam)ChatCompletionChunkChoice ChoiceDeltaChoiceDeltaToolCallChoiceDeltaToolCallFunction)CompletionCreateParamsStreaming)ResponseResponseCompletedEventResponseContentPartAddedEventResponseContentPartDoneEventResponseCreatedEvent ResponseErrorResponseErrorEventResponseFailedEventResponseInProgressEventResponseOutputItemAddedEventResponseOutputItemDoneEventResponseOutputMessageResponseOutputTextResponseTextDeltaEventResponseTextDoneEvent)ResponseCreateParamsStreaming) BaseModel TypeAdapterValidationErrorc@eZdZUdZeed<dS))TransformersResponseCreateParamsStreamingz OpenAI's ResponseCreateParamsStreaming with an additional field for the generation config (as a json string). generation_configN__name__ __module__ __qualname____doc__str__annotations__rXrXS/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/commands/serving.pyrO{  rOF)totalc@rN)+TransformersCompletionCreateParamsStreamingz OpenAI's CompletionCreateParamsStreaming with additional fields for the generation config (as a json string) and passing the request_id rPNrQrXrXrXrYr\rZr\c@s.eZdZUdZeed<eed<dZeed<dS)%TransformersTranscriptionCreateParamsz OpenAI's TranscriptionCreateParamsBase with an additional field for the generation config (as a json string). filerPFstreamN) rRrSrTrUbytesrWrVr_boolrXrXrXrYr]s r]> textuserstorepromptinclude reasoning background truncation tool_choice service_tier top_logprobsmax_tool_callsprevious_response_id>nstoprcaudiordlogprobsmetadata functions modalities predictionrjrkrl function_callstream_optionsresponse_formatpresence_penaltyreasoning_effortweb_search_optionsparallel_tool_callsmax_completion_tokens>rerflanguagerychunking_strategytimestamp_granularitiesqwenz z )startendz x-request-idc@seZdZdZdZdZdZdS)ModalityLLMVLMSTTTTSN)rRrSrTrrrrrXrXrXrYrs rargscCst|S)z~ Factory function used to instantiate serving server from provided command line arguments. Returns: ServeCommand ) ServeCommand)rrXrXrYserve_command_factorysrreqmodel_generation_configr(returncKsZ|ddurtd it|d}nt|}|jd i|}|D]\}}|dur3t|||q%|ddurBt |d|_ |ddurPt |d|_ |ddur^t |d|_ |ddurj|d|_ |ddurv|d|_|ddurt |d|_t |dd krd |_|d durt |d |_|d durt|d |S)a Creates a generation config from the parameters of the request. If a generation config is passed in the request, it will be used as a baseline for parameterization. Otherwise, we will use the model's default generation config. Other parameters in the request will be applied on top of the baseline. Args: req (`dict`): The request which may optionally contain generation parameters. model_generation_config (`GenerationConfig`): The model's default generation config. kwargs (`dict`): Additional parameters to set in the generation config. Returns: The prepared `GenerationConfig` object. rPNmax_output_tokens max_tokensfrequency_penalty logit_biasrp temperaturegFtop_pseedrX)getr(jsonloadscopydeepcopyupdateitemssetattrintmax_new_tokensfloatrepetition_penalty sequence_bias stop_stringsr do_samplertorch manual_seed)rrkwargsrPnon_standard_kwargskvrXrXrY!create_generation_config_from_reqs6    rc@s eZdZdZddZddZdS) ToolStatez7Lightweight class to keep track of the tool call state.cCs |dSN)resetselfrXrXrY__init__'s zToolState.__init__cCsd|_d|_d|_d|_dS)z>Reset the tool call state (assumes we're outside a tool call).FrN)inside_tool_callhas_tool_name_definedarg_nesting_levelbufferrrXrXrYr*s zToolState.resetN)rRrSrTrUrrrXrXrXrYr$s rc @sReZdZdZ ddddedeedfdd Zd d Zd d Z ddZ ddZ dS) TimedModelz A class that holds a PreTrainedModel instance and its associated processor. Automatically deletes the instances after a specified timeout. Nmodelr)timeout_seconds processor)r rcCs>||_t|j|_||_||_t|j|j|_ |j dSr) rrV name_or_path _name_or_pathrr threadingTimertimeout_reached_timerr)rrrrrXrXrYr8s  zTimedModel.__init__cCs*|jt|j|j|_|jdS)z2Reset the timer for the deletion of the instances.N)rcancelrrrrrrrXrXrY reset_timerEs zTimedModel.reset_timercCsZt|dr)|jdur+|`|`d|_d|_ttjr"tj|j dSdSdS)z>Delete the wrapped model and processor and clean up resources.rN) hasattrrrgccollectrcuda is_available empty_cacherrrrXrXrY delete_modelKs  zTimedModel.delete_modelcCs&|t|jd|jddS)Nz was removed from memory after z seconds of inactivity)rloggerinforrrrXrXrYr[szTimedModel.timeout_reachedcCst|d p |jduS)z)Check if the instances have been deleted.rN)rrrrXrXrY is_deleted_szTimedModel.is_deletedr) rRrSrTrUrr rrrrrrrXrXrXrYr2s     rc@seZdZUdZedddidZeed<edddidZe ed <ed d gd d dZ e e ed<eddgd d dZ e e ed<edddidZ eed<ed ddidZe e ed<edddidZeed<edddidZeed<eddddgd dZe ed<edddidZeed<eddd idZe ed!<ed"dd#idZeed$<ed%dd&idZeed'<ed(dd)idZe ed*<ed dd+idZe eed,<eddd-idZeed.<eddd/idZeed0<ed dd1idZe e ed2<d3d4Zd S)5ServeArgumentsz Arguments for the serve CLI. See the metadata arg for each argument's description -- the metadata will be printed with `transformers serve --help` Fhelpz8Whether to use continuous batching for chat completions.)defaultrscontinuous_batchingautozfDevice to use for inference; will default to `auto` andplace the model on an accelerator if available.deviceNzA`torch_dtype` is deprecated! Please use `dtype` argument instead.)rbfloat16float16float32)rchoices torch_dtypezOverride the default `torch.dtype` and load the model under this dtype. If `'auto'` is passed, the dtype will be automatically derived from the model's weights.dtypez2Whether to trust remote code when loading a model.trust_remote_codezWhich attention implementation to use; you can run --attn_implementation=flash_attention_2, in which case you must install this manually by running `pip install flash-attn --no-build-isolation`.attn_implementationzIWhether to use 8 bit precision for the base model - works only with LoRA. load_in_8bitzIWhether to use 4 bit precision for the base model - works only with LoRA. load_in_4bitnf4zQuantization type.fp4bnb_4bit_quant_typez#Whether to use nested quantization.use_bnb_nested_quant localhostz$Interface the server will listen to.hosti@zPort the server will listen to.porti,z@Time in seconds after which a model will be removed from memory. model_timeoutrz8Logging level as a string. Example: 'info' or 'warning'. log_levelz1The default seed for torch, should be an integer. default_seedztWhether to enable CORS. Some apps that make requests from external domains (e.g. Cursor) require CORS to be enabled. enable_corsz+Whether to turn on strict input validation.input_validationzName of the model to be forced on all requests. This is useful for testing Apps that don't allow changing models in the request. force_modelcCsN|jdur#|jdur|j|_dS|j|jkr%td|jd|jddSdS)z(Only used for BC `torch_dtype` argument.Nz`torch_dtype` z and `dtype` zn have different values. `torch_dtype` is deprecated and will be removed in 4.59.0, please set `dtype` instead.)rr ValueErrorrrXrXrY __post_init__s    zServeArguments.__post_init__)rRrSrTrUr rrarWrrVrr rrrrrrrrrrrrrrrrrrXrXrXrYrds  rc@s$eZdZedefddZdefddZdede d d d e fd d Z defddZ defddZ defddZ        dHdedeedeedeedeedeeddeedeedefd d!Zd"d#defd$d%Zd&d'Zejdeeeeffd(d)Zd*ededeedffd+d,Zedd-defd.d/Zed0efd1d2Z d*ede!eddffd3d4Z"d*ede!eddffd5d6Z#d*ede!eddffd7d8Z$d*ede%fd9d:Z&ededed;fdedefd?d@Z(dAefdBdCZ)dAede*d-effdDdEZ+dAede*d-e,ffdFdGZ-dS)IrparsercCs$tf}|jd|d}|jtddS)z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments serve)dataclass_types)funcN)r add_parser set_defaultsr)rr serve_parserrXrXrYregister_subcommandsz ServeCommand.register_subcommandrcCs tstd||_|jj|_|jrAt}|jjdur'||j_t d|t }|jj|vrAt d|d|jjd|d|jj |_ |jj durSt|jj td}|tj|jjtd}|tj|jji|_d|_d|_d|_d|_dS) NzaMissing dependencies for the serving CLI. Please install with `pip install transformers[serving]`z-No attn_implementation passed, defaulting to z"Continuous batching only supports z as attn_implementation, got z#Try setting `--attn_implementation=` transformersz+transformers.generation.continuous_batching)serve_dependencies_available ImportErrorrruse_continuous_batchingr* default_attention_implementationrrr#supported_attention_implementationsrrrrrr# get_loggersetLevel log_levelsrlower loaded_models#running_continuous_batching_manager last_messages last_kv_cache last_model)rrdefault_attn_implsupported_attn_impltransformers_logger cb_loggerrXrXrYrs@        zServeCommand.__init__requestschema validatorrL unused_fieldsc Cstd|t|}|j}||}|r(td|tdd|d|jjriz| |Wnt yQ}ztd| td| dd}~ww||@} | rktd| tdd| ddSdS)a Validates the request against the schema, and checks for unexpected keys. Args: request (`dict`): The request to validate. schema (`TypedDict`): The schema of the request to validate. It is a `TypedDict` definition. validator (`TypeAdapter`): The validator to use to validate the request. Built from `schema`. unused_fields (`set`): Fields accepted by `schema`, but not used in `transformers serve`. Raises: HTTPException: If the request is invalid or contains unexpected or unused fields. zValidating request: z Unexpected keys in the request: i) status_codedetailzValidation error: NzUnused fields in the request: ) rdebugsetkeys__mutable_keys__errorr.rrvalidate_pythonrMerrors) rr r r r  input_keys possible_keysunexpected_keyseunused_fields_in_requestrXrXrY_validate_requests.   zServeCommand._validate_requestcC|j|tttddSN)r r r r )rrOresponse_validatorUNUSED_RESPONSE_FIELDSrr rXrXrYvalidate_response_requestA  z&ServeCommand.validate_response_requestcCrr)rr\completion_validatorUNUSED_CHAT_COMPLETION_FIELDSr rXrXrY validate_chat_completion_requestIr"z-ServeCommand.validate_chat_completion_requestcCrr)rr]transcription_validatorUNUSED_TRANSCRIPTION_FIELDSr rXrXrYvalidate_transcription_requestQr"z+ServeCommand.validate_transcription_requestrN request_idcontentrrole finish_reason tool_callsr8 decode_stream tokenizerrc Csl|dur|dur|dur||j|}t|tt|tt|||dd|dgddd} d| jd d d S) a Builds a chunk of a streaming OpenAI Chat Completion response. IMPORTANT: The serialized chunk won't contain empty fields (fields with `None`). Some downstream apps, like Cursor, assume that when the field exists, it has data. Args: request_id (`str`): The request ID. content (`str`, *optional*): Content of the response from the model. model (`str`, *optional*): The model that generated the content. role (`str`, *optional*): The role of the next content, until a new role is defined. finish_reason (`str`, *optional*): The reason the generation by the model has finished. tool_calls (`list[ChoiceDeltaToolCall]`, *optional*): Data about the tool calls, when they are triggered. Returns: `str`: The built chunk, a string containing a JSON string with the payload. N)r*r+r-r)deltaindexr,rzchat.completion.chunk)idcreatedrrsystem_fingerprintobjectdata: T exclude_none )step _tokenizerr5rtimer6r7model_dump_json) rr)r*rr+r,r-r.r/chunkrXrXrYbuild_chat_completion_chunkYs("  z(ServeCommand.build_chat_completion_chunkresponserKcCsd|jdddS)a Builds a event of a streaming OpenAI Response response. IMPORTANT: The serialized chunk won't contain empty fields (fields with `None`). Some downstream apps, like Cursor, assume that when the field exists, it has data. Args: response (`BaseModel`): The response to build an event from. One of the multiple OpenAI Response output types Returns: `str`: The built chunk, a string containing a JSON string with the payload. r6Tr7r9)r=)rr@rXrXrYbuild_response_eventsz!ServeCommand.build_response_eventc stdtffdd }t|d}jr%|jtdgddgdgdtdd d lm}| d d |d t ffdd }| dd t ffdd }| dd |ffdd }| d| dfdd}| ddd}| dd |fdd} tj|jjjjjjdd S)!a Setup and run the FastAPI server for transformers serve. Models will be loaded and unloaded automatically based on usage and a timeout. The server will expose the following endpoints: - POST /v1/chat/completions: Generates chat completions. - POST /v1/responses: Generates responses. - POST /v1/audio/transcriptions: Generates transcriptions from audio. - GET /v1/models: Lists available models for 3rd party tools. Requires FastAPI and Uvicorn to be installed. appcsBdVjD]}|q jdurjjddddSdS)NTblocktimeout)rvaluesrrrp)rBrrrXrYlifespans  z"ServeCommand.run..lifespan)rH*T) allow_originsallow_credentials allow_methods allow_headerszUCORS allow origin is set to `*`. This is not recommended for production environments.r)Requestz/v1/chat/completionsr bodycs:j|djr||jj}n|}t|ddSNr text/event-stream media_type)r%r#continuous_batching_chat_completionstater)generate_chat_completionr1)r rOoutputrrXrYchat_completions   z)ServeCommand.run..chat_completionz /v1/responsescs"j|d|}t|ddSrP)r!generate_responser1)r rXrrXrY responsess   z#ServeCommand.run..responsesz/v1/audio/transcriptionsc s|4IdH5}t|dIdH|dd}td|djd|djd|djddd WdIdHn 1IdHsDwYj|d  |}t |d d S) Nr^r)r^rzReceived file: z ; MIME type: z; size: z.2fz KiBrQrRrS) formr]readrrfilename content_typesizer(generate_transcriptionr1)r r]parsed_requestrXrrXrYaudio_transcriptionss(  z.ServeCommand.run..audio_transcriptionsz /v1/modelscstddS)Nlist)r5data)r0get_gen_modelsrXrrXrYget_all_modelssz(ServeCommand.run..get_all_modelsz/healthcSs tddiS)Nstatusok)r0rXrXrXrY healthchecks z%ServeCommand.run..healthcheckhttpcs>|jtp tt}||j_||IdH}||jt<|Sr)headersr X_REQUEST_IDrVuuiduuid4rVr))r  call_nextr)r@rXrXrYget_or_set_request_ids  z/ServeCommand.run..get_or_set_request_id)rrrN)rr-radd_middlewarer/r warning_oncefastapirNpostdictoptionsr middlewareuvicornrunrrrr) rrHrBrNrYr[rdrhrkrrrXrrYr{s:    "zServeCommand.runcCs6gd}tr dd|DSdd|D}dd|DS)a. This is by no means a limit to which models may be instantiated with `transformers serve`: any chat-based model working with generate can work. This is a limited list of models to ensure we have a discoverable /v1/models endpoint for third-party integrations. ) zMenlo/Jan-nanozMenlo/Jan-nano-128kzQwen/Qwen2.5-0.5B-InstructzQwen/Qwen2.5-3B-InstructzQwen/Qwen2.5-7B-InstructzQwen/Qwen2.5-14B-Instructz meta-llama/Llama-3.1-8B-Instructz meta-llama/Llama-3.2-1B-Instructz!meta-llama/Llama-3.3-70B-InstructzHuggingFaceTB/SmolVLM-Instructz!ibm-granite/granite-vision-3.2-2bzQwen/Qwen2.5-VL-7B-InstructcSs.g|]}|dtj|dddqS)r/rr2r5r3owned_by)datetimenow timestampsplit.0rrXrXrY s  z/ServeCommand.get_gen_models..cSsg|]}t|qSrXrrrXrXrYr#scSs$g|]}|jd|j|jdqS)rr})r2 created_atrauthorrrXrXrYr$sr)rmodels model_infosrXrXrYrgs  zServeCommand.get_gen_modelsrc s|djk}_|r!jdur!jjdddd_\}}t|dr0|jn|t||jj j dddd jdurW|j dd _t j_ j|j|d d dd |j}fddfdd}||d|S)a' Generates an OpenAI Chat Completion using continuous batching. Args: req (`dict`): The request to generate an OpenAI Chat Completion for. Returns: `Generator[str, None, None]`: A generator that yields the OpenAI Chat Completion chunks. rNTrrDr/Ffifo)r eos_token_id pad_token_id use_cacher scheduler)rP streamingmessagespt)return_tensorsadd_generation_promptc 3sz6j|ddVj|D]#}|jtjkr&j|ddVWdSj||jd|dVqWdSty`}zt t |j |dt |dVWYd}~dSd}~ww) N assistantr+rrpr,r)r)r*rr.r/data: {"error": ""}) r?rrequest_id_iterrir+FINISHEDgenerated_tokens ExceptionrrrVcancel_request)r)r.resultr)model_id_and_revisionrr/rXrYstream_chat_completion^s2    zPServeCommand.continuous_batching_chat_completion..stream_chat_completioncsz't|d}jj||jd}||D] }|VtdIdHqWdStjyBj|t d|dYdSw)NF)r)rrzRequest z was cancelled.) rtolistr add_requestrasynciosleepCancelledErrorrrwarning)_inputsr)r.r>)rPrrrXrYcancellation_wrapperzs zNServeCommand.continuous_batching_chat_completion..cancellation_wrapperr)process_model_namerrrpload_model_and_processorrr/rrPrrinit_continuous_batchingrlogit_processorrapply_chat_templatetor)rrr)must_discard_cacherrinputsrrX)rPrrrr/rYrU.s<     z0ServeCommand.continuous_batching_chat_completionr)cCsB|jj}|tvrtj}|S|tvrtj}|Std|)NzUnknown modality: ) __class__rRrrGrrrrr)rmodel_classnamemodalityrXrXrYget_model_modalitys  zServeCommand.get_model_modalityrc Cs~g}|D]}|dgd}|tjkrEt|dtr|d}n"t|dtr@g}|dD]}|ddkr:||dq+d|}||d<nr|tjkrt|dtr^|dd|ddnY|dD]T}|ddkrr|d|qb|ddkrd |dd vrt d d |dd }t t t |}tjd dd} | j} || jn|dd } |dd| dqb||q|S)Nr+r+r*r*typerb )rrb image_urlbase64urlz^data:image/.+;base64,rz.pngF)suffixdeleteimage)rr)rr isinstancerVreappendjoinrresubr,openr r b64decodetempfileNamedTemporaryFilenamesave) rrprocessor_inputsmessageparsed_messageparsed_contentr* image_datarr^rrXrXrY*get_processor_inputs_from_inbound_messagess@            z7ServeCommand.get_processor_inputs_from_inbound_messagescsvjjdur jj|d<|d}|dddkrdS|djk}_\}}||}dtD]}|jj d vrO|nq?|j |d| d d ddd }| j}| d d d} djj d vrxd} t|| dd} t|jd} d} |r|sj} |djd| krj} i|| | d| dfdd}|| S)a Generates an OpenAI Chat Completion using `generate`. Args: req (`dict`): The request to generate an OpenAI Chat Completion for. Returns: `Generator[str, None, None]`: A generator that yields the OpenAI Chat Completion chunks. Nrrrr+rrTtoolsr)rrr return_dicttokenizer)req_0gptossFskip_special_tokens skip_promptr input_ids)streamerrPreturn_dict_in_generatepast_key_valuesc 3sd}d}djjdvrd}d}fdd}t|d}d }z#z|t}jd d V|D]}djjdvrH|d }||7}|rV||vrUd}q7q7dur|t d krhd|_ q7|t dkr| j|dddVq7|j r|j |7_ |j std|j } | durq7| d} d|_ tt| ddd|dd} n<|d krq7d|j vrq7|j|d7_|j|d8_|jdkrd |dddd}tt|dddd} j|d| gdVq7|d krj||dVq7j|dd V|Wn#ty8} ztt| d!t| d"VWYd} ~ n d} ~ wwW|dSW|dS|w)#NFrrT<|channel|>final<|message|>cjdi|}|j_dSNrXgeneraterrrgenerate_outputrrrXrYgenerate_with_cache zbServeCommand.generate_chat_completion..stream_chat_completion..generate_with_cachetargetrrrr <|return|>rrr-)r)r+r,rz\"name\": \"(.*?)\"r$)rfunction _tool_call)rr1rr2z"arguments": {{}) arguments)rr1r)r)r+r-r)r*rrprrr)config architecturesrr rrr? removesuffixstrip_TOOL_CALL_TOKENSrrrrrsearchgroupr8r9rcountrrrrrrV) r _request_id filter_cot cot_trace_endrthreadresults tool_stater tool_nametoolrgeneration_kwargsrrr)rtool_model_familyrXrYrs          zEServeCommand.generate_chat_completion..stream_chat_completion)rrrrrrr_MODELS_WITH_TOOL_SUPPORTrrrrrrrr!rrPis_continuationrget_seq_lengthshape)rrrrrrrsupported_model_familiesrrgeneration_streamerrPrseq_lenrrXrrYrWsf        zz%ServeCommand.generate_chat_completionc sdjk}_\}tdtr6dvr)dddgng}|dddnUtdtrjdvreddddkrXdddgd}n3d}d|dd <n&d}n!tdtrdvr}dddgng}|dntd |j |d d d }| j } ddd }dj jdvrd}t||d d}tjd}d}r|sՈj} |djd| krՈj}|t|||d |dfdd} | |S)a  Generates an OpenAI Response using `generate`. Args: req (`dict`): The request to generate an OpenAI Response for. Returns: `Generator[str, None, None]`: A generator that yields the OpenAI Response events. rinput instructionssystemrrcrr+r*z%inputs should be a list, dict, or strTr)rrrnrrFrrNrr)rattention_maskrrPrrc3s4d}d}djjdvrd}d}fdd}t|d}d}d}d}zzz|t} td |td | d d d ddiidggddddd d} |d7} | Vt d|td | dd d ddiidggddddd d} |d7} | Vt d||t ddddgdd} |d7} | Vt dd|||td d!gd"d#} |d7} | Vd!}|D]=}djjdvr|d$}||7}|r||vrd}d!}qqtd%d||||d!d&d'gd(}|d7} |Vqtd)d||d|d!d&d'gd*}|d7} |Vtd+d|||td |jgd"d#}|d7}|d7} |Vtd,||t ddd-d|jggd.d}|d7}|d7} |Vtd/|td | d-d d ddii|jgdgddddd0 d}|d7} |V|Wncty}zVtd1t|td2|t|d3}|d7} |Vtd4|td | d5d d ddiigdgdddtd6t|d7d8 d}|d7} |VWYd}~n d}~wwW|dSW|dS|w)9NFrrTrcrrrrrrXrYrrzTServeCommand.generate_response..stream_response..generate_with_cacherzresponse.createdresp_queuedr formatrrbr@r}rrs) r2rrirr rbr5rrXr}rjrs)rsequence_numberr@r$zresponse.in_progress in_progresszresponse.output_item.addedmsg_rr)r2rrir+r*)rr output_indexitemzresponse.content_part.added output_textr)rrb annotations)ritem_idrr content_indexpartrzresponse.output_text.deltagX@)tokenlogprob)rrrrrr0rrzresponse.output_text.done)rrrrrrbrrzresponse.content_part.donezresponse.output_item.done completed)r2rrir+r*rzresponse.completed) r2rrirr rbrXr5rr}rjrsz"Exception in response generation: r)rrrzresponse.failedfailed server_error)coder) r2rrirr rbrXr5rr}rjrsr) rrrr rr<r?r;rrArCrDrFr=rGrrHrIr>rbrErr<rrrrrrVrArBr@)rrrrrrrrrrresponse_createdresponse_in_progressresponse_output_item_addedresponse_content_part_addedrrresponse_output_text_deltaresponse_output_text_doneresponse_content_part_doneresponse_output_item_doneresponse_completedr error_eventresponse_failedrrrrr)rrXrYstream_responses                         %z7ServeCommand.generate_response..stream_response)rrrrrVrrerwrrrrrrrrr!rrPrrrrr ones_like) rrrrrrrrPrr r-rXr,rYrZsZ       czServeCommand.generate_responsec ststd||d}||\tjddd}t|jd}jj }t |d}t j ||dd\}}||dd jd jd <||dd fd d } | S)a Generates an OpenAI Transcription using the audio file. Args: req (`dict`): The request containing the audio file and model information. Returns: `Generator[str, None, None]`: A generator that yields the transcription result. z]Missing librosa dependency for audio transcription. Please install with `pip install librosa`rTrrr^)srmonor) sampling_raterinput_features)rrPrc3sHjdi}j|jddd}t|d}|jddVdS)NT)rr)rbr7rX)r batch_decode sequencesr2r=) generated_idstranscription_text transcription audio_inputs audio_modelaudio_processorrrXrY_generate_transcriptions  zDServeCommand.generate_transcription.._generate_transcription)rrrload_audio_model_and_processorr!r/rrPfeature_extractorr1ior librosaloadrrr) rrrrrPmodel_sampling_rate audio_bytes audio_array_r<rXr8rYrbs2 z#ServeCommand.generate_transcriptioncCsx|dp |d}d}|jdurd}n#t|jt|kr d}ntt|jD]}|j|||kr6d}nq'||_|S)aD Determines whether the current request is a continuation of the last request. In other words, if it is the same chat session. Args: req (`dict`): The request to check. Returns: `True` if the request is a continuation of the last request, `False` otherwise. rr TNF)rrlenrange)rrrreq_continues_last_messagesirXrXrYrs  zServeCommand.is_continuationr'cCs@|jrtd|j|j|j|jd}|S|jrtdd}|Sd}|S)a Returns the quantization config for the given CLI arguments. Args: args (`ServeArguments`): The serve arguments. May contain quantization settings, device, etc. Returns: `Optional[BitsAndBytesConfig]`: The quantization config. T)rbnb_4bit_compute_dtyperbnb_4bit_use_double_quantbnb_4bit_quant_storage)rN)rr'rrrr)rquantization_configrXrXrYget_quantization_configs  z$ServeCommand.get_quantization_configmodel_idcCs*|jjdur |jj}d|vr|S|dS)aR Applies the `force_model` CLI argument and canonicalizes the model name to the format "model_id@revision". If the model_id DOESN'T contain an @, it defaults to "model_id@main". Args: model_id (`str`): The model ID. Returns: `str`: The canonicalized model name to be used N@z@main)rr)rrOrXrXrYrs  zServeCommand.process_model_namercCs>|j}td|d|vr|dd\}}n|d}}tj|||jd}|jdvr.|jntt |j}| |}||j |d|jd}|d urK||d <t j|fi|} tt | jd } | j|fi|} t| d d d urs| |j} | jjd uo~| jjd k} | jjd uo| jjdk} | s| rd| j_td|| |fS)a Generic method to load a model and a data processor from a model ID and revision, making use of the serve CLI arguments. Args: model_id_and_revision (`str`): The model ID and revision to load. model_cls (`type[PreTrainedModel]`): The model class to load. Returns: `tuple[PreTrainedModel, Union[ProcessorMixin, PreTrainedTokenizerFast]]`: The loaded model and data processor (tokenizer, audio processor, etc.). zLoading rPr$main)revisionr)rNr)rRrr device_maprNrMr hf_device_mapr\z Loaded model )rrrrr&from_pretrainedrrgetattrrrNrrrrrrrPr max_length)rrrrOrRdata_processorrrM model_kwargsr architecturerhas_default_max_lengthhas_short_max_new_tokensrXrXrY_load_model_and_data_processor.sB   z+ServeCommand._load_model_and_data_processorcCt||jvs |j|r#||\}}t||jj|d|j|<||fS|j||j|j}|j|j}||fS)a\ Loads the text model and processor from the given model ID and revision into the ServeCommand instance. Args: model_id_and_revision (`str`): The model ID and revision to load. Returns: `tuple[PreTrainedModel, PreTrainedTokenizerFast]`: The loaded text model and processor. rr rrr^rrrrrr)rrrrrXrXrYrks     z%ServeCommand.load_model_and_processorcCr_)aU Loads the audio model and processor from the given model ID and revision into the ServeCommand instance. Args: model_id_and_revision (`str`): The model ID and revision to load. Returns: `tuple[PreTrainedModel, ProcessorMixin]`: The loaded audio model and processor. r`ra)rrr:r;rXrXrYr=s     z+ServeCommand.load_audio_model_and_processor)rNNNNNNN).rRrSrT staticmethodrrrrrwr rrr!r%r(rVr rrerrr?rAr{ functoolscacheanyrgrrUrrrrrWrZrbrarrNrr^tuplerr r=rXrXrXrYrs - 1    8_-\ -J'0=  r__main__)rrrrenumrcrr?rrrrr<roargparserrcollections.abcrrr contextlibr dataclassesrr r r typingr r rhuggingface_hubrhuggingface_hub.constantsrtokenizers.decodersrr&transformers.models.auto.modeling_autorrtransformers.utils.import_utilsrrrrrrrrrrr r!utilsr"r#r%rr&r'r(r)generation.continuous_batchingr*r+r@PILr,rrzrur-r.fastapi.middleware.corsr/fastapi.responsesr0r1 openai.types.audio.transcriptionr2.openai.types.audio.transcription_create_paramsr3openai.types.chatr4'openai.types.chat.chat_completion_chunkr5r6r7r8r9*openai.types.chat.completion_create_paramsr:openai.types.responsesr;r<r=r>r?r@rArBrCrDrErFrGrHrI-openai.types.responses.response_create_paramsrJpydanticrKrLrMrOr\r]rr#r&rr$r'rrRrrrerrrnEnumrrrwrrrrrrr{rXrXrXrYs                D      ;2q S