o iRp@sdZddlZddlZddlmZmZddlmZmZm Z m Z m Z ddl m Z ddlmZddlmZmZmZmZmZmZmZmZmZmZddlmZdd lmZmZm Z dd l!m"Z"m#Z#dd l$m%Z%dd l&m'Z'm(Z(dd l)m*Z*ddl+m,Z,ddl-m.Z.zddl/m0Z1ddl2m3Z3Wn e4ydZ1dZ3YnweGddde'j5Z6Gddde'Z7dZ8eGdddeZ9Gddde%Z:dS)a$OpenAI Speech-to-Text service implementations. Provides two STT services: - ``OpenAISTTService``: REST-based transcription using the Audio API (Whisper / GPT-4o). - ``OpenAIRealtimeSTTService``: WebSocket-based streaming transcription using the Realtime API in transcription-only mode. N) dataclassfield)AnyAsyncGeneratorLiteralOptionalUnion)logger)create_stream_resampler) CancelFrameEndFrameFrameInterimTranscriptionFrame StartFrameTranscriptionFrameUserStartedSpeakingFrameUserStoppedSpeakingFrameVADUserStartedSpeakingFrameVADUserStoppedSpeakingFrame)FrameDirection) NOT_GIVEN STTSettings _NotGiven)OPENAI_REALTIME_TTFS_P99OPENAI_TTFS_P99)WebsocketSTTService)BaseWhisperSTTService Transcription)Language)time_now_iso8601) traced_stt)connect)Statec@seZdZdZdS)OpenAISTTSettingsz$Settings for the OpenAI STT service.N)__name__ __module__ __qualname____doc__r(r(O/home/ubuntu/.local/lib/python3.10/site-packages/pipecat/services/openai/stt.pyr#9sr#cseZdZUdZeZeed<dddejddde dde e de e de e de ed e e d e e d e ed e e ffd dZ dedefddZZS)OpenAISTTServicezOpenAI Speech-to-Text service that generates text from audio. Uses OpenAI's transcription API to convert audio to text. Requires an OpenAI API key set via the api_key parameter or OPENAI_API_KEY environment variable. _settingsN)modelapi_keybase_urllanguageprompt temperaturesettingsttfs_p99_latencyr,r-r.r/r0r1r2r3c  s|ptj} |jd| ddd} |dur|dd|| _|dur(|dd|| _|dur5|dd|| _|dur>| |tj d||| |d| dS) uwInitialize OpenAI STT service. Args: model: Model to use — either gpt-4o or Whisper. .. deprecated:: 0.0.105 Use ``settings=OpenAISTTService.Settings(model=...)`` instead. api_key: OpenAI API key. Defaults to None. base_url: API base URL. Defaults to None. language: Language of the audio input. Defaults to English. .. deprecated:: 0.0.105 Use ``settings=OpenAISTTService.Settings(language=...)`` instead. prompt: Optional text to guide the model's style or continue a previous segment. .. deprecated:: 0.0.105 Use ``settings=OpenAISTTService.Settings(prompt=...)`` instead. temperature: Optional sampling temperature between 0 and 1. Defaults to 0.0. .. deprecated:: 0.0.105 Use ``settings=OpenAISTTService.Settings(temperature=...)`` instead. settings: Runtime-updatable settings. When provided alongside deprecated parameters, ``settings`` values take precedence. ttfs_p99_latency: P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark **kwargs: Additional arguments passed to BaseWhisperSTTService. gpt-4o-transcribeN)r,r/r0r1r,r0r1)r-r.r2r3r() rENSettings"_warn_init_param_moved_to_settingsr,r0r1 apply_updatesuper__init__) selfr,r-r.r/r0r1r2r3kwargs _languagedefault_settings __class__r(r)r:Js4 -     zOpenAISTTService.__init__audioreturncs|jjdus Jd|df|jj|jjd}|jr-|jjdvr)d|d<dg|d<nd |d<|jjdur9|jj|d <|jjdurE|jj|d <|jjjj d i|IdHS) Nz audio.wavz audio/wav)filer,r/)r4zgpt-4o-mini-transcribejsonresponse_formatlogprobsinclude verbose_jsonr0r1r() r+r/r,_include_prob_metricsr0r1_clientrAtranscriptionscreate)r;rAr<r(r(r) _transcribes       zOpenAISTTService._transcribe)r$r%r&r'r#r6__annotations__rr5rrstrfloatr:bytesrrM __classcell__r(r(r?r)r*@s<   Nr*i]c@sReZdZUdZedddZedBeBed<edddZ e ddBeBed <dS) OpenAIRealtimeSTTSettingsa/Settings for OpenAIRealtimeSTTService. Parameters: prompt: Optional prompt text to guide transcription style. noise_reduction: Noise reduction mode. ``"near_field"`` for close microphones, ``"far_field"`` for distant microphones, or ``None`` to disable. cCtSNrr(r(r(r)z"OpenAIRealtimeSTTSettings.)default_factoryNr0cCrTrUrVr(r(r(r)rWrX near_field far_fieldnoise_reduction) r$r%r&r'rr0rOrrNr]rr(r(r(r)rSs  rScs:eZdZUdZeZeed<ddejddddde d de d e e d e d e ed e e d e e e edfde eddede ede effddZed ede fddZdefddZdede e efffdd Zdeffdd Zdeffd d! Zdeffd"d# Zd$edee dffd%d&Z!de d'e"ffd(d) Z#fd*d+Z$fd,d-Z%d.d/Z&d0d1Z'd2e fd3d4Z(d5d6Z)d$efd7d8Z*d9d:Z+d;d<Z,d=d>Z-d?e fd@dAZ.d?e fdBdCZ/d?e fdDdEZ0d?e fdFdGZ1e2 dTdHe dIed e efdJdKZ3d?e fdLdMZ4d?e fdNdOZ5d?e fdPdQZ6d?e fdRdSZ7Z8S)UOpenAIRealtimeSTTServiceuOpenAI Realtime Speech-to-Text service using WebSocket transcription sessions. Uses OpenAI's Realtime API in transcription-only mode for real-time streaming speech recognition with optional server-side VAD and noise reduction. The model does not generate conversational responses — only transcription output. This service supports two VAD modes: **Local VAD** (default): Disable server-side VAD and use a local VAD processor in the pipeline instead. When a ``VADUserStoppedSpeakingFrame`` is received, the service commits the audio buffer so that the server begins transcription for the completed speech segment. **Server-side VAD** (``turn_detection=None``): The OpenAI server performs voice-activity detection. The service broadcasts ``UserStartedSpeakingFrame`` and ``UserStoppedSpeakingFrame`` when the server detects speech boundaries. Do **not** use a separate VAD processor in the pipeline in this mode. Audio is sent as 24 kHz 16-bit mono PCM as required by the OpenAI Realtime API. If the pipeline runs at a different sample rate (e.g. 16 kHz for Silero VAD compatibility), audio is automatically upsampled before sending. Example:: stt = OpenAIRealtimeSTTService( api_key="sk-...", settings=OpenAIRealtimeSTTService.Settings( model="gpt-4o-transcribe", noise_reduction="near_field", ), ) r+Nz wss://api.openai.com/v1/realtimeFT) r,r.r/r0turn_detectionr]should_interruptr2r3r-r,r.r/r0r_r]rZr`r2r3c  stdurtd|jdtjddd} |dur|dd|| _|dur1|tjkr1|dd|| _|dur>|dd|| _|durK|dd|| _ | durT| | t j d | | d | ||_ ||_||_||_d|_d |_t|_|d u|_dS) aInitialize the OpenAI Realtime STT service. Args: api_key: OpenAI API key for authentication. model: Transcription model. Supported values are ``"gpt-4o-transcribe"`` and ``"gpt-4o-mini-transcribe"``. .. deprecated:: 0.0.105 Use ``settings=OpenAIRealtimeSTTService.Settings(model=...)`` instead. base_url: WebSocket base URL for the Realtime API. Defaults to ``"wss://api.openai.com/v1/realtime"``. language: Language of the audio input. Defaults to English. .. deprecated:: 0.0.105 Use ``settings=OpenAIRealtimeSTTService.Settings(language=...)`` instead. prompt: Optional prompt text to guide transcription style or provide keyword hints. .. deprecated:: 0.0.105 Use ``settings=OpenAIRealtimeSTTService.Settings(prompt=...)`` instead. turn_detection: Server-side VAD configuration. Defaults to ``False`` (disabled), which relies on a local VAD processor in the pipeline. Pass ``None`` to use server defaults (``server_vad``), or a dict with custom settings (e.g. ``{"type": "server_vad", "threshold": 0.5}``). noise_reduction: Noise reduction mode. ``"near_field"`` for close microphones, ``"far_field"`` for distant microphones, or ``None`` to disable. .. deprecated:: 0.0.106 Use ``settings=OpenAIRealtimeSTTService.Settings(noise_reduction=...)`` instead. should_interrupt: Whether to interrupt bot output when speech is detected by server-side VAD. Only applies when turn detection is enabled. Defaults to True. settings: Runtime-updatable settings. When provided alongside deprecated parameters, ``settings`` values take precedence. ttfs_p99_latency: P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark **kwargs: Additional arguments passed to parent WebsocketSTTService. Nzdwebsockets is required for OpenAIRealtimeSTTService. Install it with: pip install pipecat-ai[openai]r4)r,r/r0r]r,r/r0r])r3r2Fr()websocket_connect ImportErrorr6rr5r7r,r/r0r]r8r9r:_api_key _base_url_turn_detection_should_interrupt _receive_task_session_readyr _resampler_server_vad_enabled) r;r-r,r.r/r0r_r]r`r2r3r<r>r?r(r)r:sL;     z!OpenAIRealtimeSTTService.__init__rBcCst|ddS)zConvert a Language enum value to an ISO-639-1 code. Args: language: The Language enum value. Returns: Two-letter ISO-639-1 language code. -r)rOsplitlower)r/r(r(r)_language_to_code_s z*OpenAIRealtimeSTTService._language_to_codecCsdS)zCheck if the service can generate processing metrics. Returns: True, as this service supports metrics generation. Tr(r;r(r(r)can_generate_metricslsz-OpenAIRealtimeSTTService.can_generate_metricsdeltacs0t|IdH}|r|jr|IdH|S)aTApply a settings delta and send session update if needed. Sends a ``session.update`` to the server when the session is active. Args: delta: A :class:`STTSettings` (or ``OpenAIRealtimeSTTService.Settings``) delta. Returns: Dict mapping changed field names to their previous values. N)r9_update_settingsrh_send_session_update)r;rqchangedr?r(r)rrts  z)OpenAIRealtimeSTTService._update_settingsframec&t|IdH|IdHdS)zStart the service and establish WebSocket connection. Args: frame: The start frame triggering service initialization. N)r9start_connectr;rur?r(r)rwzOpenAIRealtimeSTTService.startcrv)zStop the service and close WebSocket connection. Args: frame: The end frame triggering service shutdown. N)r9stop _disconnectryr?r(r)r{rzzOpenAIRealtimeSTTService.stopcrv)zCancel the service and close WebSocket connection. Args: frame: The cancel frame triggering service cancellation. N)r9cancelr|ryr?r(r)r}rzzOpenAIRealtimeSTTService.cancelrAcCs||IdHdVdS)uSend audio data to the transcription session. Audio is streamed over the WebSocket. Transcription results arrive asynchronously via the receive task and are pushed as ``InterimTranscriptionFrame`` or ``TranscriptionFrame``. Args: audio: Raw audio bytes (16-bit mono PCM at the pipeline sample rate). Automatically resampled to 24 kHz. Yields: None — results are delivered via the WebSocket receive task. N) _send_audio)r;rAr(r(r)run_stts z OpenAIRealtimeSTTService.run_stt directioncs\t||IdH|js*t|tr|IdHdSt|tr,|IdHdSdSdS)aProcess frames from the pipeline. Extends the base STT service to handle local VAD events when server-side VAD is disabled. On ``VADUserStoppedSpeakingFrame``, commits the audio buffer so the server begins transcription for the completed speech segment. Args: frame: The frame to process. direction: The direction of frame flow in the pipeline. N)r9 process_framerj isinstancerstart_processing_metricsr_commit_audio_buffer)r;rurr?r(r)rs   z&OpenAIRealtimeSTTService.process_framecsLtIdH|IdH|jr"|js$|||j|_dSdSdS)z:Connect to the transcription endpoint and start receiving.N)r9rx_connect_websocket _websocketrg create_task_receive_task_handler _report_errorror?r(r)rxs  z!OpenAIRealtimeSTTService._connectcsFtIdH|jr|j|jddIdHd|_|IdHdS)z)Disconnect and clean up background tasks.Ng?)timeout)r9r|rg cancel_task_disconnect_websocketror?r(r)r|s z$OpenAIRealtimeSTTService._disconnectc sz1|jr|jjtjurWdSd|_|jd}t|dd|jidIdH|_|dIdHWdSt yU}z|j d||d IdHd|_WYd}~dSd}~ww) zAEstablish the WebSocket connection to the transcription endpoint.NFz?intent=transcription AuthorizationzBearer )uriadditional_headers on_connectedz)Error connecting to OpenAI Realtime STT:  error_msg exception) rstater"OPENrhrdrarc_call_event_handler Exception push_error)r;urler(r(r)rs(   z+OpenAIRealtimeSTTService._connect_websocketc szLzd|_|jr|jIdHWnty1}z|jd||dIdHWYd}~nd}~wwWd|_|dIdHdSWd|_|dIdHdSd|_|dIdHw)zClose the WebSocket connection.FNzError disconnecting: ron_disconnected)rhrcloserrr)r;rr(r(r)rs*z.OpenAIRealtimeSTTService._disconnect_websocketmessagec sz|js|jr|jt|IdHWdSWdSWdStyI}z!|js*|js1WYd}~dS|jd||dIdHWYd}~dSd}~ww)z|Send a JSON message over the WebSocket. Args: message: The message dict to serialize and send. NzError sending message: r)_disconnectingrsendrDdumpsrr)r;rrr(r(r)_ws_sends   z!OpenAIRealtimeSTTService._ws_sendcsd|jji}|jjr||jjnd}|r||d<|jjr$|jj|d<dtd|d}|jdur6d|d <n |jdur@|j|d <|jjrLd |jji|d <|d d d|iddIdHdS)z?Send ``session.update`` to configure the transcription session.r,Nr/r0z audio/pcm)typerate)format transcriptionFr_rr]zsession.updaterinputrrA)rsession) r+r,r/rnr0_OPENAI_SAMPLE_RATErer]r)r;r language_code input_audior(r(r)rss8     z-OpenAIRealtimeSTTService._send_session_updatecsL|j||jtIdH}|sdSt|d}|d|dIdHdS)zSend audio data via ``input_audio_buffer.append``. Resamples from the pipeline sample rate to 24 kHz if needed. Args: audio: Raw audio bytes at the pipeline sample rate. Nzutf-8zinput_audio_buffer.appendr)riresample sample_raterbase64 b64encodedecoder)r;rApayloadr(r(r)r~Asz$OpenAIRealtimeSTTService._send_audioc|ddiIdHdS)z2Commit the current audio buffer for transcription.rzinput_audio_buffer.commitNrror(r(r)rTz-OpenAIRealtimeSTTService._commit_audio_buffercr)zClear the current audio buffer.rzinput_audio_buffer.clearNrror(r(r)_clear_audio_bufferXrz,OpenAIRealtimeSTTService._clear_audio_bufferc sX|j2z3dHW}zt|}Wntjy tdYqw|dd}|dkr4||IdHq|dkrA||IdHq|dkrN| |IdHq|dkr[| |IdHq|d krh| |IdHq|d kru| |IdHq|d kr| |IdHq|d krtd |ddq|dkr||IdHqtd|q6dS)zReceive and dispatch server events from the transcription session. Called by ``WebsocketService._receive_task_handler`` which wraps this method with automatic reconnection on connection errors. Nz!Failed to parse WebSocket messagerzsession.createdzsession.updatedz1conversation.item.input_audio_transcription.deltaz5conversation.item.input_audio_transcription.completedz2conversation.item.input_audio_transcription.failedz!input_audio_buffer.speech_startedz!input_audio_buffer.speech_stoppedzinput_audio_buffer.committedz Audio buffer committed: item_id=item_iderrorzUnhandled event: )rrDloadsJSONDecodeErrorr warningget_handle_session_created_handle_session_updated_handle_transcription_delta_handle_transcription_completed_handle_transcription_failed_handle_speech_started_handle_speech_stoppedtrace _handle_error)r;revtevt_typer(r(r)_receive_messages`s:  z*OpenAIRealtimeSTTService._receive_messagesrcstd|IdHdS)zHandle ``session.created``. Sent immediately after connecting. We respond by configuring the session with our desired settings. Args: evt: The session created event from the server. z4Transcription session created, sending configurationN)r debugrsr;rr(r(r)rs z0OpenAIRealtimeSTTService._handle_session_createdcstdd|_dS)zHandle ``session.updated``. The session is now fully configured and ready to transcribe. Args: evt: The session updated event from the server. z*Transcription session configured and readyTN)r rrhrr(r(r)rs  z0OpenAIRealtimeSTTService._handle_session_updatedcs:|dd}|r|t||jt|dIdHdSdS)a+Handle incremental transcription text. For ``gpt-4o-transcribe`` and ``gpt-4o-mini-transcribe``, deltas contain streaming partial text. For ``whisper-1``, each delta contains the full turn transcript. Args: evt: The delta event from the server. rqrresultN)r push_framer_user_idr)r;rrqr(r(r)rs z4OpenAIRealtimeSTTService._handle_transcription_deltacsZ|dd}|r+|t||jt|dIdH||dIdH|IdHdSdS)zHandle a completed transcription for a speech segment. Pushes a ``TranscriptionFrame`` and records the result for tracing. Args: evt: The completed event containing the full transcript. transcriptrrNT)rrrrr_handle_transcription_tracestop_processing_metrics)r;rrr(r(r)rs  z8OpenAIRealtimeSTTService._handle_transcription_completedris_finalcsdS)zRecord transcription result for tracing. Args: transcript: The transcribed text. is_final: Whether this is a final transcription result. language: Optional language of the transcription. Nr()r;rrr/r(r(r)rsz4OpenAIRealtimeSTTService._handle_transcription_tracecsBtd|tIdH|jr|IdH|IdHdS)zHandle server-side VAD speech start. Broadcasts ``UserStartedSpeakingFrame`` and optionally triggers interruption of current bot output. Args: evt: The ``input_audio_buffer.speech_started`` event. zServer VAD: speech startedN)r rbroadcast_framerrfbroadcast_interruptionrrr(r(r)rs  z/OpenAIRealtimeSTTService._handle_speech_startedcs td|tIdHdS)aHandle server-side VAD speech stop. Broadcasts ``UserStoppedSpeakingFrame``. The audio buffer is automatically committed by the server when VAD is enabled. Args: evt: The ``input_audio_buffer.speech_stopped`` event. zServer VAD: speech stoppedN)r rrrrr(r(r)rs z/OpenAIRealtimeSTTService._handle_speech_stoppedcs6|di}|dd}|jd|dIdHdS)uHandle a transcription failure for a speech segment. Logs the error but does not treat it as fatal — the session remains active for subsequent turns. Args: evt: The failed event containing error details. rrzTranscription failedzOpenAI Realtime STT error: rN)rr)r;rrrr(r(r)rs z5OpenAIRealtimeSTTService._handle_transcription_failedcsP|di}|dd}|dd}d|d|}|j|dId Ht|) zHandle a fatal error from the transcription session. Raises an exception so that ``WebsocketService`` can decide whether to attempt reconnection. Args: evt: The error event. rrz Unknown errorcoderzOpenAI Realtime STT error [z]: rN)rrr)r;rrr error_codemsgr(r(r)rs  z&OpenAIRealtimeSTTService._handle_errorrU)9r$r%r&r'rSr6rNrr5rrOrrdictrboolrPr: staticmethodrnrprrrrrrwr r{r r}rQrr rrrrxr|rrrrsr~rrrrrrrr rrrrrrRr(r(r?r)r^s "    q      -$     r^);r'rrD dataclassesrrtypingrrrrrlogurur pipecat.audio.utilsr pipecat.frames.framesr r r rrrrrrr"pipecat.processors.frame_processorrpipecat.services.settingsrrrpipecat.services.stt_latencyrrpipecat.services.stt_servicer!pipecat.services.whisper.base_sttrrpipecat.transcriptions.languagerpipecat.utils.timer(pipecat.utils.tracing.service_decoratorsr websockets.asyncio.clientr!rawebsockets.protocolr"ModuleNotFoundErrorr6r#r*rrSr^r(r(r(r)s<   0       t