o ie @sdZddlZddlmZddlmZmZddlm Z m Z m Z m Z m Z ddlmZddlmZddlmZmZmZmZmZmZmZdd lmZmZmZdd lmZdd l m!Z!m"Z"dd l#m$Z$m%Z%dd l&m'Z'ddl(m)Z)zddl*Z+Wn e,yZ-ze.de-e.de/de-dZ-[-wwde$de e0fddZ1eGdddeZ2eGdddeZ3Gddde"Z4Gddde!Z5dS)zYNVIDIA Riva Speech-to-Text service implementations for real-time and batch transcription.N)CancelledError) dataclassfield)AnyAsyncGeneratorListMappingOptional)logger) BaseModel) CancelFrameEndFrame ErrorFrameFrameInterimTranscriptionFrame StartFrameTranscriptionFrame) NOT_GIVEN STTSettings _NotGiven)NVIDIA_TTFS_P99)SegmentedSTTService STTService)Languageresolve_language)time_now_iso8601) traced_sttz Exception: zNIn order to use NVIDIA Riva STT, you need to `pip install pipecat-ai[nvidia]`.zMissing module: languagereturnc Csitjdtjdtjdtjdtjdtjdtjdtjdtj dtj dtj dtj dtj dtjdtjd tjd tjd tjd tjd tjd tjd tjd tjd i}t||ddS)a@Maps Language enum to NVIDIA Riva ASR language codes. Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-riva-build-table.html?highlight=fr%20fr Args: language: Language enum value. Returns: Optional[str]: NVIDIA Riva language code or None if not supported. zar-ARen-USzen-GBzfr-FRzde-DEzhi-INzit-ITzja-JPzko-KRzpt-BRzru-RUzes-ESzes-USF) use_base_code)rARENEN_USEN_GBFRFR_FRDEDE_DEHIHI_INITIT_ITJAJA_JPKOKO_KRPTPT_BRRURU_RUESES_ESES_USr)r LANGUAGE_MAPr9O/home/ubuntu/.local/lib/python3.10/site-packages/pipecat/services/nvidia/stt.py language_to_nvidia_riva_language*sV    %r;c@seZdZdZdS)NvidiaSTTSettingszSettings for NvidiaSTTService.N)__name__ __module__ __qualname____doc__r9r9r9r:r<^sr<c@seZdZUdZedddZeeBed<edddZ eeBed<edddZ eeBed <ed ddZ e e d BeBed <ed ddZeeBed<d S)NvidiaSegmentedSTTSettingsaSettings for NvidiaSegmentedSTTService. Parameters: profanity_filter: Whether to filter profanity from results. automatic_punctuation: Whether to add automatic punctuation. verbatim_transcripts: Whether to return verbatim transcripts. boosted_lm_words: List of words to boost in language model. boosted_lm_score: Score boost for specified words. cCtSNrr9r9r9r:qz#NvidiaSegmentedSTTSettings.)default_factoryprofanity_filtercCrBrCrDr9r9r9r:rErrFautomatic_punctuationcCrBrCrDr9r9r9r:rEsrFverbatim_transcriptscCrBrCrDr9r9r9r:rEtrFNboosted_lm_wordscCrBrCrDr9r9r9r:rEurFboosted_lm_score)r=r>r?r@rrHboolr__annotations__rIrJrKrstrrLfloatr9r9r9r:rAes  "rAcsdeZdZUdZeZeed<GdddeZddddd d d d e d d e d e de e e fde e de edede ede effddZddZddZdefddZde fddZd effd!d" Zd effd#d$ Zd effd%d& Zd'd(Zd)d*Zd+d,Ze d;d-e d.ed/e efd0d1Z d2d3Z!d4e"de#e$d ffd5d6Z%de"fd7d8Z&d9d:Z'Z(S)<NvidiaSTTServiceaReal-time speech-to-text service using NVIDIA Riva streaming ASR. Provides real-time transcription capabilities using NVIDIA's Riva ASR models through streaming recognition. Supports interim results and continuous audio processing for low-latency applications. _settingsc@s$eZdZUdZejZeeed<dS)zNvidiaSTTService.InputParamsaConfiguration parameters for NVIDIA Riva STT service. .. deprecated:: 0.0.105 Use ``settings=NvidiaSTTService.Settings(...)`` instead. Parameters: language: Target language for transcription. Defaults to EN_US. rN) r=r>r?r@rr#rr rNr9r9r9r: InputParamss  rSgrpc.nvcf.nvidia.com:443z$1598d209-5e27-4d3c-8079-4751568b1081zparakeet-ctc-1.1b-asr function_id model_nameNTservermodel_function_map sample_rateparamsuse_sslsettingsttfs_p99_latencyapi_keyrYrZr[r\r]r^r_c  s|j|dtjd} |dur|d|s|j| _|dur#| |tjd ||| d| ||_ ||_ ||_ d|_ d|_ d|_d|_d|_d|_d|_|d |_d|_d|_d|_d|_dS) a/Initialize the NVIDIA Riva STT service. Args: api_key: NVIDIA API key for authentication. server: NVIDIA Riva server address. Defaults to NVIDIA Cloud Function endpoint. model_function_map: Mapping containing 'function_id' and 'model_name' for the ASR model. sample_rate: Audio sample rate in Hz. If None, uses pipeline default. params: Additional configuration parameters for NVIDIA Riva. .. deprecated:: 0.0.105 Use ``settings=NvidiaSTTService.Settings(...)`` instead. use_ssl: Whether to use SSL for the NVIDIA Riva server. Defaults to True. settings: Runtime-updatable settings. When provided alongside deprecated parameters, ``settings`` values take precedence. ttfs_p99_latency: P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark **kwargs: Additional arguments passed to STTService. rW)modelrNr\r[r_r^rVr9)Settingsgetrr#"_warn_init_param_moved_to_settingsr apply_updatesuper__init___server_api_key_use_ssl_start_history_start_threshold _stop_history_stop_threshold_stop_history_eou_stop_threshold_eou_custom_configuration _function_id _asr_service_queue_config _thread_task selfr`rYrZr[r\r]r^r_kwargsdefault_settings __class__r9r:rks@$    zNvidiaSTTService.__init__cCsBd|jgdd|jgg}tjd|j|j|}tj||_dS)N function-id authorizationBearer ) rvrmrivaclientAuthrnrl ASRServicerwr|metadataauthr9r9r:_initialize_clients z#NvidiaSTTService._initialize_clientc Csntjjtjjtjjj|jjddddd|jdd dd}tj ||j |j |j |j |j|jtj||j|S)5Create the NVIDIA Riva ASR recognition configuration.reFT) encoding language_coderamax_alternativesrHenable_automatic_punctuationrJsample_rate_hertzaudio_channel_count)configinterim_results)rrStreamingRecognitionConfigRecognitionConfig AudioEncoding LINEAR_PCMrRrr[!add_endpoint_parameters_to_configrorprqrsrrrt"add_custom_configuration_to_configru)r|rr9r9r:_create_recognition_configs2  z+NvidiaSTTService._create_recognition_configrcCdS)zCheck if this service can generate processing metrics. Returns: False - this service does not support metrics generation. Fr9r|r9r9r:can_generate_metricsz%NvidiaSTTService.can_generate_metricsracsRddl}||d|jdtddWddS1s"wYdS)aSet the ASR model for transcription. .. deprecated:: 0.0.104 Model cannot be changed after initialization for NVIDIA Riva streaming STT. Set model and function id in the constructor instead, e.g.:: NvidiaSTTService( api_key=..., model_function_map={"function_id": "", "model_name": ""}, ) Args: model: Model name to set. rNalwaysa 'set_model' is deprecated. Model cannot be changed after initialization for NVIDIA Riva streaming STT. Set model and function id in the constructor instead, e.g.: NvidiaSTTService(api_key=..., model_function_map={'function_id': '', 'model_name': ''})) stacklevel)warningscatch_warnings simplefilterwarnDeprecationWarning)r|rarr9r9r: set_model s  "zNvidiaSTTService.set_modelframecs^t|IdH|||_t|_|js#| | |_t d|j jdS)zStart the NVIDIA Riva STT service and initialize streaming configuration. Args: frame: StartFrame indicating pipeline start. Nz)Initialized NvidiaSTTService with model: )rjstartrrryasyncioQueuerxrz create_task_thread_task_handlerr debugrRrar|rrr9r:r(s  zNvidiaSTTService.startc&t|IdH|IdHdS)zStop the NVIDIA Riva STT service and clean up resources. Args: frame: EndFrame indicating pipeline stop. N)rjstop _stop_tasksrrr9r:r9zNvidiaSTTService.stopcr)zCancel the NVIDIA Riva STT service operation. Args: frame: CancelFrame indicating operation cancellation. N)rjcancelrrrr9r:rBrzNvidiaSTTService.cancelcs(|jr||jIdHd|_dSdSrC)rz cancel_taskrr9r9r:rKs  zNvidiaSTTService._stop_taskscCs>|jj||jd}|D]}|jsq t|||q dS)N) audio_chunksstreaming_config)rwstreaming_response_generatorryresultsrrun_coroutine_threadsafe_handle_responseget_event_loop)r| responsesresponser9r9r:_response_handlerPsz"NvidiaSTTService._response_handlercs:zd|_t|jIdHWdStjyd|_w)NTF)_thread_runningr to_threadrrrr9r9r:rZsz%NvidiaSTTService._thread_task_handler transcriptis_finalrcdSz+Handle a transcription result with tracing.Nr9r|rrrr9r9r:_handle_transcriptionbz&NvidiaSTTService._handle_transcriptionc s|jD]U}|r |js q|jdj}|rYt|dkrY|jrF|IdH|t||jt |j j |dIdH|j ||j|j j dIdHq|t ||jt |j j |dIdHqdS)Nr)result)rrr)r alternativesrlenrstop_processing_metrics push_framer_user_idrrRrrr)r|rrrr9r9r:risB      z!NvidiaSTTService._handle_responseaudiocCs,|IdH|j|IdHdVdS)zProcess audio data for speech-to-text transcription. Args: audio: Raw audio bytes to transcribe. Yields: None - transcription results are pushed to the pipeline via frames. N)start_processing_metricsrxput)r|rr9r9r:run_stts  zNvidiaSTTService.run_sttcCs>|jstzt|j|}|WStytw)zGet the next audio chunk for NVIDIA Riva processing. Returns: Audio bytes from the queue. Raises: StopIteration: When the thread is no longer running. ) r StopIterationrrrxrgrrFuturesCancelledError)r|futurer9r9r:__next__s   zNvidiaSTTService.__next__cCs|S)zdReturn iterator for audio chunk processing. Returns: Self as iterator. r9rr9r9r:__iter__rzNvidiaSTTService.__iter__rC))r=r>r?r@r<rfrNr rSrrOrr intrMrPrkrrrrrrr rr rrrrrrrrbytesrrrrr __classcell__r9r9rr:rQxsl       M     " rQcs>eZdZUdZeZeed<GdddeZddddd d d d e d d e d e de e e fde e de edede ede effddZdede e fddZddZde fddZddZdefd d!Zd"effd#d$ Zd%edee efffd&d' Ze d/d(e d)ede efd*d+Zd,ede e!d ffd-d.Z"Z#S)0NvidiaSegmentedSTTServiceaSpeech-to-text service using NVIDIA Riva's offline/batch models. By default, his service uses NVIDIA's Riva Canary ASR API to perform speech-to-text transcription on audio segments. It inherits from SegmentedSTTService to handle audio buffering and speech detection. rRc@sheZdZUdZejZeeed<dZ e ed<dZ e ed<dZ e ed<dZ eeeed <d Zeed <dS) z%NvidiaSegmentedSTTService.InputParamsazConfiguration parameters for NVIDIA Riva segmented STT service. .. deprecated:: 0.0.105 Use ``settings=NvidiaSegmentedSTTService.Settings(...)`` instead. Parameters: language: Target language for transcription. Defaults to EN_US. profanity_filter: Whether to filter profanity from results. automatic_punctuation: Whether to add automatic punctuation. verbatim_transcripts: Whether to return verbatim transcripts. boosted_lm_words: List of words to boost in language model. boosted_lm_score: Score boost for specified words. rFrHTrIrJNrK@rL)r=r>r?r@rr#rr rNrHrMrIrJrKrrOrLrPr9r9r9r:rSs    rSrTz$ee8dc628-76de-4acc-8595-1836e7e857bdz canary-1b-asrrUNTrXr`rYrZr[r\r]r^r_c s|j|dtjdddddd} |dur6|d|s6|jp tj| _|j| _|j| _|j| _|j | _ |j | _ |dur?| |t j d ||| d| ||_||_||_|d |_d |_d |_d |_d |_d |_d |_d |_d|_d|_dS)aRInitialize the NVIDIA Riva segmented STT service. Args: api_key: NVIDIA API key for authentication server: NVIDIA Riva server address (defaults to NVIDIA Cloud Function endpoint) model_function_map: Mapping of model name and its corresponding NVIDIA Cloud Function ID sample_rate: Audio sample rate in Hz. If not provided, uses the pipeline's rate params: Additional configuration parameters for NVIDIA Riva .. deprecated:: 0.0.105 Use ``settings=NvidiaSegmentedSTTService.Settings(...)`` instead. use_ssl: Whether to use SSL for the NVIDIA Riva server. Defaults to True. settings: Runtime-updatable settings. When provided alongside deprecated parameters, ``settings`` values take precedence. ttfs_p99_latency: P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark **kwargs: Additional arguments passed to SegmentedSTTService rWFTNr)rarrHrIrJrKrLr\rbrVrcrdrer9)rfrgrr#rhrrHrIrJrKrLrirjrkrmrlrnrvrorprqrrrsrtruryrwr{rr9r:rksP$    z"NvidiaSegmentedSTTService.__init__rrcCst|S)zConvert pipecat Language enum to NVIDIA Riva's language code. Args: language: Language enum value. Returns: NVIDIA Riva language code or None if not supported. )r;)r|rr9r9r:language_to_service_language,s z6NvidiaSegmentedSTTService.language_to_service_languagecCsP|jdurdSd|jgdd|jgg}tjd|j|j|}tj||_dS)zCInitialize the NVIDIA Riva ASR client with authentication metadata.Nrrr) rwrvrmrrrrnrlrrr9r9r:r7s z,NvidiaSegmentedSTTService._initialize_clientcCs |jjpdS)z1Get the current NVIDIA Riva language code string.r)rRrrr9r9r:_get_language_codeFs z,NvidiaSegmentedSTTService._get_language_codec Cs~|j}tjj|d|j|j|jd}|jr tj ||j|j tj ||j |j |j|j|j|j|jr=tj||j|S)rr)rrrHrrJ)rRrrrrrHrIrJrKadd_word_boosting_to_configrLrrorprqrsrrrtrur)r|srr9r9r:rJs,  z4NvidiaSegmentedSTTService._create_recognition_configcCr)zCheck if this service can generate processing metrics. Returns: True - this service supports metrics generation. Tr9rr9r9r:rkrz.NvidiaSegmentedSTTService.can_generate_metricsrcs>t|IdH|||_td|jjdS)zInitialize the service when the pipeline starts. Args: frame: StartFrame indicating pipeline start. Nz2Initialized NvidiaSegmentedSTTService with model: ) rjrrrryr rrRrarrr9r:rss  zNvidiaSegmentedSTTService.startdeltacs&t|IdH}|r||_|S)zApply a settings delta and sync internal state. Args: delta: A :class:`STTSettings` (or ``NvidiaSegmentedSTTService.Settings``) delta. Returns: Dict mapping changed field names to their previous values. N)rj_update_settingsrry)r|rchangedrr9r:r~s   z*NvidiaSegmentedSTTService._update_settingsrrcrrr9rr9r9r:rrz/NvidiaSegmentedSTTService._handle_transcriptionrc Csz|jdus Jd|jdusJd|IdH|jj||jdd}|IdHt|dr6|}n|}d}t|dg}|D]6}t|dg}|rx|d j }|rxt d |d t ||j t|jjVd }||d |jjIdHqB|st |d WdSWdSty} zt |d| t|dt| VWYd} ~ dSd} ~ wty} zt |d| t|d| dVWYd} ~ dSd} ~ ww)zTranscribe an audio segment. Args: audio: Raw audio bytes in WAV format (already converted by base class). Yields: Frame: TranscriptionFrame containing the transcribed text. NzASR service not initializedzRecognition config not createdF)rrrrrzTranscription: []Tz8: No transcription results found in NVIDIA Riva responsez2: Unexpected response structure from NVIDIA Riva: z*: Unexpected NVIDIA Riva response format: z exception: z error: )error)rwryroffline_recognizerhasattrrgetattrrstripr rrrrrRrrAttributeErrorrrrO Exception) r|r raw_responsertranscription_foundrrrtextaeer9r9r:rsN     &$z!NvidiaSegmentedSTTService.run_sttrC)$r=r>r?r@rArfrNr rSrrOrr rrMrPrkrrrrrrrrrdictrrrrrrrrrr9r9rr:rs`       X !  "r)6r@rconcurrent.futuresrr dataclassesrrtypingrrrrr logurur pydanticr pipecat.frames.framesr r rrrrrpipecat.services.settingsrrrpipecat.services.stt_latencyrpipecat.services.stt_servicerrpipecat.transcriptions.languagerrpipecat.utils.timer(pipecat.utils.tracing.service_decoratorsr riva.clientrModuleNotFoundErrorrrrrOr;r<rArQrr9r9r9r:s>   $     4=