o iUG @sdZddlZddlmZmZddlmZddlmZm Z ddl Z ddl m Z ddlmZmZddlmZmZmZdd lmZmZmZdd lmZdd lmZmZdd lmZdd l m!Z!erzddl"m#Z#Wn e$yZ%ze &de%e &de'de%dZ%[%wwzddl(Z(Wn e$yZ%ze &de%e &de'de%dZ%[%wwGdddeZ)GdddeZ*dede e+fddZ,eGdddeZ-eGdddeZ.Gdd d eZ/Gd!d"d"e/Z0dS)#zWhisper speech-to-text services with locally-downloaded models. This module implements Whisper transcription using locally-downloaded models, supporting both Faster Whisper and MLX Whisper backends for efficient inference. N) dataclassfield)Enum)AsyncGeneratorOptional)logger) TYPE_CHECKINGoverride) ErrorFrameFrameTranscriptionFrame) NOT_GIVEN STTSettings _NotGiven)SegmentedSTTService)Languageresolve_language)time_now_iso8601) traced_stt WhisperModel Exception: GIn order to use Whisper, you need to `pip install pipecat-ai[whisper]`.zMissing module: zKIn order to use Whisper, you need to `pip install pipecat-ai[mlx-whisper]`.c@s0eZdZdZdZdZdZdZdZdZ dZ d Z d S) ModelaWhisper model selection options for Faster Whisper. Provides various model sizes and specializations for speech recognition, balancing quality and performance based on use case requirements. Parameters: TINY: Smallest multilingual model, fastest inference. BASE: Basic multilingual model, good speed/quality balance. SMALL: Small multilingual model, better speed/quality balance than BASE. MEDIUM: Medium-sized multilingual model, better quality. LARGE: Best quality multilingual model, slower inference. LARGE_V3_TURBO: Fast multilingual model, slightly lower quality than LARGE. DISTIL_LARGE_V2: Fast multilingual distilled model. DISTIL_MEDIUM_EN: Fast English-only distilled model. tinybasesmallmediumzlarge-v3z)deepdml/faster-whisper-large-v3-turbo-ct2z&Systran/faster-distil-whisper-large-v2z'Systran/faster-distil-whisper-medium.enN) __name__ __module__ __qualname____doc__TINYBASESMALLMEDIUMLARGELARGE_V3_TURBODISTIL_LARGE_V2DISTIL_MEDIUM_ENr*r*P/home/ubuntu/.local/lib/python3.10/site-packages/pipecat/services/whisper/stt.pyr-src@s(eZdZdZdZdZdZdZdZdZ dS) MLXModelasMLX Whisper model selection options for Apple Silicon. Provides various model sizes optimized for Apple Silicon hardware, including quantized variants for improved performance. Parameters: TINY: Smallest multilingual model for MLX. MEDIUM: Medium-sized multilingual model for MLX. LARGE_V3: Best quality multilingual model for MLX. LARGE_V3_TURBO: Finetuned, pruned Whisper large-v3, much faster with slightly lower quality. DISTIL_LARGE_V3: Fast multilingual distilled model for MLX. LARGE_V3_TURBO_Q4: LARGE_V3_TURBO quantized to Q4 for reduced memory usage. zmlx-community/whisper-tinyz mlx-community/whisper-medium-mlxz"mlx-community/whisper-large-v3-mlxz$mlx-community/whisper-large-v3-turboz%mlx-community/distil-whisper-large-v3z'mlx-community/whisper-large-v3-turbo-q4N) rrr r!r"r%LARGE_V3r'DISTIL_LARGE_V3LARGE_V3_TURBO_Q4r*r*r*r+r,Ksr,languagereturncCsitjdtjdtjdtjdtjdtjdtjdtjdtj d tj d tj d tj d tj d tjdtjdtjdtjdtjdtjdtjdtjdtjdtjdtjdtjdtjdtjdtjdtjdtjdi }t||dd S)!a@Maps pipecat Language enum to Whisper language codes. Args: language: A Language enum value representing the input language. Returns: str or None: The corresponding Whisper language code, or None if not supported. Note: Only includes languages officially supported by Whisper. arbncsdadeelenesfafifrhihuiditjakonlplptrorusksvthtrukurvizhT) use_base_code) rARBNCSDADEELENESFAFIFRHIHUIDITJAKONLPLPTRORUSKSVTHTRUKURVIZHr)r0 LANGUAGE_MAPr*r*r+language_to_whisper_languagecsd    "$?rpc@s,eZdZUdZedddZeeBed<dS)WhisperSTTSettingszSettings for WhisperSTTService. Parameters: no_speech_prob: Probability threshold for filtering non-speech segments. cCtSNr r*r*r*r+zWhisperSTTSettings.default_factoryno_speech_probN) rrr r!rryfloatr__annotations__r*r*r*r+rqs rqc@s`eZdZUdZedddZeeBed<edddZ eeBed<edddZ e eBed <d S) WhisperMLXSTTSettingszSettings for WhisperMLXSTTService. Parameters: no_speech_prob: Probability threshold for filtering non-speech segments. temperature: Sampling temperature (0.0-1.0). engine: Whisper engine identifier. cCrrrsrtr*r*r*r+rurvzWhisperMLXSTTSettings.rwrycCrrrsrtr*r*r*r+rurv temperaturecCrrrsrtr*r*r*r+rurvengineN) rrr r!rryrzrr{r}r~strr*r*r*r+r|s r|cseZdZUdZeZeed<ddddddddeee Bded ed ee d ee d eef fd dZ de fddZd e deefddZddZe ddede d ee fddZdedeedffddZZS)WhisperSTTServicezClass to transcribe audio with a locally-downloaded Whisper model. This service uses Faster Whisper to perform speech-to-text transcription on audio segments. It supports multiple languages and various model sizes. _settingsNautodefault)modeldevice compute_typeryr0settingsrrrryr0rc  s|jtjjtjdd}|dur |ddt|tr|n|j|_ |dur-|dd||_ |dur:|dd||_ |durC| |t jdd|i|||_||_d|_|dS) aInitialize the Whisper STT service. Args: model: The Whisper model to use for transcription. Can be a Model enum or string. .. deprecated:: 0.0.105 Use ``settings=WhisperSTTService.Settings(model=...)`` instead. device: The device to run inference on ('cpu', 'cuda', or 'auto'). Defaults to ``"auto"``. compute_type: The compute type for inference ('default', 'int8', 'int8_float16', etc.). Defaults to ``"default"``. no_speech_prob: Probability threshold for filtering out non-speech segments. .. deprecated:: 0.0.105 Use ``settings=WhisperSTTService.Settings(no_speech_prob=...)`` instead. language: The default language for transcription. .. deprecated:: 0.0.105 Use ``settings=WhisperSTTService.Settings(language=...)`` instead. settings: Runtime-updatable settings. When provided alongside deprecated parameters, ``settings`` values take precedence. **kwargs: Additional arguments passed to SegmentedSTTService. g?)rr0ryNrryr0rr*)Settingsrr)valuerrW"_warn_init_param_moved_to_settings isinstancerrryr0 apply_updatesuper__init___device _compute_type_model_load) selfrrrryr0rkwargsdefault_settings __class__r*r+rs2&      zWhisperSTTService.__init__r1cCdS)zIndicates whether this service can generate metrics. Returns: bool: True, as this service supports metric generation. Tr*rr*r*r+can_generate_metricssz&WhisperSTTService.can_generate_metricscCst|S)zConvert from pipecat Language to Whisper language code. Args: language: The Language enum value to convert. Returns: str or None: The corresponding Whisper language code, or None if not supported. )rp)rr0r*r*r+language_to_service_language's z.WhisperSTTService.language_to_service_languagec Cszddlm}td||jj|j|jd|_tdWdSt yB}zt d|t dd|_WYd}~dSd}~ww) zLoads the Whisper model. Note: If this is the first time this model is being run, it will take time to download from the Hugging Face model hub. rrzLoading Whisper model...)rrzLoaded Whisper modelrrN) faster_whisperrrdebugrrrrrModuleNotFoundErrorerror)rrer*r*r+r2s   zWhisperSTTService._load transcriptis_finalcdSz+Handle a transcription result with tracing.Nr*rrrr0r*r*r+_handle_transcriptionFz'WhisperSTTService._handle_transcriptionaudiocCs|js tdVdS|IdHtj|tjdtjd}tj |jj ||j j dIdH\}}d}|D]}|j |j j krF||jd7}q5|IdH|rs||d|j j IdHtd |d t||jt|j j VdSdS) aTranscribe audio data using Whisper. Args: audio: Raw audio bytes in 16-bit PCM format. Yields: Frame: Either a TranscriptionFrame containing the transcribed text or an ErrorFrame if transcription fails. Note: The audio is expected to be 16-bit signed PCM data. The service will normalize it to float32 in the range [-1, 1]. zWhisper model not availableNdtype@)r0 TTranscription: [])rr start_processing_metricsnp frombufferint16astypefloat32asyncio to_thread transcriberr0rytextstop_processing_metricsrrrr _user_idr)rr audio_floatsegments_rsegmentr*r*r+run_sttMs2  zWhisperSTTService.run_sttrs)rrr r!rqrr{rrrrzrrboolrrrrrbytesrr r __classcell__r*r*rr+rsH  J  "rc @seZdZUdZeZeed<dddddddeee Bdee dee dee d eef d d Z e d d Ze ddededee fddZe dedeedffddZdS)WhisperSTTServiceMLXzSubclass of `WhisperSTTService` with MLX Whisper model support. This service uses MLX Whisper to perform speech-to-text transcription on audio segments. It's optimized for Apple Silicon and supports multiple languages and quantizations. rN)rryr0r}rrryr0r}rcKs|jtjjtjdddd}|dur"|ddt|tr|n|j|_ |dur/|dd||_ |dur<|dd||_ |durI|d d ||_ |durR| |tj|fd |i|dS) aInitialize the MLX Whisper STT service. Args: model: The MLX Whisper model to use for transcription. Can be an MLXModel enum or string. .. deprecated:: 0.0.105 Use ``settings=WhisperSTTServiceMLX.Settings(model=...)`` instead. no_speech_prob: Probability threshold for filtering out non-speech segments. .. deprecated:: 0.0.105 Use ``settings=WhisperSTTServiceMLX.Settings(no_speech_prob=...)`` instead. language: The default language for transcription. .. deprecated:: 0.0.105 Use ``settings=WhisperSTTServiceMLX.Settings(language=...)`` instead. temperature: Temperature for sampling. Can be a float or tuple of floats. .. deprecated:: 0.0.105 Use ``settings=WhisperSTTServiceMLX.Settings(temperature=...)`` instead. settings: Runtime-updatable settings. When provided alongside deprecated parameters, ``settings`` values take precedence. **kwargs: Additional arguments passed to SegmentedSTTService. g333333?mlx)rr0ryr}r~Nrryr0r}r)rr,r"rrrWrrrrryr0r}rrr)rrryr0r}rrrr*r*r+rs8&      zWhisperSTTServiceMLX.__init__cCr)z7MLX Whisper loads models on demand, so this is a no-op.Nr*rr*r*r+rszWhisperSTTServiceMLX._loadrrcrrr*rr*r*r+rrz*WhisperSTTServiceMLX._handle_transcriptionrr1c Cs\zddl}|IdHtj|tjdtjd}tj|j ||j j |j j |j j dIdH}d}|dgD] }|ddd krBq7|d d |j jkrW||d dd 7}q7t|dkrbd}|IdH|r||d|j j IdHtd|dt||jt|j j VWdSWdSty}ztd|dVWYd}~dSd}~ww)aTranscribe audio data using MLX Whisper. The audio is expected to be 16-bit signed PCM data. MLX Whisper will handle the conversion internally. Args: audio: Raw audio bytes in 16-bit PCM format. Yields: Frame: Either a TranscriptionFrame containing the transcribed text or an ErrorFrame if transcription fails. rNrr)path_or_hf_repor}r0rrcompression_ratiogrq?ryrrrTrrzUnknown error occurred: )r) mlx_whisperrrrrrrrrrrrr}r0getrylenstriprrrrr rr Exceptionr )rrrrchunkrrrr*r*r+rsH   zWhisperSTTServiceMLX.run_sttrs)rrr r!r|rr{rrr,rzrrr rrrrrrr rr*r*r*r+rysB   K   r)1r!r dataclassesrrenumrtypingrrnumpyrlogururtyping_extensionsrr pipecat.frames.framesr r r pipecat.services.settingsr rrpipecat.services.stt_servicerpipecat.transcriptions.languagerrpipecat.utils.timer(pipecat.utils.tracing.service_decoratorsrrrrrrrrrr,rrprqr|rrr*r*r*r+sR        N /