o i7 @s2dZddlZddlZddlmZddlmZddlZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZmZddlmZdd lmZdd lmZzdd lmZdd lm Z dd l!m"Z"m#Z#Wn e$yZ%ze &de%e &de'de%dZ%[%wweGdddeZ(GdddeZ)dS)z4Simli video service for real-time avatar generation.N) dataclass)Optional)logger) BaseModel) CancelFrameEndFrameFrameInterruptionFrameOutputImageRawFrame StartFrameTTSAudioRawFrameTTSStoppedFrameUserStartedSpeakingFrame)FrameDirection) AIService)ServiceSettings) AudioFrame)AudioResampler) SimliClient SimliConfigz Exception: zCIn order to use Simli, you need to `pip install pipecat-ai[simli]`.zMissing module: c@seZdZdZdS)SimliVideoSettingsz%Settings for the Simli video service.N)__name__ __module__ __qualname____doc__rrP/home/ubuntu/.local/lib/python3.10/site-packages/pipecat/services/simli/video.pyr+srcseZdZUdZeZeed<GdddeZdddddddddddd d e e d e e d e e d e d e de de ede e de e de e de effddZdeffdd Zdeffdd Zdeffdd ZddZdd Zd!d"Zded#effd$d% Zd&d'ZZS)(SimliVideoServiceaSimli video service for real-time avatar generation. Provides real-time avatar video generation by processing audio frames and producing synchronized video output using the Simli API. Handles audio resampling, video frame processing, and connection management. _settingsc@sBeZdZUdZdZeeed<dZee ed<dZ ee ed<dS)zSimliVideoService.InputParamsaInput parameters for Simli video configuration. .. deprecated:: 0.0.106 Use ``SimliVideoService.Settings(...)`` instead. Parameters: enable_logging: Whether to enable Simli logging. max_session_length: Absolute maximum session duration in seconds. Avatar will disconnect after this time even if it's speaking. max_idle_time: Maximum duration in seconds the avatar is not speaking before the avatar disconnects. Nenable_loggingmax_session_length max_idle_time) rrrrrrbool__annotations__r intr!rrrr InputParams=s  r%NFzhttps://api.simli.ai) api_keyface_id simli_configuse_turn_server simli_urlis_trinity_avatarparamsr r!rsettingsr&r'r(r)r*r+r,r r!rr-c  stdd} |dur2|d|durt|dr|j}| dur&t|dr&|j} | dur2t|dr2|j} | dur;| | tjdd| i| |dura|dusR|durVt dt j d t d d |}n+|durit d |durqt d d|i}|dur}||d<| dur| |d<t di|}|rt j dt d d d|_|jd7_|jd7_t|||dd|_d|_t|_tddd|_d|_d|_||_||_t|_dS)aInitialize the Simli video service. Args: api_key: Simli API key for authentication. face_id: Simli Face ID. For Trinity avatars, specify "faceId/emotionId" to use a different emotion than the default. simli_config: Configuration object for Simli client settings. Use api_key and face_id instead. .. deprecated:: 0.0.92 The 'simli_config' parameter is deprecated and will be removed in a future version. Please use 'api_key' and 'face_id' parameters instead. use_turn_server: Whether to use TURN server for connection. Defaults to False. .. deprecated:: 0.0.95 The 'use_turn_server' parameter is deprecated and will be removed in a future version. simli_url: URL of the simli servers. Can be changed for custom deployments of enterprise users. is_trinity_avatar: Boolean to tell simli client that this is a Trinity avatar which reduces latency when using Trinity. params: Additional input parameters for session configuration. .. deprecated:: 0.0.106 Use ``settings=SimliVideoService.Settings(...)`` instead. max_session_length: Absolute maximum session duration in seconds. Avatar will disconnect after this time even if it's speaking. max_idle_time: Maximum duration in seconds the avatar is not speaking before the avatar disconnects. enable_logging: Whether to enable Simli logging. settings: Service settings. **kwargs: Additional arguments passed to the parent AIService. N)modelr,r r!rr-zrCannot specify both simli_config and api_key/face_id. Please use api_key and face_id (simli_config is deprecated).zThe 'simli_config' parameter is deprecated and will be removed in a future version. Please use 'api_key' and 'face_id' parameters instead, with optional 'params' for max_session_length and max_idle_time configuration.) stacklevelzapi_key is requiredzface_id is requiredfaceIdmaxSessionLength maxIdleTimezVThe 'use_turn_server' parameter is deprecated and will be removed in a future version.FT)r&configsimliURL enableSFUs16monoi>r)r"_warn_init_param_moved_to_settingshasattrr r!r apply_updatesuper__init__ ValueErrorwarningswarnDeprecationWarningr _initializedr3r2r _simli_client_pipecat_resamplerasyncioEvent_pipecat_resampler_eventr_simli_resampler _audio_task _video_task_is_trinity_avatar_previously_interrupted bytearray _audio_buffer)selfr&r'r(r)r*r+r,r r!rr-kwargsdefault_settingsr5 config_kwargs __class__rrr>Osr 4     zSimliVideoService.__init__framec&t|IdH|IdHdS)zStart the Simli video service. Args: frame: The start frame containing initialization parameters. N)r=start_start_connectionrPrVrTrrrXzSimliVideoService.startcrW)zWStop the Simli video service. Args: frame: The end frame. N)r=stop_stop_connectionrZrTrrr\r[zSimliVideoService.stopcrW)z\Cancel the Simli video service. Args: frame: The cancel frame. N)r=cancelr]rZrTrrr^r[zSimliVideoService.cancelc sz)|js|jIdHd|_|jIdH|||_|||_WdSt yJ}z|j d||dIdHWYd}~dSd}~ww)zAStart the connection to Simli service and begin processing tasks.NTzUnable to start connection:  error_msg exception) rCrDrX sendSilence create_task_consume_and_process_audiorJ_consume_and_process_videorK Exception push_error)rPerrrrYs(z#SimliVideoService._start_connectionc s||jIdH|j}|2z*3dHW}|j|}|D]}|}|r9|t | |jj ddIdHqq6dS)z9Consume audio frames from Simli and push them downstream.N)audio sample_rate num_channels) rHwaitrDgetAudioStreamIteratorrEresample to_ndarrayany push_framer tobytesrate)rPaudio_iterator audio_frameresampled_framesresampled_frame audio_arrayrrrrds$   z,SimliVideoService._consume_and_process_audiocsr|jIdH|jjdd}|2z#3dHW}t||j|j fdd}|j |_ | |IdHq6dS)zBConsume video frames from Simli and convert them to output frames.Nrgb24) targetFormatRGB)imagesizeformat) rHrmrDgetVideoStreamIteratorr to_rgbto_imagerswidthheightptsrr)rPvideo_iterator video_frameconvertedFramerrrres z,SimliVideoService._consume_and_process_video directionc st||IdHt|trztjtj|jtj ddddf|j dkr(dndd}|j |_ |j durCt d|j|j |_ |j|j|}|D]c}|tj }|jr|j|t|jdkrz*|jdD]}|j|tj qnW|j|jIdHd |_t|_qK|j|jIdHd |_t|_wqK|j|IdHqKWdSty}z|jd ||d IdHWYd}~nid}~wwt|t rz$|jrt|jd kr|j|jIdHd |_t|_WdSWdSWdSty}z|jd ||d IdHWYd}~dSd}~wwt|t!t"fr5|js1|j#IdH|j$|_|%||IdHdS)zProcess incoming frames and handle Simli video generation. Args: frame: The frame to process. direction: The direction of frame processing. N)dtyperir9stereo)layoutr8iFzError sending audio: r_rzError stopping TTS: )&r= process_frame isinstancer r from_ndarraynp frombufferrjint16rlrkrErrrHsetrIrorpastypersrMrOextendlenrD playImmediaterNsendrfrgr r r clearBufferrLrr) rPrVr old_framerwrx audioBytes flushFramerhrTrrr sx         &  $zSimliVideoService.process_framecsV|jIdH|jr||jIdHd|_|jr)||jIdHd|_dSdS)z2Stop the Simli client and cancel processing tasks.N)rDr\rJ cancel_taskrK)rPrrrr][s z"SimliVideoService._stop_connection)rrrrrSettingsr#rr%rstrrr"r$r>r rXrr\rr^rYrdrerrrr] __classcell__rrrTrr2s`          ;r)*rrFr@ dataclassesrtypingrnumpyrlogururpydanticrpipecat.frames.framesrrrr r r r r r"pipecat.processors.frame_processorrpipecat.services.ai_servicerpipecat.services.settingsrav.audio.framerav.audio.resamplerrsimlirrModuleNotFoundErrorrherrorrfrrrrrrs2    ,