o iQ9@sdZddlmZmZmZddlZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZddlmZmZmZddlmZddlmZdd lm Z m!Z!m"Z"dd l#m$Z$dd l%m&Z&dd l'm(Z(m)Z)Gd dde$Z*Gddde&Z+Gddde)Z,Gddde(Z-dS)a;HeyGen implementation for Pipecat. This module provides integration with the HeyGen platform for creating conversational AI applications with avatars. It manages conversation sessions and provides real-time audio/video streaming capabilities through the HeyGen API. The module consists of three main components: - HeyGenInputTransport: Handles incoming audio and events from HeyGen conversations - HeyGenOutputTransport: Manages outgoing audio and events to HeyGen conversations - HeyGenTransport: Main transport implementation that coordinates input/output transports )AnyOptionalUnionN)logger) AudioRawFrameBotConnectedFrameBotStartedSpeakingFrameBotStoppedSpeakingFrame CancelFrameClientConnectedFrameEndFrameFrameInputAudioRawFrameInterruptionFrameOutputAudioRawFrame StartFrameUserStartedSpeakingFrameUserStoppedSpeakingFrame)FrameDirectionFrameProcessorFrameProcessorSetup)NewSessionRequest)LiveAvatarNewSessionRequest)HeyGenCallbacks HeyGenClient ServiceType)BaseInputTransport)BaseOutputTransport) BaseTransportTransportParamscseZdZdZdedeffdd Zdeffdd Zfd d Z d e ffd d Z d e ffdd Z d effdd ZdefddZdefddZZS)HeyGenInputTransportzInput transport for receiving audio and events from HeyGen conversations. Handles incoming audio streams from participants and manages audio capture from the Daily room connected to the HeyGen conversation. clientparamsc s*tj|fi|||_||_d|_dS)zInitialize the HeyGen input transport. Args: client: The HeyGen transport client instance. params: Transport configuration parameters. **kwargs: Additional arguments passed to parent class. FN)super__init___client_params _initializedselfr!r"kwargs __class__W/home/ubuntu/.local/lib/python3.10/site-packages/pipecat/transports/heygen/transport.pyr$8s  zHeyGenInputTransport.__init__setupc*t|IdH|j|IdHdS)znSetup the input transport. Args: setup: The frame processor setup configuration. Nr#r/r%r)r/r+r-r.r/KzHeyGenInputTransport.setupc&tIdH|jIdHdS)z"Cleanup input transport resources.Nr#cleanupr%r)r+r-r.r6TzHeyGenInputTransport.cleanupframecs8t|IdH|jrdSd|_||IdHdS)z{Start the input transport. Args: frame: The start frame containing initialization parameters. NT)r#startr'set_transport_readyr)r9r+r-r.r:Ys zHeyGenInputTransport.startc(t|IdH|jIdHdS)zpStop the input transport. Args: frame: The end frame signaling transport shutdown. Nr#stopr%r<r+r-r.r?hzHeyGenInputTransport.stopc(t|IdH|jIdHdS)zyCancel the input transport. Args: frame: The cancel frame signaling immediate cancellation. Nr#cancelr%r?r<r+r-r.rCqr@zHeyGenInputTransport.cancelparticipant_idcs8|jjrtd||j||jIdHdSdS)zStart capturing audio from a participant. Args: participant_id: The participant to capture audio from. z6HeyGenTransport start capturing audio for participant N)r&audio_in_enabledrinfor%capture_participant_audio_on_participant_audio_datar)rDr-r-r.start_capturing_audiozsz*HeyGenInputTransport.start_capturing_audio audio_framecs*t|j|j|jd}||IdHdS)z'Handle received participant audio data.)audio sample_rate num_channelsN)rrLrMrNpush_audio_frame)r)rKr9r-r-r.rHsz/HeyGenInputTransport._on_participant_audio_data)__name__ __module__ __qualname____doc__rrr$rr/r6rr:r r?r rCstrrJrrH __classcell__r-r-r+r.r 1s    r cseZdZdZdedeffdd Zdeffdd Zfd d Z d e ffd d Z d e ffdd Z d effdd Zejfd edeffdd Zd edeffdd Zd edefddZZS)HeyGenOutputTransportzOutput transport for sending audio and events to HeyGen conversations. Handles outgoing audio streams to participants and manages the custom audio track expected by the HeyGen platform. r!r"c s0tj|fi|||_||_d|_d|_dS)zInitialize the HeyGen output transport. Args: client: The HeyGen transport client instance. params: Transport configuration parameters. **kwargs: Additional arguments passed to parent class. FN)r#r$r%r&r' _event_idr(r+r-r.r$s   zHeyGenOutputTransport.__init__r/cr0)zoSetup the output transport. Args: setup: The frame processor setup configuration. Nr1r2r+r-r.r/r3zHeyGenOutputTransport.setupcr4)z#Cleanup output transport resources.Nr5r7r+r-r.r6r8zHeyGenOutputTransport.cleanupr9csXt|IdH|jrdSd|_|j||jIdH||IdH|jdS)z|Start the output transport. Args: frame: The start frame containing initialization parameters. NT)r#r:r'r%audio_chunk_sizer;transport_readyr<r+r-r.r:szHeyGenOutputTransport.startcr=)zqStop the output transport. Args: frame: The end frame signaling transport shutdown. Nr>r<r+r-r.r?r@zHeyGenOutputTransport.stopcrA)zzCancel the output transport. Args: frame: The cancel frame signaling immediate cancellation. NrBr<r+r-r.rCr@zHeyGenOutputTransport.cancel directioncst|tjkr.t|tr|jdurtdt|j|_nt|t r.|j |jIdHd|_t ||IdHdS)zPush a frame to the next processor in the pipeline. Args: frame: The frame to push. direction: The direction to push the frame. Nz"self._event_id is already defined!)r DOWNSTREAM isinstancerrWrwarningrTidr r%agent_speak_endr# push_framer)r9rZr+r-r.r`s    z HeyGenOutputTransport.push_framecst||IdHt|tr#|j|jIdH|||IdHt|tr;|j IdH|||IdHdSt|t rS|j IdH|||IdHdSdS)aeProcess frames and handle interruptions. Handles various types of frames including interruption events and user speaking states. Updates the HeyGen client state based on the received frames. Args: frame: The frame to process direction: The direction of frame flow in the pipeline Note: Special handling is implemented for: - InterruptionFrame: Triggers interruption of current speech - UserStartedSpeakingFrame: Initiates agent listening mode - UserStoppedSpeakingFrame: Stops agent listening mode N) r# process_framer\rr% interruptrWr`rstart_agent_listeningrstop_agent_listeningrar+r-r.rbs   z#HeyGenOutputTransport.process_framereturncs"|jt|j|jIdHdS)zrWrite an audio frame to the HeyGen transport. Args: frame: The audio frame to write. NT)r% agent_speakbytesrLrWr<r-r-r.write_audio_frame sz'HeyGenOutputTransport.write_audio_frame)rPrQrRrSrrr$rr/r6rr:r r?r rCrr[r r`rbrboolrirUr-r-r+r.rVs   rVc@s*eZdZUdZdZeed<dZeed<dS) HeyGenParamszConfiguration parameters for the HeyGen transport. Parameters: audio_in_enabled: Whether to enable audio input from participants. audio_out_enabled: Whether to enable audio output to participants. TrEaudio_out_enabledN)rPrQrRrSrErj__annotations__rlr-r-r-r.rks  rkcseZdZdZeddddfdejdededeedeedee e e fd ee ffd d Z d d ZdefddZdefddZdefddZdefddZdefddZdefddZZS)HeyGenTransportaTransport implementation for HeyGen video calls. When used, the Pipecat bot joins the same virtual room as the HeyGen Avatar and the user. This is achieved by using `HeyGenTransport`, which initiates the conversation via `HeyGenApi` and obtains a room URL that all participants connect to. Event handlers available: - on_client_connected(transport, participant): Participant connected to the session - on_client_disconnected(transport, participant): Participant disconnected from the session Example:: @transport.event_handler("on_client_connected") async def on_client_connected(transport, participant): ... Nsessionapi_keyr" input_name output_namesession_request service_typec sptj||d||_t|||||t|j|j|jdd|_d|_ d|_ d|_ | d| d| ddS)aInitialize the HeyGen transport. Sets up a new HeyGen transport instance with the specified configuration for handling video calls between the Pipecat bot and HeyGen Avatar. Args: session: aiohttp session for making async HTTP requests api_key: HeyGen API key for authentication params: HeyGen-specific configuration parameters (default: HeyGenParams()) input_name: Optional custom name for the input transport output_name: Optional custom name for the output transport session_request: Configuration for the HeyGen session service_type: Service type for the avatar session Note: The transport will automatically join the same virtual room as the HeyGen Avatar and user through the HeyGenClient, which handles session initialization via HeyGenApi. )rqrr) on_connectedon_participant_connectedon_participant_disconnected)rpror"rsrt callbacksNruon_client_connectedon_client_disconnected) r#r$r&rr _on_connected_on_participant_connected_on_participant_disconnectedr%_input_output_HeyGen_participant_id_register_event_handler)r)rorpr"rqrrrsrtr+r-r.r$3s(   zHeyGenTransport.__init__cs4|dIdH|jr|jtIdHdSdS)z%Handle bot connected to LiveKit room.ruN)_call_event_handlerr~r`rr7r-r-r.r{gs zHeyGenTransport._on_connectedrDcs4td|d|dkr||IdHdSdS)NHeyGen participant z disconnectedheygen)rdebug_on_client_disconnectedrIr-r-r.r}ms z,HeyGenTransport._on_participant_disconnectedcsPtd|d|dkr$||IdH|jr&|j|IdHdSdSdS)Nrz connectedr)rr_on_client_connectedr~rJrIr-r-r.r|rsz)HeyGenTransport._on_participant_connectedrfcC|js t|j|jd|_|jS)zGet the input transport for receiving media and events. Returns: The HeyGen input transport instance. r!r")r~r r%r&r7r-r-r.inputyzHeyGenTransport.inputcCr)zGet the output transport for sending media and events. Returns: The HeyGen output transport instance. r)rrVr%r&r7r-r-r.outputrzHeyGenTransport.output participantcs6|d|IdH|jr|jtIdHdSdS)zHandle client connected events.ryN)rr~r`r r)rr-r-r.rs z$HeyGenTransport._on_client_connectedcs|d|IdHdS)z"Handle client disconnected events.rzN)rrr-r-r.rsz'HeyGenTransport._on_client_disconnected)rPrQrRrSrkaiohttp ClientSessionrTrrrrrr$r{r}r|rrrrrrrUr-r-r+r.rn s:4  rn).rStypingrrrrlogururpipecat.frames.framesrrrr r r r r rrrrrr"pipecat.processors.frame_processorrrr.pipecat.services.heygen.api_interactive_avatarr&pipecat.services.heygen.api_liveavatarrpipecat.services.heygen.clientrrrpipecat.transports.base_inputrpipecat.transports.base_outputr!pipecat.transports.base_transportrrr rVrkrnr-r-r-r.s"  @    _