o it@sZdZddlmZddlmZmZmZmZmZddl Z ddl m Z ddl m Z ddlmZddlmZmZmZmZmZmZmZmZmZmZmZdd lmZmZmZdd l m!Z!dd l"m#Z#dd l$m%Z%m&Z&dd l'm(Z(m)Z)m*Z*ddl+m,Z,GdddeZ-GdddeZ.Gddde)Z/GdddZ0Gddde!Z1Gddde#Z2Gddde%Z3dS)zLemonSlice transport for Pipecat. This module adds LemonSlice avatars to Daily rooms, enabling real-time voice conversations with synchronized avatars. )partial)Any AwaitableCallableMappingOptionalN) AudioData)logger) BaseModel) BotStartedSpeakingFrameBotStoppedSpeakingFrame CancelFrameEndFrameFrameInputAudioRawFrameInterruptionFrameOutputAudioRawFrameOutputTransportMessageFrame!OutputTransportMessageUrgentFrame StartFrame)FrameDirectionFrameProcessorFrameProcessorSetup)BaseInputTransport)BaseOutputTransport) BaseTransportTransportParams)DailyCallbacks DailyParamsDailyTransportClient) LemonSliceApic@seZdZUdZdZeeed<dZeeed<dZ eeed<dZ ee ed<dZ eeed<dZ eeed<dZeeed <dS) LemonSliceNewSessionRequestaRequest model for creating a new LemonSlice session. Parameters: agent_image_url: URL to an agent image. Provide either agent_id or agent_image_url. agent_id: ID of a LemonSlice agent. Provide either agent_id or agent_image_url. agent_prompt: A high-level system prompt that subtly influences the avatar's movements, expressions, and emotional demeanor. idle_timeout: Idle timeout in seconds. daily_room_url: Daily room URL to use for the session. daily_token: Daily token for authenticating with the room. lemonslice_properties: Additional properties to pass to the session. Nagent_image_urlagent_id agent_prompt idle_timeoutdaily_room_url daily_tokenlemonslice_properties)__name__ __module__ __qualname____doc__r"rstr__annotations__r#r$r%intr&r'r(dictr1r1[/home/ubuntu/.local/lib/python3.10/site-packages/pipecat/transports/lemonslice/transport.pyr!.s  r!c@sPeZdZUdZeeeefgedfe d<eeeefegedfe d<dS)LemonSliceCallbackszCallback handlers for LemonSlice events. Parameters: on_participant_joined: Called when a participant joins the conversation. on_participant_left: Called when a participant leaves the conversation. Non_participant_joinedon_participant_left) r)r*r+r,rrr-rrr.r1r1r1r2r3Es $r3c@s6eZdZUdZdZeed<dZeed<dZeed<dS)LemonSliceParamsa-Configuration parameters for the LemonSlice transport. Parameters: audio_in_enabled: Whether to enable audio input from participants. audio_out_enabled: Whether to enable audio output to participants. microphone_out_enabled: Whether to enable microphone output track. Taudio_in_enabledaudio_out_enabledFmicrophone_out_enabledN) r)r*r+r,r7boolr.r8r9r1r1r1r2r6Qs   r6c@speZdZdZedddededededeed e j d dfd d Z d efd dZ de fddZddZddZddZddZd efddZdefddZdd Z ! " #dFd$ed%ed&ed'ed(ef d)d*Z + , -dGd$ed%ed.ed/ed0ef d1d2ZdeeBfd3d4Zed efd5d6Zed efd7d8Z dHd9d:Z!dHd;d<Z"dHd=d>Z#dId?d@Z$de%d e&fdAdBZ'dCefdDdEZ(dS)JLemonSliceTransportClientaTransport client that integrates Pipecat with the LemonSlice platform. A transport client that integrates a Pipecat Bot with the LemonSlice platform by managing conversation sessions using the LemonSlice API. This client uses `LemonSliceApi` to interact with the LemonSlice backend. LemonSlice either provides a room URL where the avatar is already present, or adds the LemonSlice avatar to a Daily room the user supplies. N)paramssession_requestbot_namer< callbacksapi_keyr=sessionreturncCs@||_t|||_|p t|_d|_d|_d|_||_||_ dS)aInitialize the LemonSlice transport client. Args: bot_name: The name of the Pipecat bot instance. params: Optional parameters for LemonSlice operation. callbacks: Callback handlers for LemonSlice-related events. api_key: API key for authenticating with LemonSlice API. session_request: Optional session creation parameters. If not provided, a default agent will be used. session: The aiohttp session for making async HTTP requests. N) _bot_namer _apir!_session_request _session_id _control_url_daily_transport_client _callbacks_params)selfr>r<r?r@r=rAr1r1r2__init__js   z"LemonSliceTransportClient.__init__c sZ|jj|jj|jj|jj|jj|jj|jj|jj dIdH}|d|_ |d|_ |dS)z4Initialize the conversation and return the room URL.)r"r#r$r%r&r' propertiesN session_id control_urlroom_url) rDcreate_sessionrEr"r#r$r%r&r'r(rFrG)rKresponser1r1r2 _initializes z%LemonSliceTransportClient._initializesetupc s@|jdurtd|jdSz|IdH}td!idt|jdd|jd|jdt|jddt|jddt|jdd t|jd d t|jd d t|jd d t|jd d t|jd dt|jddt|jddt|jddt|jddt|jddt|jddt|jddt|jdd|j j d|j j dt|jddt|jddt|jddt|jddt|jddt|jddt|jd}t |d|j |j|d|_|j|IdHWdSty}z#td ||jr|jr|j|j|jIdHd|_d|_d}~ww)"zSetup the client and initialize the conversation. Args: setup: The frame processor setup configuration. NzSession ID already defined: on_active_speaker_changed on_joinedon_lefton_before_leaveon_erroron_app_messageon_call_state_updatedon_client_connectedon_client_disconnectedon_dialin_connectedon_dialin_readyon_dialin_stoppedon_dialin_erroron_dialin_warningon_dialout_answeredon_dialout_connectedon_dialout_stoppedon_dialout_erroron_dialout_warningr4r5on_participant_updatedon_transcription_messageon_recording_startedon_recording_stoppedon_recording_erroron_transcription_stoppedon_transcription_errorLemonSlicePipecatz+Failed to setup LemonSliceTransportClient: r1)rFr debugrSrr_on_handle_callback _on_joined_on_leftrIr4r5rrCrJrHrT ExceptionerrorrGrD end_session)rKrTrPdaily_callbackser1r1r2rTs                      "$zLemonSliceTransportClient.setupc s\z|jr|jIdHWdSWdSty-}ztd|WYd}~dSd}~ww)zCleanup client resources.NzException during cleanup: )rHcleanuprtr ru)rKrxr1r1r2rysz!LemonSliceTransportClient.cleanupctddS)zHandle joined event.z!LemonSliceTransportClient joined!Nr rp)rKdatar1r1r2rrz$LemonSliceTransportClient._on_joinedcrz)zHandle left event.zLemonSliceTransportClient left!Nr{rKr1r1r2rsr}z"LemonSliceTransportClient._on_leftcs"td|d|d|dS)zHandle generic callback events.z [Callback] z called with args=z , kwargs=N)r trace)rK event_nameargskwargsr1r1r2rqs z-LemonSliceTransportClient._on_handle_callbackcsdS)zzGet the name of the LemonSlice participant. Returns: The name of the LemonSlice participant. LemonSlicer1r~r1r1r2 get_bot_namesz&LemonSliceTransportClient.get_bot_nameframecs(|j|IdH|jIdHdS)zStart the client and join the room. Args: frame: The start frame containing initialization parameters. N)rHstartjoinrKrr1r1r2rzLemonSliceTransportClient.startcsF|jIdH|jr|jr|j|j|jIdHd|_d|_dS)z)Stop the client and end the conversation.N)rHleaverFrGrDrvr~r1r1r2stops   zLemonSliceTransportClient.stopcameraRGBparticipant_idcallback framerate video_source color_formatc |j|||||IdHdS)aoCapture video from a participant. Args: participant_id: ID of the participant to capture video from. callback: Callback function to handle video frames. framerate: Desired framerate for video capture. video_source: Video source to capture from. color_format: Color format for video frames. N)rHcapture_participant_video)rKrrrrrr1r1r2r z3LemonSliceTransportClient.capture_participant_video microphone> audio_source sample_ratecallback_interval_mscr)aCapture audio from a participant. Args: participant_id: ID of the participant to capture audio from. callback: Callback function to handle audio data. audio_source: Audio source to capture from. sample_rate: Desired sample rate for audio capture. callback_interval_ms: Interval between audio callbacks in milliseconds. N)rHcapture_participant_audio)rKrrrrrr1r1r2rrz3LemonSliceTransportClient.capture_participant_audioc|j|IdHdS)eSend a message to participants. Args: frame: The message frame to send. N)rH send_messagerr1r1r2r)sz&LemonSliceTransportClient.send_messagecC|jjS)z`Get the output sample rate. Returns: The output sample rate in Hz. )rHout_sample_rater~r1r1r2r3z)LemonSliceTransportClient.out_sample_ratecCr)z^Get the input sample rate. Returns: The input sample rate in Hz. )rHin_sample_rater~r1r1r2r<rz(LemonSliceTransportClient.in_sample_ratec2tdtd|jdd}||IdHdS)z4Send an interrupt message to the LemonSlice session.zSending interrupt message interrupteventrNmessageN)r rprrFrrKtransport_framer1r1r2send_interrupt_messageE z0LemonSliceTransportClient.send_interrupt_messagecr)z:Send a response_started message to the LemonSlice session.z Sending response_started messageresponse_startedrrNr rrrFrrr1r1r2send_response_started_messagePrz7LemonSliceTransportClient.send_response_started_messagecr)z;Send a response_finished message to the LemonSlice session.z!Sending response_finished messageresponse_finishedrrNrrr1r1r2send_response_finished_message[rz8LemonSliceTransportClient.send_response_finished_messagecs&|jsdS|jj||dIdHdS)Update subscription settings for participants. Args: participant_settings: Per-participant subscription settings. profile_settings: Global subscription profile settings. Nparticipant_settingsprofile_settings)rHupdate_subscriptionsrKrrr1r1r2rfs z.LemonSliceTransportClient.update_subscriptionscs|jsdS|j|IdHS)zWrite an audio frame to the transport. Args: frame: The audio frame to write. Returns: True if the audio frame was written successfully, False otherwise. FN)rHwrite_audio_framerr1r1r2rts z+LemonSliceTransportClient.write_audio_frame destinationcs"|jsdS|j|IdHdS)zRegister an audio destination for output. Args: destination: The destination identifier to register. N)rHregister_audio_destinationrKrr1r1r2rsz4LemonSliceTransportClient.register_audio_destination)rrr)rrr)rBNNN))r)r*r+r,r6r-r3rr!aiohttp ClientSessionrLrSrrTryrrrsrqrrrrrr/rrrrrpropertyrrrrrrrr:rrr1r1r1r2r;_s ;        r;cseZdZdZdedeffdd Zdeffdd Zfd d Z d e ffd d Z d e ffdd Z d effdd ZddZdededefddZZS)LemonSliceInputTransportzInput transport for receiving audio and events from LemonSlice. Handles incoming audio streams from participants and manages audio capture from the Daily room connected to LemonSlice. clientr<c s*tj|fi|||_||_d|_dS)zInitialize the LemonSlice input transport. Args: client: The LemonSlice transport client instance. params: Transport configuration parameters. **kwargs: Additional arguments passed to parent class. FN)superrL_clientrJ _initializedrKrr<r __class__r1r2rLs  z!LemonSliceInputTransport.__init__rTc*t|IdH|j|IdHdS)znSetup the input transport. Args: setup: The frame processor setup configuration. NrrTrrKrTrr1r2rTzLemonSliceInputTransport.setupc&tIdH|jIdHdS)z"Cleanup input transport resources.Nrryrr~rr1r2ryz LemonSliceInputTransport.cleanuprcsJt|IdH|jrdSd|_|j|IdH||IdHdS)z{Start the input transport. Args: frame: The start frame containing initialization parameters. NT)rrrrset_transport_readyrrr1r2rszLemonSliceInputTransport.startc(t|IdH|jIdHdS)zpStop the input transport. Args: frame: The end frame signaling transport shutdown. Nrrrrrr1r2rrzLemonSliceInputTransport.stopc(t|IdH|jIdHdS)zyCancel the input transport. Args: frame: The cancel frame signaling immediate cancellation. Nrcancelrrrrr1r2rrzLemonSliceInputTransport.cancelcsH|jjr"td|d|jj|d|j|jjdIdHdSdS)zStart capturing audio from a participant. Args: participant: The participant to capture audio from. z@LemonSliceTransportClient start capturing audio for participant id)rrrN)rJr7r rprr_on_participant_audio_datarrK participantr1r1r2start_capturing_audios z.LemonSliceInputTransport.start_capturing_audioraudiorcs0t|j|j|jd}||_||IdHdS)aHandle received participant audio data. Args: participant_id: ID of the participant who sent the audio. audio: The audio data from the participant. audio_source: The source of the audio (e.g., microphone). )rr num_channelsN)r audio_framesrrtransport_sourcepush_audio_frame)rKrrrrr1r1r2rs z3LemonSliceInputTransport._on_participant_audio_data)r)r*r+r,r;rrLrrTryrrrrr rrr-rr __classcell__r1r1rr2rs(   rcseZdZdZdedeffdd Zdeffdd Zfd d Z d e ffd d Z d e ffdd Z d effdd Zd eeBfddZejfd edeffdd Zd edeffdd ZddZddZddZd edefd d!Zd"efd#d$ZZ S)%LemonSliceOutputTransportzOutput transport for sending audio and events to LemonSlice. Handles outgoing audio streams to participants and manages the custom audio track expected by the LemonSlice platform. rr<c s0tj|fi|||_||_d|_d|_dS)zInitialize the LemonSlice output transport. Args: client: The LemonSlice transport client instance. params: Transport configuration parameters. **kwargs: Additional arguments passed to parent class. FstreamN)rrLrrJr_transport_destinationrrr1r2rLs   z"LemonSliceOutputTransport.__init__rTcr)zoSetup the output transport. Args: setup: The frame processor setup configuration. Nrrrr1r2rTrzLemonSliceOutputTransport.setupcr)z#Cleanup output transport resources.Nrr~rr1r2ry!rz!LemonSliceOutputTransport.cleanuprcsdt|IdH|jrdSd|_|j|IdH|jr(|j|jIdH||IdHdS)z|Start the output transport. Args: frame: The start frame containing initialization parameters. NT)rrrrrrrrrr1r2r&szLemonSliceOutputTransport.startcr)zqStop the output transport. Args: frame: The end frame signaling transport shutdown. Nrrrr1r2r:rzLemonSliceOutputTransport.stopcr)zzCancel the output transport. Args: frame: The cancel frame signaling immediate cancellation. Nrrrr1r2rCrz LemonSliceOutputTransport.cancelcs(td||j|IdHdS)rz$LemonSliceTransport sending message N)r rrrrr1r1r2rLsz&LemonSliceOutputTransport.send_message directioncsT|tjkrt|tr|IdHt|tr|IdHt||IdHdS)zPush a frame to the next processor in the pipeline. Args: frame: The frame to push. direction: The direction to push the frame. N) r DOWNSTREAM isinstancer _handle_response_startedr _handle_response_finishedr push_framerKrrrr1r2rWs  z$LemonSliceOutputTransport.push_framecs6t||IdHt|tr|IdHdSdS)zProcess frames and handle interruptions. Args: frame: The frame to process. direction: The direction of frame flow in the pipeline. N)r process_framerr_handle_interruptionsrrr1r2rgs  z'LemonSliceOutputTransport.process_framec|jIdHdS)z8Handle interruption events by sending interrupt message.N)rrr~r1r1r2rrz/LemonSliceOutputTransport._handle_interruptionscr)zGHandle bot started speaking events by sending response_started message.N)rrr~r1r1r2rvrz2LemonSliceOutputTransport._handle_response_startedcr)zHHandle tts response stopped events by sending response_finished message.N)rrr~r1r1r2rzrz3LemonSliceOutputTransport._handle_response_finishedrBcs|j|_|j|IdHS)zWrite an audio frame to the LemonSlice transport. Args: frame: The audio frame to write. Returns: True if the audio frame was written successfully, False otherwise. N)rtransport_destinationrrrr1r1r2r~s z+LemonSliceOutputTransport.write_audio_framercr)zwRegister an audio destination. Args: destination: The destination identifier to register. N)rrrr1r1r2rsz4LemonSliceOutputTransport.register_audio_destination)!r)r*r+r,r;rrLrrTryrrrrr rrrrrrrrrrrrrr:rr-rrr1r1rr2rs,       rcseZdZdZdeddfdedejdedee dedeed eeffd d Z d d Z ddZ dddZ defddZdefddZdefddZdefddZZS)LemonSliceTransportaTransport implementation to add a LemonSlice avatar to Daily calls. When used, the Pipecat bot joins the same virtual room as the LemonSlice Avatar and the user. This is achieved by using `LemonSliceTransportClient`, which initiates the conversation via `LemonSliceApi` and obtains a room URL that all participants connect to. Event handlers available: - on_client_connected(transport, participant): Participant connected to the session - on_client_disconnected(transport, participant): Participant disconnected from the session Example:: @transport.event_handler("on_client_connected") async def on_client_connected(transport, participant): ... Nr>rAr@r=r< input_name output_namec sftj||d||_t|j|jd}t||||||d|_d|_d|_ d|_ | d| ddS)a;Initialize the LemonSlice transport. Args: bot_name: The name of the Pipecat bot. session: aiohttp session used for async HTTP requests. api_key: LemonSlice API key for authentication. session_request: Optional session creation parameters. If not provided, a default agent will be used. params: Optional LemonSlice-specific configuration parameters. input_name: Optional name for the input transport. output_name: Optional name for the output transport. )rr)r4r5)r>r?r@r=rAr<Nr\r]) rrLrJr3_on_participant_joined_on_participant_leftr;r_input_output_lemonslice_participant_id_register_event_handler) rKr>rAr@r=r<rrr?rr1r2rLs& zLemonSliceTransport.__init__csB|jIdH}|didd|kr||IdHdSdS)zHandle participant left events.NinfouserName)rrget_on_client_disconnected)rKrreason ls_bot_namer1r1r2rs z(LemonSliceTransport._on_participant_leftcs|jIdH}|didd|kr|d|_dS||IdH|jrAtd|jd|j|jdd d iiid IdH|jrO|j |IdHdSdS) z!Handle participant joined events.Nrrrrz Ignoring z 's microphonemediar unsubscribed)r) rrrr_on_client_connectedr rprrr)rKrrr1r1r2rs  z*LemonSliceTransport._on_participant_joinedcs|jj||dIdHdS)rrN)rrrr1r1r2rs z(LemonSliceTransport.update_subscriptionsrBcC|js t|j|jd|_|jS)zGet the input transport for receiving media and events. Returns: The LemonSlice input transport instance. rr<)rrrrJr~r1r1r2inputzLemonSliceTransport.inputcCr)zGet the output transport for sending media and events. Returns: The LemonSlice output transport instance. r)rrrrJr~r1r1r2outputrzLemonSliceTransport.outputrc|d|IdHdS)zHandle client connected events.r\N_call_event_handlerrr1r1r2rz(LemonSliceTransport._on_client_connectedcr )z"Handle client disconnected events.r]Nr rr1r1r2rr z+LemonSliceTransport._on_client_disconnectedr)r)r*r+r,r6r-rrrr!rLrrrrrrrrrrr1r1rr2rs8.    r)4r, functoolsrtypingrrrrrr daily.dailyrlogurur pydanticr pipecat.frames.framesr r r rrrrrrrr"pipecat.processors.frame_processorrrrpipecat.transports.base_inputrpipecat.transports.base_outputr!pipecat.transports.base_transportrr"pipecat.transports.daily.transportrrr!pipecat.transports.lemonslice.apir r!r3r6r;rrrr1r1r1r2s0    4    1m