o *iE@sddlZddlZddlZddlmZddlmZddlmZddl Z ddl m Z ddl mZmZmZeeZdZded ed dfd d ZGd ddeeZeGdddZeGdddZeGdddZeGdddZGdddZdS)N) dataclass)Enum)Any)Audio) AudioChunk AudioURLChunk AudioURLType num_samplesmult_ofreturncCs$||dksJd|d|dS)Nrz num_samples=z must be a multiple of mult_of=)r r r r c/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/audio.py_check_mult_ofs$rc@seZdZdZdZdZdS)TranscriptionFormatzTranscription format. Should be set by the tokenizer for correct encoding. Attributes: - INSTRUCT: The instruct format. - STREAMING: The streaming format. instruct streamingN)__name__ __module__ __qualname____doc__INSTRUCT STREAMINGr r r rrs rc@s4eZdZUdZeed<eed<eed<d ddZdS) AudioSpectrogramConfiga^Configuration for generating an audio spectrogram. Attributes: num_mel_bins: Number of mel bins, typically 80 or 128. hop_length: Length of the overlapping windows for the STFT used to obtain the Mel Frequency coefficients, typically 160. window_size: Window size of the Fourier transform, typically 400. num_mel_bins hop_length window_sizer NcCs@|jdks J|j|jdksJ|j|jdksJ|jdS)Nr)rrrselfr r r __post_init__;sz$AudioSpectrogramConfig.__post_init__r N)rrrrint__annotations__rr r r rr)s  rc@sdeZdZUdZeed<eed<eed<dZedBed<e j Z e ed<dZ edBed<dZ edBed <dZedBed <dZedBed <d&d dZed efddZded efddZed efddZd'dedBd efddZded efddZed efddZed efddZed efddZed efd d!Zd'dedBd efd"d#Zed efd$d%ZdS)( AudioConfiga`Configuration for audio processing. Attributes: sampling_rate: Sampling rate of the audio. frame_rate: Number of frames per second accepted by the tokenizer model. encoding_config: Configuration for audio spectrogram. chunk_length_s: Whether to pad an audio into multiples of chunk_length_s seconds (optional). sampling_rate frame_rateencoding_configNchunk_length_stranscription_formattranscription_delay_msstreaming_look_ahead_msstreaming_look_back_msstreaming_n_left_pad_tokensr cCs|jdks J|j|jdksJ|j|jdur4|jdks#J|j|jdks4Jd|jd|j|jso|jdusEJd|jd|jdusSJd|jd|jdusaJd|jd|jdusoJd|jd|jr|jdusJd|jd |jdusJd|jd |jdusJd|jd |jdusJd|jd d |j}|jdksJd |j|dksJd|jd ||jdusJd |jddSdS)Nrz7chunk_length_s and sampling_rate must both be > 0, got z and zself.transcription_delay_ms=z must be None.zself.streaming_look_ahead_ms=zself.streaming_look_back_ms=!self.streaming_n_left_pad_tokens= must be set.@@z*{self.transcription_delay_ms=} must be > 0z) must be a multiple of frame_duration_ms=zself.chunk_length_s=z cannot be set in streaming.) r%r$r' chunk_frames is_streamingr)r*r+r,)rframe_duration_msr r rr_s2    zAudioConfig.__post_init__cCs |jtjkSN)r(rrrr r rr1~s zAudioConfig.is_streaming audio_lencCsD||jjdkrt||jjd}n||jj}t||jSNr)r&rmathceilaudio_length_per_tok)rr4r r rnum_audio_tokenss zAudioConfig.num_audio_tokenscCstdt|S)Nz4Use get_num_delay_tokens instead of num_delay_tokens)warningswarnDeprecationWarningget_num_delay_tokensrr r rnum_delay_tokenss zAudioConfig.num_delay_tokenscCsN|js Jd|jd|dur|j}|dusJd|d|||S)Nz5Can't call get_num_delay_tokens if self.is_streaming=.z:Can't call get_num_delay_tokens if transcription_delay_ms=)r1r)r: delay_lenrr)r r rr>s z AudioConfig.get_num_delay_tokenscCst|d|jSNr/)r!r$rBr r rrAszAudioConfig.delay_lencCs d|jSrC)r%rr r rr2s zAudioConfig.frame_duration_mscCs,|jdusJd|jdt|j|jS)z)Calculate the number of frames per chunk.Nz/Can't call chunk_frames if self.chunk_length_s=r@)r'r!r$rr r rr0szAudioConfig.chunk_framescCst|j|jSr3)r!r$r%rr r rraw_audio_length_per_toksz$AudioConfig.raw_audio_length_per_tokcCst|j}||jj}t|S)z(Calculate the length of audio per token.)floatrDr&rr!)rdownsample_factorr r rr9s  z AudioConfig.audio_length_per_tokcCs*|js Jd|jd||dtS)Nz3Can't call n_right_pad_tokens if self.is_streaming=r@r6)r1r>OFFLINE_STREAMING_BUFFER_TOKENSrBr r rn_right_pad_tokensszAudioConfig.n_right_pad_tokenscCs:|js Jd|jd|jdusJd|jd|jS)Nz2Can't call n_left_pad_tokens if self.is_streaming=r@r-r.)r1r,rr r rn_left_pad_tokensszAudioConfig.n_left_pad_tokensr r3)rrrrr!r"rErr'rrr(r)r*r+r,rpropertyboolr1r:r?r>rAr2r0rDr9rHrIr r r rr#As<    r#c@s&eZdZUdZeeed<eed<dS) AudioEncodingzEncapsulates the tokens and audio data for an audio chunk. Attributes: tokens: Text tokens corresponding to this audio chunk. audio: Original audio waveform data. tokensaudioN)rrrrlistr!r"rr r r rrLs   rLc@s6eZdZUdZedBed<edBed<edBed<dS)SpecialAudioIDsaSpecial text tokens corresponding to audio token sequence. Attributes: audio: Token representing audio. begin_audio: Token representing the beginning of audio. streaming_pad: Token representing streaming pad of audio. Only relevant for steaming models. NrN begin_audio streaming_pad)rrrrr!r"r r r rrPs   rPc @sheZdZdZdededdfddZ d+dejd e d e dBd e dejf d d Z d+d e dBde eeffddZd+de d e dBde e e ffddZde d e de fddZd+d e dBdee fddZde dee fddZd+ded e dBdefddZdedefdd Zdedefd!d"ZdeeBdefd#d$Zede fd%d&Zede fd'd(Zede fd)d*ZdS), AudioEncodera Encodes audio chunks into a format suitable for further processing. Attributes: audio_config: Configuration for audio processing. encoding_config: Configuration for audio spectrogram. special_ids: Special tokens for audio encoding. audio_config special_idsr NcCs||_|j|_||_dSr3)rTr&rU)rrTrUr r r__init__s zAudioEncoder.__init__ audio_arrayr$r)kwargscKs|jjr||jd|}t|d||jdf}|S|jjr5||jd|\}}t|||f}|St|j t rS|jd|j j krSt|d|j j |jdf}|S)a9Pad the audio array to the desired length. Args: audio_array: Audio data as a numpy array. sampling_rate: Sampling rate of the audio. transcription_delay_ms (optional): Delay in milliseconds for transcription. Returns: Padded audio array. r) rTr'next_multiple_of_chunk_framesshapenppadr1_get_streaming_pad isinstancer&rr)rrWr$r)rXrZleft_pad right_padr r rr]s  zAudioEncoder.padcCsT|d|\}}ttj|tjd|jjdd}ttj|tjd|jjdd}||fS)zGets left and right padding for realtime audio models. Args: transcription_delay_ms (optional): Delay in milliseconds for transcription. Returns: Tuple of left and right padding for realtime audio models. r)dtypewav)rWr$format)r^rr\zerosfloat32rTr$)rr)r`raleft_pad_audioright_pad_audior r rget_padding_audios zAudioEncoder.get_padding_audior c Csl|jj}t||||}|j|}t||}t||||7}|jj}t||}t||||fSr3)rTrDr!rHrrI) rr r)r ra_extra_right_pad_tokens_extra_right_pad_samples_extra_left_pad_tokensr`r r rr^,s     zAudioEncoder._get_streaming_padaudio_array_lencCs^||jjksJd|d|jj|jjdus"Jd|jjdt||jj|jjS)zCalculate the next multiple of chunk frames. Args: audio_array_len: Length of the audio array. sampling_rate: Sampling rate of the audio. Returns: The next multiple of chunk frames. zExpected sampling_rate=z' to be self.audio_config.sampling_rate=NzMCan't call next_multiple_of_chunk_frames if self.audio_config.chunk_length_s=r@)rTr$r'r7r8r0)rrmr$r r rrZAs z*AudioEncoder.next_multiple_of_chunk_framescCsTt|jjtsJd|jj|jjdusJ|jj|j|}|jg|}|S)z8Encode the streaming tokens given a transcription delay.zQAudio encoder must be spectrogram encoder, got self.audio_config.encoding_config=N)r_rTr&rr)rIr>rR)rr)stream_pad_prefix_lenrMr r rencode_streaming_tokensTs   z$AudioEncoder.encode_streaming_tokens signal_lengthcCs^||jjdkrt||jjd}n||jj}t||jj}|jg|jg|}|Sr5)r&rr7r8rTr9begin_audio_token audio_token)rrpr:rMr r r_encode_audio_tokenscs  z!AudioEncoder._encode_audio_tokensrNcCs\||jj||j|jj||_|jjtjkr||}n | |jj d}t ||dS)z4Encode an audio optionally with transcription delay.r)rMrN) resamplerTr$r]rWr(rrrorsr[rL)rrNr)rMr r r encode_audioos zAudioEncoder.encode_audiocontentcCst|j}||Sr3)rfrom_raw_audio input_audioru)rrvrNr r r_encode_audio_chunk~s  z AudioEncoder._encode_audio_chunkcCsT|}|tjtjhvrt|j}n|tjkrt|j}nt|j}| |Sr3) get_url_typerfilefile_urir from_fileurlfrom_url from_base64ru)rrvurl_typerNr r r_encode_audio_url_chunks   z$AudioEncoder._encode_audio_url_chunkcCs:t|tr ||St|tr||Stdt|)zCall the encoder on an audio chunk or URL chunk. Args: content: Audio or URL chunk to encode. Returns: Encoded audio data and tokens. zUnsupported content type: )r_rrrry ValueErrortype)rrvr r r__call__s   zAudioEncoder.__call__cC(|jjdusJd|jjd|jjS)zGet the audio token.Nzself.special_ids.audio=r.)rUrNrr r rrr zAudioEncoder.audio_tokencCr)zGet the begin audio token.Nzself.special_ids.begin_audio=r.)rUrQrr r rrqrzAudioEncoder.begin_audio_tokencCr)zGet the streaming pad token.Nzself.special_ids.streaming_pad=r.)rUrRrr r rrRrzAudioEncoder.streaming_padr3) rrrrr#rPrVr\ndarrayr!rErr]tuplerrir^rZrOrorsrLrurryrrrrJrrrqrRr r r rrSs<  $$  rS)loggingr7r; dataclassesrenumrtypingrnumpyr\mistral_common.audior&mistral_common.protocol.instruct.chunkrrr getLoggerrloggerrGr!rstrrrr#rLrPrSr r r rs.