o c۷i>T@sddlZddlZddlZddlmZddlmZddlmZddl Z ddl m Z ddl mZmZmZeeZdZded ed dfd d ZGd ddeeZeGdddZeGdddZeGdddZeGdddZGdddZdS)N) dataclass)Enum)Any)Audio) AudioChunk AudioURLChunk AudioURLType num_samplesmult_ofreturncCs$||dksJd|d|dS)Nrz num_samples=z must be a multiple of mult_of=)r r r r \/home/ubuntu/vllm_env/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/audio.py_check_mult_ofs$rc@seZdZdZdZdZdS)TranscriptionFormatzTranscription format. Should be set by the tokenizer for correct encoding. Attributes: - INSTRUCT: The instruct format. - STREAMING: The streaming format. instruct streamingN)__name__ __module__ __qualname____doc__INSTRUCT STREAMINGr r r rrs rc@s4eZdZUdZeed<eed<eed<d ddZdS) AudioSpectrogramConfiga^Configuration for generating an audio spectrogram. Attributes: num_mel_bins: Number of mel bins, typically 80 or 128. hop_length: Length of the overlapping windows for the STFT used to obtain the Mel Frequency coefficients, typically 160. window_size: Window size of the Fourier transform, typically 400. num_mel_bins hop_length window_sizer NcCs@|jdks J|j|jdksJ|j|jdksJ|jdS)Nr)rrrselfr r r __post_init__;sz$AudioSpectrogramConfig.__post_init__r N)rrrrint__annotations__rr r r rr)s  rc@s|eZdZUdZeed<eed<eed<dZedBed<e j Z e ed<dZ edBed<dZ edBed <dZedBed <dZedBed <dZeeefdBed <d'ddZed efddZded efddZed efddZd(dedBd efddZded efddZed efddZed efddZed efdd Zed efd!d"Zd(dedBd efd#d$Zed efd%d&Z dS)) AudioConfigaConfiguration for audio processing. Attributes: sampling_rate: Sampling rate of the audio. frame_rate: Number of frames per second accepted by the tokenizer model. encoding_config: Configuration for audio spectrogram. chunk_length_s: Whether to pad an audio into multiples of chunk_length_s seconds (optional). voice_num_audio_tokens: Mapping from speaker voice name to number of audio tokens for that speaker's reference audio (optional, only for TTS). sampling_rate frame_rateencoding_configNchunk_length_stranscription_formattranscription_delay_msstreaming_look_ahead_msstreaming_look_back_msstreaming_n_left_pad_tokensvoice_num_audio_tokensr cCs|jdks J|j|jdksJ|j|jdur4|jdks#J|j|jdks4Jd|jd|j|jso|jdusEJd|jd|jdusSJd|jd|jdusaJd|jd|jdusoJd|jd|jr|jdusJd|jd |jdusJd|jd |jdusJd|jd |jdusJd|jd d |j}|jdksJd |j|dksJd|jd ||jdusJd |jddSdS)Nrz7chunk_length_s and sampling_rate must both be > 0, got z and zself.transcription_delay_ms=z must be None.zself.streaming_look_ahead_ms=zself.streaming_look_back_ms=!self.streaming_n_left_pad_tokens= must be set.@@z*{self.transcription_delay_ms=} must be > 0z) must be a multiple of frame_duration_ms=zself.chunk_length_s=z cannot be set in streaming.) r%r$r' chunk_frames is_streamingr)r*r+r,)rframe_duration_msr r rrcs2    zAudioConfig.__post_init__cCs |jtjkSN)r(rrrr r rr2s zAudioConfig.is_streaming audio_lencCsD||jjdkrt||jjd}n||jj}t||jSNr)r&rmathceilaudio_length_per_tok)rr5r r rnum_audio_tokenss zAudioConfig.num_audio_tokenscCstdt|S)Nz4Use get_num_delay_tokens instead of num_delay_tokens)warningswarnDeprecationWarningget_num_delay_tokensrr r rnum_delay_tokenss zAudioConfig.num_delay_tokenscCsN|js Jd|jd|dur|j}|dusJd|d|||S)Nz5Can't call get_num_delay_tokens if self.is_streaming=.z:Can't call get_num_delay_tokens if transcription_delay_ms=)r2r)r; delay_lenrr)r r rr?s z AudioConfig.get_num_delay_tokenscCst|d|jSNr0)r!r$rCr r rrBszAudioConfig.delay_lencCs d|jSrD)r%rr r rr3s zAudioConfig.frame_duration_mscCs,|jdusJd|jdt|j|jS)z)Calculate the number of frames per chunk.Nz/Can't call chunk_frames if self.chunk_length_s=rA)r'r!r$rr r rr1szAudioConfig.chunk_framescCst|j|jSr4)r!r$r%rr r rraw_audio_length_per_toksz$AudioConfig.raw_audio_length_per_tokcCst|j}||jj}t|S)z(Calculate the length of audio per token.)floatrEr&rr!)rdownsample_factorr r rr:s  z AudioConfig.audio_length_per_tokcCs*|js Jd|jd||dtS)Nz3Can't call n_right_pad_tokens if self.is_streaming=rAr7)r2r?OFFLINE_STREAMING_BUFFER_TOKENSrCr r rn_right_pad_tokensszAudioConfig.n_right_pad_tokenscCs:|js Jd|jd|jdusJd|jd|jS)Nz2Can't call n_left_pad_tokens if self.is_streaming=rAr.r/)r2r,rr r rn_left_pad_tokensszAudioConfig.n_left_pad_tokensr r4)!rrrrr!r"rFrr'rrr(r)r*r+r,r-dictstrrpropertyboolr2r;r@r?rBr3r1rEr:rIrJr r r rr#As>    r#c@s*eZdZUdZeeed<edBed<dS) AudioEncodingaEncapsulates the tokens and audio data for an audio chunk. Attributes: tokens: Text tokens corresponding to this audio chunk. audio: Original audio waveform data, or None when using a preset voice (no reference audio to forward to the model). tokensNaudio)rrrrlistr!r"rr r r rrOs  rOc@sNeZdZUdZedBed<edBed<edBed<edBed<edBed<dS)SpecialAudioIDsaSpecial text tokens corresponding to audio token sequence. Attributes: audio: Token representing audio. begin_audio: Token representing the beginning of audio. streaming_pad: Token representing streaming pad of audio. Only relevant for steaming models. text_to_audio: Token representing intent to convert text to audio. audio_to_text: Token representing intent to convert audio to text. NrQ begin_audio streaming_pad text_to_audio audio_to_text)rrrrr!r"r r r rrSs    rSc @seZdZdZdededdfddZ d8dejd e d e dBd e dejf d d Z d8d e dBde eeffddZd8de d e dBde e e ffddZde d e de fddZd8d e dBdee fddZde dee fddZd8ded e dBdefddZde dee fdd Zd!e de fd"d#ZdedBd$edBdefd%d&Zd'edefd(d)Zd'edefd*d+Zd'eeBdefd,d-Ze de fd.d/Z!e de fd0d1Z"e de fd2d3Z#e de fd4d5Z$e de fd6d7Z%dS)9 AudioEncodera Encodes audio chunks into a format suitable for further processing. Attributes: audio_config: Configuration for audio processing. encoding_config: Configuration for audio spectrogram. special_ids: Special tokens for audio encoding. audio_config special_idsr NcCs||_|j|_||_dSr4)rYr&rZ)rrYrZr r r__init__s zAudioEncoder.__init__ audio_arrayr$r)kwargscKs|jjr||jd|}t|d||jdf}|S|jjr5||jd|\}}t|||f}|St|j t rS|jd|j j krSt|d|j j |jdf}|S)a9Pad the audio array to the desired length. Args: audio_array: Audio data as a numpy array. sampling_rate: Sampling rate of the audio. transcription_delay_ms (optional): Delay in milliseconds for transcription. Returns: Padded audio array. r) rYr'next_multiple_of_chunk_framesshapenppadr2_get_streaming_pad isinstancer&rr)rr\r$r)r]r_left_pad right_padr r rrbs  zAudioEncoder.padcCsT|d|\}}ttj|tjd|jjdd}ttj|tjd|jjdd}||fS)zGets left and right padding for realtime audio models. Args: transcription_delay_ms (optional): Delay in milliseconds for transcription. Returns: Tuple of left and right padding for realtime audio models. r)dtypewav)r\r$format)rcrrazerosfloat32rYr$)rr)rerfleft_pad_audioright_pad_audior r rget_padding_audios zAudioEncoder.get_padding_audior c Csl|jj}t||||}|j|}t||}t||||7}|jj}t||}t||||fSr4)rYrEr!rIrrJ) rr r)r rf_extra_right_pad_tokens_extra_right_pad_samples_extra_left_pad_tokensrer r rrc3s     zAudioEncoder._get_streaming_padaudio_array_lencCs^||jjksJd|d|jj|jjdus"Jd|jjdt||jj|jjS)zCalculate the next multiple of chunk frames. Args: audio_array_len: Length of the audio array. sampling_rate: Sampling rate of the audio. Returns: The next multiple of chunk frames. zExpected sampling_rate=z' to be self.audio_config.sampling_rate=NzMCan't call next_multiple_of_chunk_frames if self.audio_config.chunk_length_s=rA)rYr$r'r8r9r1)rrrr$r r rr_Hs z*AudioEncoder.next_multiple_of_chunk_framescCsTt|jjtsJd|jj|jjdusJ|jj|j|}|jg|}|S)z8Encode the streaming tokens given a transcription delay.zQAudio encoder must be spectrogram encoder, got self.audio_config.encoding_config=N)rdrYr&rr)rJr?rU)rr)stream_pad_prefix_lenrPr r rencode_streaming_tokens[s   z$AudioEncoder.encode_streaming_tokens signal_lengthcCs^||jjdkrt||jjd}n||jj}t||jj}|jg|jg|}|Sr6)r&rr8r9rYr:begin_audio_token audio_token)rrur;rPr r r_encode_audio_tokensjs  z!AudioEncoder._encode_audio_tokensrQcCs\||jj||j|jj||_|jjtjkr||}n | |jj d}t ||dS)z4Encode an audio optionally with transcription delay.rrPrQ) resamplerYr$rbr\r(rrrtrxr`rO)rrQr)rPr r r encode_audiovs zAudioEncoder.encode_audior;cCs&g}||j||jg||S)zBuild the token sequence for a speech request's audio segment. Args: num_audio_tokens: Number of audio placeholder tokens to emit. Returns: List of token IDs: [BEGIN_AUDIO, AUDIO * num_audio_tokens]. )appendrvextendrw)rr;rPr r r'_encode_audio_tokens_for_speech_requests z4AudioEncoder._encode_audio_tokens_for_speech_request audio_lengthcCst||jj|jjdS)zCompute the number of audio tokens needed for a given audio length. Args: audio_length: Number of audio samples. Returns: Number of audio tokens (includes +1 for END_OUTPUT_AUDIO). r7)r8r9rYr$r%)rrr r r'_get_num_audio_token_for_speech_requests z4AudioEncoder._get_num_audio_token_for_speech_requestvoicecCs|dus|dusJd|d||dur&||jj|t|j}n(|jjdus0Jd|dur:||jjvsHJd|dt|jj|jj|}||}t ||dS)aEncode audio or voice preset into an AudioEncoding for speech synthesis. Either ``audio`` (reference audio for voice cloning) or ``voice`` (preset name) must be provided. When ``audio`` is given it takes precedence. Args: audio: Reference audio waveform, or None to use a voice preset. voice: Preset voice name (e.g. 'Neutral Male', 'Neutral Female'), or None when using ref audio. Returns: AudioEncoding containing the token sequence and optional audio data. NzAEither audio or voice must be defined to encode audio, got audio=z and voice=zUvoice_num_audio_tokens must be set in audio config to use voice-based speech requestszUnknown voice z, expected one of ry) rzrYr$rlenr\r-rRr~rO)rrQrr;rPr r rencode_audio_for_speech_requests$   z,AudioEncoder.encode_audio_for_speech_requestcontentcCst|j}||Sr4)rfrom_raw_audio input_audior{)rrrQr r r_encode_audio_chunks  z AudioEncoder._encode_audio_chunkcCsT|}|tjtjhvrt|j}n|tjkrt|j}nt|j}| |Sr4) get_url_typerfilefile_urir from_fileurlfrom_url from_base64r{)rrurl_typerQr r r_encode_audio_url_chunks   z$AudioEncoder._encode_audio_url_chunkcCs:t|tr ||St|tr||Stdt|)zCall the encoder on an audio chunk or URL chunk. Args: content: Audio or URL chunk to encode. Returns: Encoded audio data and tokens. zUnsupported content type: )rdrrrr ValueErrortype)rrr r r__call__s   zAudioEncoder.__call__cC(|jjdusJd|jjd|jjS)zGet the audio token.Nzself.special_ids.audio=r/)rZrQrr r rrw zAudioEncoder.audio_tokencCr)zGet the begin audio token.Nzself.special_ids.begin_audio=r/)rZrTrr r rrvrzAudioEncoder.begin_audio_tokencCr)zGet the streaming pad token.Nzself.special_ids.streaming_pad=r/)rZrUrr r rrUrzAudioEncoder.streaming_padcCr)zGet the text_to_audio token.Nzself.special_ids.text_to_audio=r/)rZrVrr r rtext_to_audio_tokenrz AudioEncoder.text_to_audio_tokencCr)zGet the audio_to_text token.Nzself.special_ids.audio_to_text=r/)rZrWrr r raudio_to_text_tokenrz AudioEncoder.audio_to_text_tokenr4)&rrrrr#rSr[randarrayr!rFrrbtuplerrnrcr_rRrtrxrOr{r~rrLrrrrrrrMrwrvrUrrr r r rrXsJ  $$  # rX)loggingr8r< dataclassesrenumrtypingrnumpyramistral_common.audior&mistral_common.protocol.instruct.chunkrrr getLoggerrloggerrHr!rrLrrr#rOrSrXr r r rs.