o %ݫiT@sPdZddlZddlmZmZddlmZddlmZee Z GdddeZ dS)aThis lobe enables the integration of huggingface pretrained Mimi. Mimi codec is a state-of-the-art audio neural codec, developed by Kyutai. It combines semantic and acoustic information into audio tokens running at 12Hz and a bitrate of 1.1kbps. Note that you need to install `transformers>=4.45.1` to use this module. Repository: https://huggingface.co/kyutai/mimi Paper: https://kyutai.org/Moshi.pdf Authors * Pooneh Mousavi 2024 N)clean_padding_length_to_mask)HFTransformersInterface) get_loggercsReZdZdZ   dfdd ZeddZd d Zd d Z dddZ Z S)MimiaThis lobe enables the integration of HuggingFace pretrained Mimi model. Mimi codec is a state-of-the-art audio neural codec, developed by Kyutai. It combines semantic and acoustic information into audio tokens running at 12Hz and a bitrate of 1.1kbps. Source paper: https://kyutai.org/Moshi.pdf Transformers>=4.45.1 from HuggingFace needs to be installed: https://huggingface.co/transformers/installation.html The code is adapted from the official HF Kyutai repository: https://huggingface.co/kyutai/mimi Arguments --------- source : str A HuggingFace repository identifier or a path save_path : str The location where the pretrained model will be saved sample_rate : int (default: 24000) The audio sampling rate freeze : bool whether the model will be frozen (e.g. not trainable if used as part of training another model) num_codebooks : int (default: 8) Number of codebooks. It could be [2,3,4,5,6,7,8] Example ------- >>> model_hub = "kyutai/mimi" >>> save_path = "savedir" >>> model = Mimi(model_hub, save_path) >>> audio = torch.randn(4, 48000) >>> length = torch.tensor([1.0, .5, .75, 1.0]) >>> tokens, emb = model.encode(audio, length) >>> tokens.shape torch.Size([4, 8, 25]) >>> emb.shape torch.Size([4, 8, 25, 256]) >>> rec = model.decode(tokens, length) >>> rec.shape torch.Size([4, 1, 48000]) ]Tcs(tj|||d||_||_d|_dS)N)source save_pathfreeze)super__init__ num_codebooks sample_rate embeddings)selfr r rr r __class__j/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/huggingface_transformers/mimi.pyr Fs  z Mimi.__init__cCsF|jjjj}|jjjj}||d|j}dd|D}t|}|S)NcSsg|]}|jjqSr)codebookembed).0layerrrr ]sz+Mimi._compute_embedding..)model quantizer"semantic_residual_vector_quantizerlayers"acoustic_residual_vector_quantizerrtorchstack)rsemantic_layersacoustic_layersrembsrrr_compute_embeddingTs   zMimi._compute_embeddingcCs&|||\}}|||}|||fS)a`Encodes the input audio as tokens and embeddings and decodes audio from tokens Arguments --------- inputs : torch.Tensor A (Batch x Samples) or (Batch x Channel x Samples) tensor of audio length : torch.Tensor A tensor of relative lengths Returns ------- tokens : torch.Tensor A (Batch x Tokens x Heads) tensor of audio tokens emb : torch.Tensor Raw vector embeddings from the model's quantizers audio : torch.Tensor the reconstructed audio )encodedecode)rinputslengthtokens embeddingaudiorrrforwardas  z Mimi.forwardcCs|jdur ||_|dkr|d}|d}t||||jdd}|jj|||j dd}|d ddd|jj d}t |jd |j ddddd|}||fS)aEncodes the input audio as tokens and embeddings Arguments --------- inputs : torch.Tensor A (Batch x Samples) or (Batch x Channel x Samples) tensor of audio length : torch.Tensor A tensor of relative lengths Returns ------- tokens : torch.Tensor A (Batch x num_codebooks x Length) tensor of audio tokens emb : torch.Tensor Raw vector embeddings from the model's quantizers N)device)num_quantizersr)rr%dim unsqueezesizerr1rr&rexpandshaper gather)rr(r)max_len padding_maskr* input_tensorrrrrr&|s0       z Mimi.encodeNcCs<|jdur ||_|j|}|j}|durt|||S)aXDecodes audio from tokens Arguments --------- tokens : torch.Tensor A (Batch x num_codebooks x Length) tensor of audio tokens length : torch.Tensor A 1-D tensor of relative lengths Returns ------- audio : torch.Tensor the reconstructed audio N)rr%rr' audio_valuesr)rr*r)resultr,rrrr's    z Mimi.decode)rTr)N) __name__ __module__ __qualname____doc__r r no_gradr%r-r&r' __classcell__rrrrrs/  .r) rAr speechbrain.dataio.dataiorr=speechbrain.lobes.models.huggingface_transformers.huggingfacerspeechbrain.utils.loggerrr>loggerrrrrrs