o %ݫi1@s`dZddlZddlmZddlmZmZddlm Z ddl m Z dZ e e ZGdd d e ZdS) aThis lobe enables the integration of huggingface pretrained EnCodec. EnCodec makes it possible to compress audio into a sequence of discrete tokens at different bandwidths - and to reconstruct audio from such sequences, with some loss of quality depending on the bandwidth. Note that while encodec can be used to reconstruct speech data, for a high-quality reconstruction, it is recommended to use a specially trained vocoder, such as Vocos (speechbrain.lobes.models.huggingface_transformers.vocos) Repository: https://huggingface.co/docs/transformers/v4.31.0/en/model_doc/encodec Paper: https://arxiv.org/abs/2210.13438 Authors * Artem Ploujnikov 2023 N) functional)clean_padding_length_to_mask)HFTransformersInterface) get_loggeri]cseZdZdZ      dfdd Zdd Zdd d Zd d ZddZddZ ddZ ddZ ddZ dddZ dddZddZZS) EncodecaUAn wrapper for the HuggingFace encodec model Arguments --------- source : str A HuggingFace repository identifier or a path save_path : str The location where the pretrained model will be saved sample_rate : int The audio sampling rate bandwidth : float The encoding bandwidth, in kbps (optional) Supported bandwidths: 1.5, 3.0, 6.0, 12.0, 24.0 flat_embeddings : bool If set to True, embeddings will be flattened into (Batch x Length x (Heads * Embedding)) freeze : bool whether the model will be frozen (e.g. not trainable if used as part of training another model) renorm_embeddings : bool whether embeddings should be renormalized. In the original model. Example ------- >>> model_hub = "facebook/encodec_24khz" >>> save_path = "savedir" >>> model = Encodec(model_hub, save_path) >>> audio = torch.randn(4, 1000) >>> length = torch.tensor([1.0, .5, .75, 1.0]) >>> tokens, emb = model.encode(audio, length) >>> tokens.shape torch.Size([4, 4, 2]) >>> emb.shape torch.Size([4, 4, 2, 128]) >>> rec = model.decode(tokens, length) >>> rec.shape torch.Size([4, 1, 1280]) >>> rec_emb = model.decode_emb(emb, length) >>> rec_emb.shape torch.Size([4, 1, 1280]) >>> rec_tokens = model.tokens(emb, length) >>> rec_tokens.shape torch.Size([4, 4, 2]) >>> model = Encodec(model_hub, save_path, flat_embeddings=True) >>> _, emb = model.encode(audio, length) >>> emb.shape torch.Size([4, 4, 256]) N?FTcs<tj|||d|s t}||_||_||_|jj||_ |jj j |_ |jjj d|j }tdd|D} |d| |jj\} |_ |_|j|j |j |j} |d| t|j ddddf|j } |d| ||_|jr|\} }|d| |d||jrtd |jD]}d |_qdSdS) N)source save_pathfreezecSsg|]}|jjqS)codebookembed).0layerr r m/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/huggingface_transformers/encodec.py jsz$Encodec.__init__.. vocabularyvocabulary_flattoken_index_offsetsemb_meanemb_stdz(huggingface_Encodec - Encodec is frozen.F)super__init__DEFAULT_SAMPLE_RATE sample_rate bandwidthflat_embeddingsmodel quantizer get_num_quantizers_for_bandwidth num_headsconfig codebook_size num_tokenslayerstorchstackregister_bufferrshapeemb_dimreshapearangerenorm_embeddings _precalibrater loggerwarning parameters requires_grad)selfr r rrrr r-quantizer_layersr_rrrrparam __class__r rrTsD          zEncodec.__init__cCs2t|jddddfd|j|j}||S)z5Compute parameters required to renormalize embeddingsN)r&r,r$expandr!_compute_embedding_norm)r3sampler r rr.s  zEncodec._precalibratec Cs|durtjt||jd}|d}||}t|||ddddddf|}|d d|d dddddddf}|||d}|j gdd||j gddd dddddf}||fS) aComputes the normalization for embeddings based on a sample. Arguments --------- sample : torch.Tensor A (Batch x Samples) or (Batch x Channel x Samples) audio sample length : torch.Tensor A tensor of relative lengths Returns ------- emb_mean : torch.Tensor emb_std : torch.Tensor Norm stats for embeddings. Ndevicer9r)rr9dim) r&oneslenr>size_raw_embeddingsr expand_asmeansumsqrt) r3r<lengthmax_lenembmaskr emb_diff_sqrr r rr;s&   $zEncodec._compute_embedding_normcCsB|jstd|||}|||\|_|_|j|jfS)aCalibrates the normalization on a sound sample Arguments --------- sample : torch.Tensor A (Batch x Samples) or (Batch x Channel x Samples) audio sample length : torch.Tensor A tensor of relative lengths Returns ------- emb_mean : torch.Tensor The embedding mean emb_std : torch.Tensor The embedding standard deviation z0Not supported when renorm_embeddings is disabled)r- ValueError_encode_tokensr;rrsqueeze)r3r<rL sample_tokensr r r calibrates  zEncodec.calibratecCs |||S)aEncodes the input audio as tokens Arguments --------- inputs : torch.Tensor A (Batch x Samples) or (Batch x Channel x Samples) tensor of audio length : torch.Tensor A tensor of relative lengths Returns ------- tokens : torch.Tensor A (Batch X Tokens) tensor of audio tokens )encode)r3inputsrLr r rforwards zEncodec.forwardcCsPt|j |||}||}||fWdS1s!wYdS)aEncodes the input audio as tokens and embeddings Arguments --------- inputs : torch.Tensor A (Batch x Samples) or (Batch x Channel x Samples) tensor of audio length : torch.Tensor A tensor of relative lengths Returns ------- tokens : torch.Tensor A (Batch x Tokens x Heads) tensor of audio tokens emb : torch.Tensor Raw vector embeddings from the model's quantizers N)r&set_grad_enabledr rR embeddings)r3rWrLtokensrNr r rrVs   $zEncodec.encodecCsf|dkr |d}|d}t||||jdd}|jj|||jd}|j d dd}|S)aEncodes audio as tokens only Arguments --------- inputs : torch.Tensor A (Batch x Samples) or (Batch x Channel x Samples) tensor of audio length : torch.Tensor A tensor of relative lengths Returns ------- tokens : torch.Tensor A (Batch x Tokens x Heads) tensor of audio tokens r@r9r?r=)rr) rC unsqueezerFrr>rrVr audio_codesrS transpose)r3rWrLrMrOresultr[r r rrRs    zEncodec._encode_tokenscCs||j}t||j}|S)aConverts token indexes to vector embeddings, for each quantizer Arguments --------- tokens : torch.Tensor a (Batch x Length x Heads) tensor of token indexes Returns ------- emb : torch.Tensor a (Batch x Length x Heads x Embedding) tensor of raw vector embeddings from the model's quantizer codebooks )rF embeddingr)r3r[idxrNr r rrG s zEncodec._raw_embeddingscCsJ||}|jr||j|j}|jr#|j\}}}}|||||}|S)aConverts token indexes to vector embeddings Arguments --------- tokens : torch.Tensor a (Batch x Length x Heads) tensor of token indexes Returns ------- emb : torch.Tensor a (Batch x Length x Heads x Embedding) tensor of raw vector embeddings from the model's quantizer codebooks )rGr-rrrr)r+)r3r[rN batch_sizerMr!r*r r rrZs zEncodec.embeddingscCslt|j %|j|ddddg}|j}|dur#t|||WdS1s/wYdS)aPDecodes audio from tokens Arguments --------- tokens : torch.Tensor A (Batch x Length x Heads) tensor of audio tokens length : torch.Tensor A 1-D tensor of relative lengths Returns ------- audio : torch.Tensor the reconstructed audio rr?r\N) r&rYr rdecoder]r_ audio_valuesr)r3r[rLr`audior r rre5s $zEncodec.decodec Cst|j s|jr|j\}}}||||j|j}|jr&||j |j }| dj ddd}|j ddd}|dddd}||dd} | dj ddddd} |d| |  } | jdd j} |d urqt| || Wd S1s}wYd S) a{Comberts embeddings to raw tokens Arguments --------- emb : torch.Tensor Raw embeddings length : torch.Tensor A 1-D tensor of relative lengths. If supplied, padded positions will be zeroed out Returns ------- tokens : torch.Tensor A (Batch x Length) tensor of token indicesr@r?T)keepdimr\rr9rArBN)r&rYr rr)r+r!r*r-rrpowrJrr_r]permutemoveaxismaxindicesr) r3rNrLrdrMr5 scaled_statesvocabemb_permemb_vocab_prod vocab_sumdistr[r r rr[Ms&   $zEncodec.tokenscCsHt|j ||}|||WdS1swYdS)aDecodes raw vector embeddings into audio Arguments --------- emb : torch.Tensor A (Batch x Length x Heads x Embedding) tensor of raw vector embeddings length : torch.Tensor The corresponding lengths of the inputs. Returns ------- audio : torch.Tensor the reconstructed audio N)r&rYr r[re)r3rNrLr[r r r decode_embos  $zEncodec.decode_emb)NNrFTT)N)__name__ __module__ __qualname____doc__rr.r;rUrXrVrRrGrZrer[rt __classcell__r r r7rr s(6, #  "r)rxr&torch.nnrraspeechbrain.dataio.dataiorr=speechbrain.lobes.models.huggingface_transformers.huggingfacerspeechbrain.utils.loggerrrrur/rr r r rs