o i^@sdZddlZddlmZddlmZmZddlZddlm Z ddl m m Z ddl mZddlmZmZdd lmZd d lmZeGd d d eZeGdddeZeGdddeZGddde jZGddde jZGddde jZGddde jZGddde jZGddde jZGddde jZ Gd d!d!e jZ!eGd"d#d#eZ"ed$d%Gd&d'd'e"Z#d'd#gZ$dS)(zTransformers Xcodec model.N) dataclass)OptionalUnion)PreTrainedAudioTokenizerBase) ModelOutputauto_docstring) AutoModel) XcodecConfigc@s6eZdZUdZdZeejed<dZ eej ed<dS) XcodecOutputao Args: audio_codes (`torch.LongTensor` of shape `(batch_size, num_quantizers, codes_length)`, *optional*): Discrete code indices computed using `model.encode`. audio_values (`torch.FloatTensor` of shape `(batch_size, channels, num_samples)`, *optional*) Decoded audio values obtained using the decoder part of Xcodec. N audio_codes audio_values) __name__ __module__ __qualname____doc__rrtorch LongTensor__annotations__r FloatTensorrr^/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/xcodec/modeling_xcodec.pyr s r c@$eZdZUdZdZeejed<dS)XcodecEncoderOutputz Args: audio_codes (`torch.LongTensor` of shape `(batch_size, num_quantizers, codes_length)`, *optional*): Discrete code indices computed using `model.encode`. Nr) rrrrrrrrrrrrrr- rc@r)XcodecDecoderOutputz Args: audio_values (`torch.FloatTensor` of shape `(batch_size, channels, num_samples)`, *optional*): Decoded audio values obtained using the decoder part of Xcodec. Nr) rrrrrrrrrrrrrr8rrcsHeZdZdZdedededeffdd Zdejd ejfd d Z Z S) ResidualUnitzFResidual block for SemanticEncoder and SemanticDecoder used in Xcodec.config in_channels out_channelsdilationc s\tt|_|jdd|}tj|||jd||ddd|_tj||ddd|_dS)Nr r F)stridepaddingr"groupsbias)r r! kernel_sizer&) super__init__nnELU activationunit_kernel_sizeConv1dconv1conv2)selfrr r!r"r$ __class__rrr)Fs   zResidualUnit.__init__ hidden_statereturncCs0||}||}||}||}||SN)r,r/r0)r1r4 output_tensorrrrforwardVs    zResidualUnit.forward) rrrrr intr)rTensorr8 __classcell__rrr2rrCsrcDeZdZdedededeffdd Zdejdejfd d ZZ S) SemanticEncoderBlockrr r!r#csdttfddjD|_|dkrdnd|}|dd}tj||||dd|_dS)Ncg|] }t|qSrr.0r"rr rr bz1SemanticEncoderBlock.__init__..r rr Tr'r#r$r&)r(r)r* ModuleListblock_dilations res_unitsr.conv)r1rr r!r#kernelr$r2rBrr)_s  zSemanticEncoderBlock.__init__r4r5cCs"|jD]}||}q||}|Sr6)rHrIr1r4unitrrrr8j   zSemanticEncoderBlock.forward rrrr r9r)rr:r8r;rrr2rr=^s r=c2eZdZfddZdejdejfddZZS)SemanticEncodercstt|jt|jkrtdtj|j|j|j d|j ddd|_ |j}g}t |jD]\}}t |j|j|}|t ||||g7}|}q/t||_dS)Nz:Number of strides must match the number of channel_ratios.r r Fr&)r(r)lenstrideschannel_ratios ValueErrorr*r.semantic_hidden_sizer'rI enumerater9r=rF conv_blocks)r1rr rXir#r!r2rrr)rs$  zSemanticEncoder.__init__r4r5cC"||}|jD]}||}q|Sr6)rIrXr1r4blockrrrr8rMzSemanticEncoder.forwardrrrr)rr:r8r;rrr2rrPqs rPcr<) SemanticDecoderBlockrr r!r#c st|dkrtj|ddddd|_n!d|}|dd}|ddkr(dnd}tj|||||dd|_tfd d jD|_dS) Nr rTrEr rFrQcr>rr?r@rr!rrrCrDz1SemanticDecoderBlock.__init__..) r(r)r*r.rIConvTranspose1drFrGrH)r1rr r!r#r'r$output_paddingr2r_rr)s&    zSemanticDecoderBlock.__init__r4r5cCrZr6)rIrHrKrrrr8rMzSemanticDecoderBlock.forwardrNrrr2rr^sr^crO)SemanticDecodercsttj|jt|j|jd|jd|jddd|_g}t |j D]1\}}t|j|j|}|t |jdkrIt|j|j|d}n|j}|t ||||g7}q%t ||_tj|j|j|jd|jddd|_dS)Nrr r F)r r!r'r#r$r&)r#r$r&)r(r)r*r.rVr9rTr'r/rWrSrRr^rFrXr0)r1rrXrYr#r r!r2rrr)s2   zSemanticDecoder.__init__r4r5cCs,||}|jD]}||}q||}|Sr6)r/rXr0r[rrrr8s    zSemanticDecoder.forwardr]rrr2rrbs rbcs8eZdZdZfddZddZddZdd ZZS) XcodecEuclideanCodebookz!Codebook with Euclidean distance.csjtt|j|j}|j|_|dtdg|dt|j|d||d|dS)NinitedT cluster_sizeembed embed_avg) r(r)rzeros codebook_size codebook_dimregister_bufferr:clone)r1rrfr2rrr)s  z XcodecEuclideanCodebook.__init__cCsV|j}|djddd}|d|||djddd }|jddj}|S)Nr r T)keepdimrdim)rftpowsummaxindices)r1 hidden_statesrf scaled_statesdist embed_indrrrquantizes &z XcodecEuclideanCodebook.quantizecCs8|j}|d|df}||}|j|dd}|S)Nrn)shapereshaperzview)r1rvr{ryrrrencodes  zXcodecEuclideanCodebook.encodecCst||j}|Sr6)F embeddingrf)r1ry quantizedrrrdecodeszXcodecEuclideanCodebook.decode) rrrrr)rzr~rr;rrr2rrcs   rccs6eZdZdZdeffdd ZddZddZZS) XcodecVectorQuantizationzY Vector quantization implementation. Currently supports only euclidean distance. rcstt||_dSr6)r(r)rccodebookr1rr2rrr)s z!XcodecVectorQuantization.__init__cCs|ddd}|j|}|SNrr r )permuterr~)r1rvembed_inrrrr~s zXcodecVectorQuantization.encodecCs|j|}|ddd}|Sr)rrr)r1ryrzrrrrs zXcodecVectorQuantization.decode) rrrrr r)r~rr;rrr2rrs rcsleZdZdZdeffdd ZddZddefd d Zdd e j de j fd d Z de j de j fddZ Z S) XcodecResidualVectorQuantizationzv Residual vector quantization implementation. Follows Algorithm 1 in https://huggingface.co/papers/2107.03312 rcsFttfddtjD|_j|_j|_j|_dS)Ncsg|]}tqSr)r)rA_rrrrCsz=XcodecResidualVectorQuantization.__init__..) r(r)r*rFrangenum_quantizers quantizers frame_raterirr2rrr)s   z)XcodecResidualVectorQuantization.__init__cCst|j|jdS)zReturn bandwidth per quantizer.i)mathlog2rir)r1rrrget_bandwidth_per_quantizerszs rz$The Xcodec neural audio codec model.) custom_introcseZdZfddZedejfddZdej dej fdd Z e ddej d e ed e edeej effd dZe ddej d e edeej effddZe ddej de ej d e ed e edeeej ej feff ddZZS)rcst|||_|jd|_t|j}|j|_ |j |_ | |j t ||_t||_t|j|_t|j|j|_t|j|jj|_t|j|jj|_t||_|dS)Nr )r(r)r hop_lengthpadr from_configacoustic_model_configencoderrdecoderr_adjust_dac_decoderrPencoder_semanticrbdecoder_semanticsemantic_model_configevalsemantic_modelr*r hidden_sizefcfc1fc2rr post_init)r1racoustic_modelr2rrr)s        zXcodecModel.__init__rcCsp|D]}t|tjr t|jtr|jdn|j}|df|_qt|dr4t|jtj r6t |_dSdSdS)z DAC implemented in Xcodec is slightly different from the HF version. DAC in Xcodec adjusts the output padding in every ConvTranspose1d in the decoder and removes the final `nn.Tanh` activation function. rr tanhN) rrr*r`r#tuplerarrTanhIdentity)rrr#rrrrs   zXcodecModel._adjust_dac_decoderrr5cCs|dddddf}t||j|jf}t|j|dd}|j}Wdn1s.wYtj|dd}|jddS)NrT)output_hidden_statesr ro)rrrno_gradrrvrr)r1routputsrvstackedrrr_extract_semantic_featuress  z&XcodecModel._extract_semantic_featuresNr return_dictc Cs6|dur|n|jj}|jd}|dkrtd||dur%|jjd}n||jjvr8td|d|jjd||}||dd}| |}|jd|jdkrp| t |ddd ddf|j |j f d}t j||gdd }||dddd}|j||} | d d} |s| St| S) ac input_values (`torch.FloatTensor` of shape `(batch_size, channels, num_samples)`): Float values of the input audio waveform. bandwidth (`float`, *optional*): The target bandwidth in (kbps) supports only values in `config.target_bandwidths`. Defaults to the highest available bandwidth `4.0` kbps. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`]. Returns: `torch.LongTensor` of shape `(batch_size, num_quantizers, codes_length)` containing the discrete encoded audio codes. Nr zAudio must be mono, but got rnz)This model doesn't support the bandwidth z. Select one of .r rro)rrr{rUtarget_bandwidthsrdetachr transposerrr unsqueezercatrrr~r) r1rrrchannelse_semantic_input e_semantic e_acousticrrrrrr~s,   2 zXcodecModel.encodercCs`|dur|n|jj}|dd}|j|}||dddd}||}|s,|St|S)a audio_codes (`torch.LongTensor` of shape `(batch_size, num_quantizers, codes_length)`): Discrete code indices computed using `model.encode`. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] Returns: Decoded audio values of shape `(batch_size, channels, num_samples)` obtained using the decoder part of Xcodec. Nrr r )rrrrrrrr)r1rrrquantized_acousticrrrrrs   zXcodecModel.decodecCsl|dur|n|jj}|jd}|dur|j||dd}|j||dddd|f}|s0||fSt||dS)a input_values (`torch.FloatTensor` of shape `(batch_size, channels, num_samples)`): The raw float values of the input audio waveform. audio_codes (`torch.LongTensor` of shape `(batch_size, num_quantizers, codes_length)`: Discrete code indices computed using `model.encode`. bandwidth (`float`, *optional*): Target bandwidth in kbps. Must be one of `config.target_bandwidths`. Defaults to the highest available bandwidth. bandwidth (`float`, *optional*): Target bandwidth in kbps. Must be one of `config.target_bandwidths`. Defaults to the highest available bandwidth. return_dict (`bool`, *optional*): Whether to return a [`XcodecOutput`] instead of a plain tuple. Returns: `XcodecOutput` or tuple `(audio_codes, audio_values)`: - `audio_codes` of shape `(batch_size, num_quantizers, codes_length)`: the quantized discrete codes. - `audio_values` of shape `(batch_size, channels, num_samples)`: the reconstructed audio waveform given the codes. Example: ```python >>> from datasets import load_dataset >>> from transformers import AutoFeatureExtractor, XcodecModel >>> model_id = "hf-audio/xcodec-hubert-librispeech" >>> model = XcodecModel.from_pretrained(model_id) >>> feature_extractor = AutoFeatureExtractor.from_pretrained(model_id) >>> dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> dataset = dataset.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate)) >>> audio_sample = dataset[0]['audio']['array'] >>> inputs = feature_extractor(raw_audio=audio_sample, return_tensors="pt") >>> outputs = model(**inputs) >>> audio_codes = outputs.audio_codes >>> audio_values = outputs.audio_values ``` NrnF)rr.)rr)rrr{r~rr )r1rrrrlengthrrrrr8s.  zXcodecModel.forward)NNr6)NNN)rrrr) staticmethodr*Modulerrrrrr:rfloatboolrrr~rrrr r8r;rrr2rrsR     1  r)%rr dataclassesrtypingrrrtorch.nnr*torch.nn.functional functionalrmodeling_utilsrrrrautor configuration_xcodecr r rrrrr=rPr^rbrcrrrr__all__rrrrs>        ( 2N 7