o eic@sdZddlZddlmZddlmZddlZddlmZddl mm Z ddl m ZddlmZddlmZdd lmZmZd d lmZd d lmZeGdddeZeGdddeZeGdddeZGdddejZGdddejZGdddejZ GdddejZ!GdddejZ"GdddejZ#Gd d!d!ejZ$Gd"d#d#ejZ%eGd$d%d%eZ&ed&d'Gd(d)d)e&Z'd)d%gZ(dS)*zTransformers Xcodec model.N) dataclass) lru_cache)initialization)conv1d_output_length)PreTrainedAudioTokenizerBase) ModelOutputauto_docstring) AutoModel) XcodecConfigc@s6eZdZUdZdZejdBed<dZej dBed<dS) XcodecOutputao Args: audio_codes (`torch.LongTensor` of shape `(batch_size, num_quantizers, codes_length)`, *optional*): Discrete code indices computed using `model.encode`. audio_values (`torch.FloatTensor` of shape `(batch_size, channels, num_samples)`, *optional*) Decoded audio values obtained using the decoder part of Xcodec. N audio_codes audio_values) __name__ __module__ __qualname____doc__rtorch LongTensor__annotations__r FloatTensorrrh/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/xcodec/modeling_xcodec.pyr s rc@$eZdZUdZdZejdBed<dS)XcodecEncoderOutputz Args: audio_codes (`torch.LongTensor` of shape `(batch_size, num_quantizers, codes_length)`, *optional*): Discrete code indices computed using `model.encode`. Nr)rrrrrrrrrrrrr. rc@r)XcodecDecoderOutputz Args: audio_values (`torch.FloatTensor` of shape `(batch_size, channels, num_samples)`, *optional*): Decoded audio values obtained using the decoder part of Xcodec. Nr)rrrrrrrrrrrrr9rrcsHeZdZdZdedededeffdd Zdejd ejfd d Z Z S) ResidualUnitzFResidual block for SemanticEncoder and SemanticDecoder used in Xcodec.config in_channels out_channelsdilationc s\tt|_|jdd|}tj|||jd||ddd|_tj||ddd|_dS)Nr r F)stridepaddingr#groupsbias)r!r" kernel_sizer') super__init__nnELU activationunit_kernel_sizeConv1dconv1conv2)selfr r!r"r#r% __class__rrr*Gs   zResidualUnit.__init__ hidden_statereturncCs0||}||}||}||}||SN)r-r0r1)r2r5 output_tensorrrrforwardWs    zResidualUnit.forward) rrrrr intr*rTensorr9 __classcell__rrr3rrDsrcDeZdZdedededeffdd Zdejdejfd d ZZ S) SemanticEncoderBlockr r!r"r$csdttfddjD|_|dkrdnd|}|dd}tj||||dd|_dS)Ncg|] }t|qSrr.0r#r r!rr cz1SemanticEncoderBlock.__init__..r rr Tr(r$r%r')r)r*r+ ModuleListblock_dilations res_unitsr/conv)r2r r!r"r$kernelr%r3rCrr*`s  zSemanticEncoderBlock.__init__r5r6cCs"|jD]}||}q||}|Sr7)rIrJr2r5unitrrrr9k   zSemanticEncoderBlock.forward rrrr r:r*rr;r9r<rrr3rr>_s r>c2eZdZfddZdejdejfddZZS)SemanticEncodercstt|jt|jkrtdtj|j|j|j d|j ddd|_ |j}g}t |jD]\}}t |j|j|}|t ||||g7}|}q/t||_dS)Nz:Number of strides must match the number of channel_ratios.r r Fr')r)r*lenstrideschannel_ratios ValueErrorr+r/semantic_hidden_sizer(rJ enumerater:r>rG conv_blocks)r2r r!rYir$r"r3rrr*ss$  zSemanticEncoder.__init__r5r6cC"||}|jD]}||}q|Sr7)rJrYr2r5blockrrrr9rNzSemanticEncoder.forwardrrrr*rr;r9r<rrr3rrQrs rQcr=) SemanticDecoderBlockr r!r"r$c st|dkrtj|ddddd|_n!d|}|dd}|ddkr(dnd}tj|||||dd|_tfd d jD|_dS) Nr rTrFr rFrRcr?rr@rAr r"rrrDrEz1SemanticDecoderBlock.__init__..) r)r*r+r/rJConvTranspose1drGrHrI)r2r r!r"r$r(r%output_paddingr3r`rr*s&    zSemanticDecoderBlock.__init__r5r6cCr[r7)rJrIrLrrrr9rNzSemanticDecoderBlock.forwardrOrrr3rr_sr_crP)SemanticDecodercsttj|jt|j|jd|jd|jddd|_g}t |j D]1\}}t|j|j|}|t |jdkrIt|j|j|d}n|j}|t ||||g7}q%t ||_tj|j|j|jd|jddd|_dS)Nrr r F)r!r"r(r$r%r')r$r%r')r)r*r+r/rWr:rUr(r0rXrTrSr_rGrYr1)r2r rYrZr$r!r"r3rrr*s2   zSemanticDecoder.__init__r5r6cCs,||}|jD]}||}q||}|Sr7)r0rYr1r\rrrr9s    zSemanticDecoder.forwardr^rrr3rrcs rccs8eZdZdZfddZddZddZdd ZZS) XcodecEuclideanCodebookz!Codebook with Euclidean distance.csjtt|j|j}|j|_|dtdg|dt|j|d||d|dS)NinitedT cluster_sizeembed embed_avg) r)r*rzeros codebook_size codebook_dimregister_bufferr;clone)r2r rgr3rrr*s  z XcodecEuclideanCodebook.__init__cCsV|j}|djddd}|d|||djddd }|jddj}|S)Nr r T)keepdimrdim)rgtpowsummaxindices)r2 hidden_statesrg scaled_statesdist embed_indrrrquantizes &z XcodecEuclideanCodebook.quantizecCs8|j}|d|df}||}|j|dd}|S)Nro)shapereshaper{view)r2rwr|rzrrrencodes  zXcodecEuclideanCodebook.encodecCst||j}|Sr7)F embeddingrg)r2rz quantizedrrrdecodeszXcodecEuclideanCodebook.decode) rrrrr*r{rrr<rrr3rrds   rdcs6eZdZdZdeffdd ZddZddZZS) XcodecVectorQuantizationzY Vector quantization implementation. Currently supports only euclidean distance. r cstt||_dSr7)r)r*rdcodebookr2r r3rrr*s z!XcodecVectorQuantization.__init__cCs|ddd}|j|}|SNrr r )permuterr)r2rwembed_inrrrrs zXcodecVectorQuantization.encodecCs|j|}|ddd}|Sr)rrr)r2rzr{rrrrs zXcodecVectorQuantization.decode) rrrrr r*rrr<rrr3rrs rcsleZdZdZdeffdd ZddZddefd d Zdd e j de j fd d Z de j de j fddZ Z S) XcodecResidualVectorQuantizationzv Residual vector quantization implementation. Follows Algorithm 1 in https://huggingface.co/papers/2107.03312 r csFttfddtjD|_j|_j|_j|_dS)Ncsg|]}tqSr)r)rB_r rrrDsz=XcodecResidualVectorQuantization.__init__..) r)r*r+rGrangenum_quantizers quantizers frame_raterjrr3rrr*s   z)XcodecResidualVectorQuantization.__init__cCst|j|jdS)zReturn bandwidth per quantizer.i)mathlog2rjr)r2rrrget_bandwidth_per_quantizersz.get_conv1d_layers_recursive)r+Moduletuple)r2rrrr_get_conv1d_layerss z(XcodecPreTrainedModel._get_conv1d_layersNcCs.|dur|}||}|D]}t||}q |S)zo For a given module, compute the output length that would be obtained after all Conv1d layers. N)rr)r2 input_lengthr conv1d_layerslayerrrr_get_conv1d_output_lengthss   z0XcodecPreTrainedModel._get_conv1d_output_lengthsr7)rrrrr config_classbase_model_prefixmain_input_nameinput_modalitiesrno_gradrrrrrrrrrrr?s % rz$The Xcodec neural audio codec model.) custom_introcseZdZfddZedejfddZdej dej fdd Z e ddej d e d Bd ed Bdej eBfd dZe ddej d ed Bdej eBfddZe ddej dej d Bd e d Bd ed Bdeej ej feBf ddZZS)rcst|||_|jd|_t|j}|j|_ |j |_ | |j t ||_t||_t|j|_t|j|j|_t|j|jj|_t|j|jj|_t||_|dS)Nr )r)r*r hop_lengthpadr from_configacoustic_model_configencoderrdecoderr_adjust_dac_decoderrQencoder_semanticrcdecoder_semanticsemantic_model_configevalsemantic_modelr+r hidden_sizefcfc1fc2rr post_init)r2r acoustic_modelr3rrr*s        zXcodecModel.__init__rcCsp|D]}t|tjr t|jtr|jdn|j}|df|_qt|dr4t|jtj r6t |_dSdSdS)z DAC implemented in Xcodec is slightly different from the HF version. DAC in Xcodec adjusts the output padding in every ConvTranspose1d in the decoder and removes the final `nn.Tanh` activation function. rr tanhN) rrr+rar$rrbrrTanhIdentity)rrr$rrrrs   zXcodecModel._adjust_dac_decoderrr6cCs|dddddf}t||j|jf}t|j|dd}|j}Wdn1s.wYtj|dd}|jddS)NrT)output_hidden_statesr rp)rrrrrrwrr)r2routputsrwstackedrrr_extract_semantic_featuress  z&XcodecModel._extract_semantic_featuresNr return_dictc Cs*|dur|n|jj}|jd}|dkrtd||dur%|jjd}n||jjvr8td|d|jjd||}||dd}| |jd|j |jdkre| t ||j |j f}n| |}t j||gdd }||dddd}|j||} | d d} |s| St| S) ac input_values (`torch.FloatTensor` of shape `(batch_size, channels, num_samples)`): Float values of the input audio waveform. bandwidth (`float`, *optional*): The target bandwidth in (kbps) supports only values in `config.target_bandwidths`. Defaults to the highest available bandwidth `4.0` kbps. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`]. Returns: `torch.LongTensor` of shape `(batch_size, num_quantizers, codes_length)` containing the discrete encoded audio codes. Nr zAudio must be mono, but got roz)This model doesn't support the bandwidth z. Select one of .r rpr)r rr|rVtarget_bandwidthsrdetachr transposerrrrrcatrrrr) r2rrrchannelse_semantic_input e_semantic e_acousticrrrrrrs,    zXcodecModel.encodercCs`|dur|n|jj}|dd}|j|}||dddd}||}|s,|St|S)a audio_codes (`torch.LongTensor` of shape `(batch_size, num_quantizers, codes_length)`): Discrete code indices computed using `model.encode`. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] Returns: Decoded audio values of shape `(batch_size, channels, num_samples)` obtained using the decoder part of Xcodec. Nrr r )r rr rrrrr)r2rrrquantized_acousticrrrrrs   zXcodecModel.decodecCsl|dur|n|jj}|jd}|dur|j||dd}|j||dddd|f}|s0||fSt||dS)a input_values (`torch.FloatTensor` of shape `(batch_size, channels, num_samples)`): The raw float values of the input audio waveform. audio_codes (`torch.LongTensor` of shape `(batch_size, num_quantizers, codes_length)`: Discrete code indices computed using `model.encode`. bandwidth (`float`, *optional*): Target bandwidth in kbps. Must be one of `config.target_bandwidths`. Defaults to the highest available bandwidth. bandwidth (`float`, *optional*): Target bandwidth in kbps. Must be one of `config.target_bandwidths`. Defaults to the highest available bandwidth. return_dict (`bool`, *optional*): Whether to return a [`XcodecOutput`] instead of a plain tuple. Returns: `XcodecOutput` or tuple `(audio_codes, audio_values)`: - `audio_codes` of shape `(batch_size, num_quantizers, codes_length)`: the quantized discrete codes. - `audio_values` of shape `(batch_size, channels, num_samples)`: the reconstructed audio waveform given the codes. Example: ```python >>> from datasets import load_dataset >>> from transformers import AutoFeatureExtractor, XcodecModel >>> model_id = "hf-audio/xcodec-hubert-librispeech" >>> model = XcodecModel.from_pretrained(model_id) >>> feature_extractor = AutoFeatureExtractor.from_pretrained(model_id) >>> dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> dataset = dataset.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate)) >>> audio_sample = dataset[0]['audio']['array'] >>> inputs = feature_extractor(raw_audio=audio_sample, return_tensors="pt") >>> outputs = model(**inputs) >>> audio_codes = outputs.audio_codes >>> audio_values = outputs.audio_values ``` NroF)rr.)rr)r rr|rrr)r2rrrrlengthrrrrr92s.  zXcodecModel.forward)NNr7)NNN)rrrr* staticmethodr+rrrrrr r;floatboolrrrrrrr9r<rrr3rrsR    3 r))rr dataclassesr functoolsrrtorch.nnr+torch.nn.functional functionalrrr audio_utilsrmodeling_utilsrrrr autor configuration_xcodecr rrrrrr>rQr_rcrdrrrr__all__rrrrsB            ( 2u 9