o ϯi[@sddlZddlmZmZddlmZddlZddlZddlZddl m Z ddlm Z ddl m Z ddlmZmZmZmZmZdd lmZdd lmZdd lmZd d ZGddde jZGddde jZGddde jZGddde jZddgddgddgdZ Gddde jZ!ddgddgddgdZ"Gdd d e jZ#Gd!d"d"e jZ$Gd#d$d$e jZ%Gd%d&d&e jZ&Gd'd(d(e e Z'Gd)d*d*e'Z(dS)+N)ListOptional)Union) BaseModel)nn) CodecMixin) MsgProcessor NormConv1dNormConvTranspose1dSnake1d activation)ResidualVectorQuantize) VAEBottleneck)hf_hub_downloadcCs6t|tjrtjj|jddtj|jddSdS)Ng{Gz?)stdr) isinstancerConv1dinit trunc_normal_weight constant_bias)mrG/home/ubuntu/.local/lib/python3.10/site-packages/dacvae/model/dacvae.py init_weightss rcszeZdZ          dded ed ed ed ed ededededeffdd ZdejdejfddZ ddZ Z S) ResidualUnitrSnakenoneF weight_normdimkerneldilationactstridecompresspad_modecausalnorm true_skipc  st|dg} |dg} ||} |dkrd|i}n|dkr#ddi}ntd|g}tt| | D]1\}\}}|dkr?|n| }|t| dkrK|n| }|td d |i|t|||||| ||d g7}q3tj ||_ | |_ dS) Nrr channelsELUalpha?zUnsupported activation: rr&) kernel_sizer'r%r+r*r)r) super__init__ ValueError enumerateziplenr r r Sequentialblockr,)selfr#r$r%r&r'r(r)r*r+r,kernels dilationshidden act_paramslayersir1in_chsout_chs __class__rrr3s8     zResidualUnit.__init__xycCs@|jr|S|jd|jdd}|dkr|d|| f}|S)Nr.)r,shape)r:rErFpadrrrshortcutJs zResidualUnit.shortcutcCs||}||||SN)r9rK)r:rErFrrrforwardRs zResidualUnit.forward) rrrr rrr!Fr"F) __name__ __module__ __qualname__intstrboolr3torchTensorrKrM __classcell__rrrCrrsD   0rcs0eZdZd dedeffdd ZddZZS) EncoderBlockrrr#r'c sjttt|ddddt|ddddt|ddddt|dt|d|d||dd|_dS) NrHrr)r$r% r!)r1r'r))r2r3rr8rr r r9)r:r#r'rCrrr3Xs   zEncoderBlock.__init__cC ||SrLr9r:rErrrrMh zEncoderBlock.forward)rr)rNrOrPrQr3rMrVrrrCrrWWsrWcs0eZdZdddeffddZddZZS) LSTMBlockT)skipr_cs&t||_tj|i||_dSrL)r2r3r_rLSTMlstm)r:r_argskwargsrCrrr3ms zLSTMBlock.__init__cCs8|ddd}||\}}|jr||}|dddS)NrHrr)permuterar_)r:rErF_rrrrMrs zLSTMBlock.forward)rNrOrPrSr3rMrVrrrCrr^lsr^cs>eZdZdgddfdededeffdd Zdd ZZS) Encoder@rHrjd_modelstridesd_latentcsrttd|dddg}|D]}|d9}|t||dg7}q|t|t||dddg7}tj||_||_dS)Nrrr!)r1r)rH)r'rX) r2r3r rWr rr8r9enc_dim)r:rkrlrmr?r'rCrrr3{s   zEncoder.__init__cCrZrLr[r\rrrrMr]zEncoder.forward)rNrOrPrQlistr3rMrVrrrCrrfzsrfr r.r!autor")actsr)r+cseZdZ         ddededed ed eeed eeed eeed edeeffdd ZddZddZ ddZ Z S) DecoderBlockrrjrNrX input_dim output_dimr' stride_wmrq pad_modesnormsdownsampling_factorlast_kernel_sizec st|dur td}|durtd}|durtd}||g} |} |} g} t|||| D]7\}}}}|dkr?d|i}d}nddi}d }| |} | |} | tdd |i|t| | d |||||d g7} q.| t|d dd dddddt|ddd dddddt||ddd d ddd dt||ddd d ddd dt|ddd dddddg7} | dur| t|| ddddd dg7} n| tg7} | tj ddt | | d ||d ddd g7} t | |_ t ||_dS)Nrqr)r+r r-Fr/r0Tr&rH)r1r'r*r)r+rr!r")r%r&r(r*r)r+r,rXr.rp)r$r&r(r*r)r+r,rY)r$r&r)r+r*r,r/r)r2r3default_decoder_convtr_kwargsr6r r rrIdentityr.r ModuleListr9r7 _chunk_size)r:rsrtr'rurqrvrwrxry conv_strides conv_in_dim conv_out_dimr?r&r+r) conv_strider>r*rCrrr3s^    zDecoderBlock.__init__csNtj}fddtd|jD}fddt|D}tj|}||S)Nc g|] }j||jqSrr9r~.0r@r:rr  z(DecoderBlock.forward..rcs,g|]\}}|jdkr|D]}|qqSrr~rjchunklayerrrrr,r7r9ranger~r5rr8)r:rE layer_cntchunksgrouprrrrMs  zDecoderBlock.forwardcsVtj}fddtd|jD}fddt|D}tj|t|ddS)Ncrrrrrrrrrz/DecoderBlock.upsample_group..rc,g|]\}}|jdkr|D]}|qqSrrrrrrrrrHrr:rrrrrrupsample_group zDecoderBlock.upsample_groupcsVtj}fddtd|jD}fddt|D}tj|dt|dS)Ncrrrrrrrrrz1DecoderBlock.downsample_group..rcrrrrrrrrrrHrrrrrdownsample_grouprzDecoderBlock.downsample_group) rrjrrNNNrXN) rNrOrPrQrrrRr3rMrrrVrrrCrrrs@     ErrTanhcseZdZ        ddededed ed eed eeed eeed eeeffdd ZddZddZ ddZ ddZ Z S)WatermarkEncoderBlock` Nin_dimout_dim wm_channelsr= lstm_layersrqrvrwc  s"t|dur td}|durtd}|durtd}g} tt|||D];\} \} } } | dkr4|nd}| dkrr* post_layersrCrrr3sJ    zWatermarkEncoderBlock.__init__cCrZrLrr\rrrrM:r]zWatermarkEncoderBlock.forwardc CsH|jd}ztjj|jdj||}|W||jd<S||jd<wNrG)rrTrutilsremove_weight_normconvr:rE_convrrr forward_conv=s  z"WatermarkEncoderBlock.forward_convc Cs<|jd}zt|jd<||W||jd<S||jd<wrrrr|rrrrforward_no_convFs  z%WatermarkEncoderBlock.forward_no_convcCrZrLrr\rrr post_processNr]z"WatermarkEncoderBlock.post_process)rrrrNNNN) rNrOrPrQrrrRr3rMrrrrVrrrCrrs<   9 rc sZeZdZ     ddededed ed eef fd d Zd dZddZddZZ S)WatermarkDecoderBlockrrrrNrrr-r=rc svtt||dddddg}|dur|t|||g7}tj||_tjddt||dddddg}tj||_dS)NrTr!rprr0rz) r2r3r r^rr8rr.r)r:rrr-r=rrrrCrrr3Ts    zWatermarkDecoderBlock.__init__cCs||}|SrLrr\rrrrMps zWatermarkDecoderBlock.forwardc Cs@|jd}zt|jd<||}W||jd<dS||jd<wrrrrrrrts  z%WatermarkDecoderBlock.forward_no_convcCrZrLrr\rrrr|r]z"WatermarkDecoderBlock.post_process)rrrrN) rNrOrPrQrr3rMrrrVrrrCrrSs(rcspeZdZ      ddeded ed ed ed ed eeffdd ZdefddZdejdejfddZ Z S) WatermarkerrrrrrNr#d_outrmr-r=nbitsrcsBtt|||||d|_t|||_t|||||d|_dS)N)r=r)r2r3r encoder_blockr msg_processorr decoder_block)r:r#rrmr-r=rrrCrrr3s  zWatermarker.__init__bszcCs0|jdur |jj}nd}tjdd||ftjdS)NrrrH)dtype)rrrTrandintfloat32)r:rrrrrrandom_messages  zWatermarker.random_messagerEmsgcCs$||}|||}||}|SrL)rrr)r:rErrrrrMs   zWatermarker.forward)rrrrrN) rNrOrPrQrr3rrTrUrMrVrrrCrrs0rc sjeZdZ     ddededed ed ef fd d ZddeejfddZ ddeejfddZ Z S)Decoderrrrrlinearrrrd_wm_outblendingc  stt||dddg} tt||D]\} \} } |d| }|d| d}| t||| | g7} qt| |_t ||||d|dd|_ |||_ | |_ dS)Nrr)r1r'rHr)r=rr) r2r3r r5r6rrrr}modelrwm_modelr/r)r: input_channelr-rateswm_ratesrrrrrr?r@r' wm_stridersrtrCrrr3s     zDecoder.__init__NmessagecCs |jD]}||}q|||SrL)r watermark)r:rErrrrrrMs   zDecoder.forwardcCs|jdkr|S|j|}tdd|jdd}t|ddd}|D]}||}q$|jj|}|durA|jd}|j|}| |j }|j ||}|j |}tdd|jdd}|D]}||}qb|jj |}|j dkr|jj||j|S|jj||j|S) NgcS|SrL)rrErrrz#Decoder.watermark..rrGrcSrrL)rrrrrrrr)r/rrmaprrorrIrtodevicerrrrMr)r:rErh upsamplerrr downsamplerrrrrs*         zDecoder.watermark)rrrrrrL) rNrOrPrQrRr3rrTrUrMrrVrrrCrrs&   rcseZdZdgdddgdddddd d f d ed eed eededeedeeedededeeefdedeffdd Z ddZ d$de j deefddZ d$de j dee j fd d!Z  d%de j deedeefd"d#ZZS)&DACrgrhNrjrjrirHrYrjFD encoder_dim encoder_rates latent_dim decoder_dim decoder_ratesr n_codebooks codebook_size codebook_dimquantizer_dropout sample_ratec st||_||_||_||_| |_|dur |dt|}t|t s'J|dur/gd}||_ t ||_ t||||_||_||_| |_t|||| | d|_t|||||_| |_|t||_dS)NrH)rjrirH)rsrrrr)r2r3rrrrrr7rrQrnpprod hop_lengthrfencoderrrrr quantizerrdecoderapplyr get_delaydelay) r:rrrrrrrrrrrrCrrr3sB   z DAC.__init__cCsV|dur|j}||jksJ|jd}t||j|j|}tj|d|f}|S)NrGr)rrImathceilrr functionalrJ)r: audio_datarlength right_padrrr preprocesss zDAC.preprocessr n_quantizerscCs.||}|||\}}}}}|||||fS)aeEncode given audio data and return quantized latent codes Parameters ---------- audio_data : Tensor[B x 1 x T] Audio data to encode n_quantizers : int, optional Number of quantizers to use, by default None If None, all quantizers are used. Returns ------- dict A dictionary with the following keys: "z" : Tensor[B x D x T] Quantized continuous representation of input "codes" : Tensor[B x N x T] Codebook indices for each codebook (quantized discrete representation of input) "latents" : Tensor[B x N*D x T] Projected latents (continuous representation of input before quantization) "vq/commitment_loss" : Tensor[1] Commitment loss to train encoder to predict vectors closer to codebook entries "vq/codebook_loss" : Tensor[1] Codebook loss to update the codebook "length" : int Number of samples in input audio )rr)r:rrzcodeslatentscommitment_loss codebook_lossrrrencode&s "z DAC.encoderrcCs|j||dS)a.Decode given latent codes and return audio data Parameters ---------- z : Tensor[B x D x T] Quantized continuous representation of input length : int, optional Number of samples in output audio, by default None message : Tensor[B x nbits], optional Message to embed in the audio, by default None Returns ------- dict A dictionary with the following keys: "audio" : Tensor[B x 1 x length] Decoded audio data. )r)r)r:rrrrrdecodeOsz DAC.decodec CsT|jd}|||}|||\}}}}} ||} | dd|f||||| dS)a%Model forward pass Parameters ---------- audio_data : Tensor[B x 1 x T] Audio data to encode sample_rate : int, optional Sample rate of audio data in Hz, by default None If None, defaults to `self.sample_rate` n_quantizers : int, optional Number of quantizers to use, by default None. If None, all quantizers are used. Returns ------- dict A dictionary with the following keys: "z" : Tensor[B x D x T] Quantized continuous representation of input "codes" : Tensor[B x N x T] Codebook indices for each codebook (quantized discrete representation of input) "latents" : Tensor[B x N*D x T] Projected latents (continuous representation of input before quantization) "vq/commitment_loss" : Tensor[1] Commitment loss to train encoder to predict vectors closer to codebook entries "vq/codebook_loss" : Tensor[1] Codebook loss to update the codebook "length" : int Number of samples in input audio "audio" : Tensor[B x 1 x length] Decoded audio data. rG.N)audiorrrzvq/commitment_losszvq/codebook_loss)rIrrr) r:rrrrrrrrrrErrrrMds (  z DAC.forwardrL)NN)rNrOrPrQrrrrorSr3rrTrUrrrMrVrrrCrrsh     8 )rcseZdZdgdddgddddd d f d ed eed eededeedededeeefdedeffdd Z e fddZ ddZ de jfddZde jfdd ZZS)!DACVAErgrhNrrrYrrjFrrrrrrrrrrrc s6tj||||||||| | d t|j||d|_dS)N) rrrrrrrrrr)rsrr)r2r3rrr) r:rrrrrrrrrrrCrrr3s"   zDACVAE.__init__cs.tj|s|drt|dd}t|S)Nz facebook/z weights.pth)repo_idfilename)ospathexists startswithrr2load)clsrrCrrr s  z DACVAE.loadcCs>|d}||jrd|j||jf}tjj||dS|S)NrGrreflect)sizerrTrrrJ)r:wavsrp1drrr_pads  z DACVAE._padrcCs@|||}|j|jddd\}}|j||\}}|S)NrHr)r#)rrrin_projr _vae_sample)r:rrmeanscaleencoded_framesrerrrrsz DACVAE.encodercCs|j|}||SrL)rout_projr)r:rembrrrrs  z DACVAE.decode)rNrOrPrQrrrrorSr3 classmethodr rrTrUrrrVrrrCrrsN     r))rtypingrrrnumpyrrrT audiotools.mlrrbaserdacvae.nn.layersr r r r r dacvae.nn.quantizerdacvae.nn.bottleneckrhuggingface_hubrrModulerrWr^rfr{rrrrrrrrrrrrrsD       >[S-C<