o i4m@sddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl mZmZddlmZmZmZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZddZ d/deddej!fddZ"Gdddej!Z#Gdddej!Z$Gdddej!Z%Gdddej!Z&Gdddej!Z'Gd d!d!ej!Z(Gd"d#d#ej!Z)Gd$d%d%ej!Z*d&ee+effd'd(Z,d)ee+effd*d+Z-d,ee+effd-d.Z.dS)0N)nn) functional) transforms) Activation1d)WNConv1dWNConvTranspose1d)LiteralDictAny) prepare_audio) SnakeBeta) BottleneckDiscreteBottleneck)create_pretransform_from_configcreate_bottleneck_from_config) PretransformcOs(|ddtjjj|g|Ri|S)N use_reentrantF) setdefaulttorchutils checkpoint)functionargskwargsri/home/ubuntu/.local/lib/python3.10/site-packages/solospeech/vae_modules/stable_vae/models/autoencoders.pyrs rF activation)elusnakenonereturncCsT|dkr t}n|dkrt|}n|dkrt}ntd||r(t|}|S)Nrrr zUnknown activation )rELUr Identity ValueErrorr)r antialiaschannelsactrrrget_activations   r(c&eZdZdfdd ZddZZS) ResidualUnitFc snt||_|dd}tt|rdnd||dt||d||dt|r'dnd||dt||dd |_dS) Nrrr%r&) in_channels out_channels kernel_sizedilationpaddingr)r/r0r1)super__init__r2r Sequentialr(rlayers)selfr/r0r2 use_snakeantialias_activationr3 __class__rrr5's   zResidualUnit.__init__cCs|}||}||SNr7)r8xresrrrforward7s zResidualUnit.forwardFF__name__ __module__ __qualname__r5rA __classcell__rrr;rr*&sr*cr)) EncoderBlockFcstttt||d|dt||d|dt||d|dt|r!dnd||dt||d||t|dd |_ dS) Nrr/r0r2r9 rrr-r,r/r0r1strider3) r4r5rr6r*r(rmathceilr7)r8r/r0rMr9r:r;rrr5@s  zEncoderBlock.__init__cC ||Sr=r>r8r?rrrrAO zEncoderBlock.forwardrBrCrrr;rrH?srHcr)) DecoderBlockFc st|rttj|ddt||d|dddd}nt||d||t|dd}tt |r4d nd ||d |t ||d|d t ||d |d t ||d|d |_ dS)Nnearest) scale_factormoder,rFsame)r/r0r1rMbiasr3rLrrr-rIrJrK) r4r5rr6UpsamplerrrNrOr(r*r7)r8r/r0rMr9r:use_nearest_upsampleupsample_layerr;rrr5Ss:    zDecoderBlock.__init__cCrPr=r>rQrrrrAprRzDecoderBlock.forwardFFFrCrrr;rrSRsrScs<eZdZdddgdgdddffdd Zd d ZZS) OobleckEncoderr, rr,r,rarbrbFc stdg|}t||_t||d|dddg}t|jdD]} |t|| ||| d||| |dg7}q#|t|rCdnd||d |d t|d ||dddg7}tj ||_ dS) Nrrr.rJr/r0r1r3)r/r0rMr9rrr-) r4r5lendepthrrangerHr(rr6r7) r8r/r& latent_dimc_multsstridesr9r:r7ir;rrr5ts  0zOobleckEncoder.__init__cCrPr=r>rQrrrrArRzOobleckEncoder.forwardrCrrr;rr]ssr]c s@eZdZdddgdgdddddf fdd Zd d ZZS) OobleckDecoderr,r^r_r`rcFTc stdg|}t||_t||d|dddg} t|jdddD]} | t|| ||| d||| d|||dg7} q%| t|rIdnd ||d|d t|d||ddd d | rdt nt g7} tj | |_ dS) Nrrer.rJrdr)r/r0rMr9r:rZrrr-F)r/r0r1r3rX) r4r5rfrgrrhrSr(rTanhr#r6r7) r8r0r&rirjrkr9r:rZ final_tanhr7rlr;rrr5s(      zOobleckDecoder.__init__cCrPr=r>rQrrrrArRzOobleckDecoder.forwardrCrrr;rrms'rmcr))DACEncoderWrapperrc stddlm}|dd}|ddt|d}|dd|i||_||_|dur8tj |jj |dd nt |_ |dkrSt ||dd d d d |jjd<dSdS)Nr)Encoderrid_modelr,rkd_latentr)r1@r.rJ)r1r3r)r4r5 dac.model.dacrqpoprfencoderrirConv1denc_dimr#proj_outrgetblock)r8r/r DACEncoderriencoder_out_dimr;rrr5s   &$zDACEncoderWrapper.__init__cCs||}||}|Sr=)rwrzrQrrrrAs  zDACEncoderWrapper.forwardrrCrrr;rrpsrpcr))DACDecoderWrapperrc s:tddlm}|di|||d|_||_dS)Nr)Decoder) input_channeld_outr)r4r5rurdecoderri)r8rir0r DACDecoderr;rrr5s   zDACDecoderWrapper.__init__cCrPr=)rrQrrrrArRzDACDecoderWrapper.forwardrrCrrr;rrs rcsteZdZ      ddedeffdd Zddd Zdd d Zd d ZddZ ddZ dddZ dddZ Z S)AudioAutoencoderr,NF bottleneck pretransformc st||_||_||_||_||_||_|j|_| dur"| |_| dur)| |_||_ ||_ ||_ ||_ | |_ |j duo@|j j|_dSr=)r4r5downsampling_ratio sample_rateri io_channelsr/r0 min_lengthrrwrr soft_clip is_discrete) r8rwrrirrrrrr/r0rr;rrr5s$ zAudioAutoencoder.__init__c Ksi}|jdurx|sx|jjr9|r2g}t|jdD]}||j|||dqtj|dd}nF|j|}n?t3|rcg}t|jdD]}||j|||dqItj|dd}n|j|}Wdn1sswY|j dur|rg} t|jdD]}| | |||dqtj| dd} n| |} n|} |j dur|j j| fddi|\} } | | |r| |fS| S)Nrrdim return_infoT) r enable_gradrhshapeappendencodercatno_gradrwrupdate) r8audiorskip_pretransform iterate_batchrinfoaudiosrllatentsbottleneck_inforrrrs@        zAudioAutoencoder.encodec Ks|jdur0|r*g}t|jdD]}||j|||dqtj|dd}n|j|}|rTg}t|jdD]}|||||dq;tj|dd}n |j|fi|}|jdur|jj r|rg}t|jdD]}||j|||dqqtj|dd}nF|j|}n?t 3|rg}t|jdD]}||j|||dqtj|dd}n|j|}Wdn1swY|j rt |}|S)Nrrr) rrhrrdecoderrrrrrrtanh)r8rrrdecodedrldecodedsrrrr?sB         zAudioAutoencoder.decodecKs:t|jts Jd|jj|fi|}|j|fi|S)z_ Decode discrete tokens to audio Only works with discrete autoencoders z3decode_tokens only works with discrete autoencoders) isinstancerr decode_tokensr)r8tokensrrrrrrjszAudioAutoencoder.decode_tokenscCs||g|gS)a Preprocess single audio tensor (Channels x Length) to be compatible with the encoder. If the model is mono, stereo audio will be converted to mono. Audio will be silence-padded to be a multiple of the model's downsampling ratio. Audio will be resampled to the model's sample rate. The output will have batch size 1 and be shape (1 x Channels x Length) )!preprocess_audio_list_for_encoder)r8rin_srrrrpreprocess_audio_for_encoderwsz-AudioAutoencoder.preprocess_audio_for_encoderc CsRt|}t|tr|g|}t||ksJdg}d}t|D]Z}||}||}t|jdkr>|jddkr>|d}n t|jdkrJ|d}t|jdksUJd||jkrit ||j |j } | |}| ||jd|krz|jd}q ||j ||j |j } t|D]}t||||| |j||j dd||<qt|S) a Preprocess a [list] of audio (Channels x Length) into a batch tensor to be compatable with the encoder. The audio in that list can be of different lengths and channels. in_sr can be an integer or list. If it's an integer it will be assumed it is the input sample_rate for every audio. All audio will be resampled to the model's sample rate. Audio will be silence-padded to the longest length, and further padded to be a multiple of the model's downsampling ratio. If the model is mono, all audio will be converted to mono. The output will be a tensor of shape (Batch x Channels x Length) z:list of sample rates must be the same length of audio_listrrJrr,zAAudio should be shape (Channels x Length) with no batch dimensionre)r target_sr target_lengthtarget_channelsdevice)rfrintrhrsqueeze unsqueezerTResampletorrrr r/rstack) r8 audio_list in_sr_list batch_size new_audio max_lengthrlrr resample_tfpadded_audio_lengthrrrrs:           z2AudioAutoencoder.preprocess_audio_list_for_encoderr_r^cKs|s |j|fi|S|j}|jd}|jd}||9}||9}||} g} td||d| D]} |dddd| | |f} | | q0| ||kr`|dddd| df} | | t| } | jd} ||}t||j|f |j }t| D]j} | | ddf}||}| | dkr|}||jd}n | | |}|||}||d}d}|jd}| dkr||7}||7}| | dkr||8}||8}|dddd||f|dddd||f<q|S)aM Encode audios into latents. Audios should already be preprocesed by preprocess_audio_for_encoder. If chunked is True, split the audio into chunks of a given maximum size chunk_size, with given overlap. Overlap and chunk_size params are both measured in number of latents (not audio samples) # and therefore you likely could use the same values with decode_audio. A overlap of zero will cause discontinuity artefacts. Overlap should be => receptive field size. Every autoencoder will have a different receptive field size, and thus ideal overlap. You can determine it empirically by diffing unchunked vs chunked output and looking at maximum diff. The final chunk may have a longer overlap in order to keep chunk_size consistent for all chunks. Smaller chunk_size uses less memory, but more compute. The chunk_size vs memory tradeoff isn't linear, and possibly depends on the GPU and CUDA version For example, on a A6000 chunk_size 128 is overall faster than 256 and 512 even though it has more chunks r,rrN) rrrrhrrrzerosrirr)r8rchunkedoverlap chunk_sizersamples_per_latent total_sizerhop_sizechunksrlchunk num_chunksy_sizey_finalx_chunky_chunkt_endt_startol chunk_start chunk_endrrr encode_audiosL               2zAudioAutoencoder.encode_audiocKs|s |j|fi|S||}|jd}|jd}g} td||d|D]} |dddd| | |f} | | q%| ||krU|dddd| df} | | t| } | jd} |j} || }t||j|f |j }t| D]j} | | ddf}||}| | dkr|}||jd}n | || }||| }|d| }d}|jd}| dkr||7}||7}| | dkr||8}||8}|dddd||f|dddd||f<qw|S)a Decode latents to audio. If chunked is True, split the latents into chunks of a given maximum size chunk_size, with given overlap, both of which are measured in number of latents. A overlap of zero will cause discontinuity artefacts. Overlap should be => receptive field size. Every autoencoder will have a different receptive field size, and thus ideal overlap. You can determine it empirically by diffing unchunked vs chunked audio and looking at maximum diff. The final chunk may have a longer overlap in order to keep chunk_size consistent for all chunks. Smaller chunk_size uses less memory, but more compute. The chunk_size vs memory tradeoff isn't linear, and possibly depends on the GPU and CUDA version For example, on a A6000 chunk_size 128 is overall faster than 256 and 512 even though it has more chunks r,rrN) rrrhrrrrrr0rr)r8rrrrrrrrrrlrrrrrrrrrrrrrrr decode_audiosH                2zAudioAutoencoder.decode_audio)r,NNNNFr\)F)Fr_r^)rDrErFrrr5rrrrrrrrGrrr;rrs&  , -+  +Frencoder_configc Cs|dd}|dusJd|dkrtdi|d}nQ|dkr@ddlm}|d}tt|dgd |d<|di|}n-|d krP|d}tdi|}n|d krfd d lm}|d}|di|}nt d||dd}|s| D]} d| _ qy|S)NtypezEncoder type must be specifiedoobleckconfigseanetr) SEANetEncoderratios)r,r,r,r,r,dac local_attnr)TransformerEncoder1DzUnknown encoder type requires_gradTFr) r{r]encodec.modulesrlistreversedrplocal_attentionrr$ parametersr) r encoder_typerwrseanet_encoder_config dac_configrlocal_attn_configrparamrrrcreate_encoder_from_config4s8     rdecoder_configc Cs|dd}|dusJd|dkrtdi|d}nA|dkr0ddlm}|di|d}n-|dkr@|d}tdi|}n|d krVd d lm}|d}|di|}ntd ||d d}|so|D]}d|_ qi|S)NrzDecoder type must be specifiedrrrr) SEANetDecoderrrr)TransformerDecoder1DzUnknown decoder type rTFr) r{rmrrrrrr$rr) r decoder_typerrrrrrrrrrcreate_decoder_from_config\s4     rrc Cs|d}t|d}t|d}|dd}|dd}|dus$Jd|dd}|dus2Jd|d d}|dus@Jd |d d}|dusNJd |d d} |dd} |dd} | durit| |} |durqt|}|ddd} t|||||||| | | | d S)Nmodelrwrrriz,latent_dim must be specified in model configrz4downsampling_ratio must be specified in model configrz-io_channels must be specified in model configrz-sample_rate must be specified in model configr/r0rrF) rrirrrrr/r0r)rrr{rrr) r ae_configrwrrrirrrr/r0rrrrrcreate_autoencoder_from_configsB           r)FN)/rrNnumpynprtorch.nnrF torchaudiorralias_free_torchr nn.layersrrtypingr r r rr blocksr rrrfactoryrr pretransformsrrModuler(r*rHrSr]rmrprrstrrrrrrrrs:       !", Q($