o ²i@ ã@sLddlZddlmZddlmZddlmZmZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZGd d „d ejƒZedkr¤e dƒZejddZdZe dde edƒ¡Z!e de edƒd¡Z"e"e!e!dœZ#ee#ƒZ$e %e#¡\Z&Z'e (e&e'¡Z)e *e$d +¡e)¡rže,dƒdSe,dƒdSdS)éN)ÚPath)ÚDictÚAny)Ú DictConfig)Ú load_file)Ú load_config)ÚSpeakerEncoder)ÚEncoder)ÚDecoder)Ú WaveGenerator)ÚFactorizedVectorQuantizecsÚeZdZdZdeeefdejdejdejdejdejdejd d f‡fd d „ Z e d e d dfdd„ƒZ deeefd eeeffdd„Z e ¡deeeffdd„ƒZe ¡dd„ƒZdeeeffdd„Zdd„Z‡ZS)ÚBiCodeczŠ BiCodec model for speech synthesis, incorporating a speaker encoder, feature encoder/decoder, quantizer, and wave generator. Ú mel_paramsÚencoderÚdecoderÚ quantizerÚspeaker_encoderÚprenetÚpostnetÚreturnNc  s<tƒ ¡||_||_||_||_||_||_| |¡dS)aå Initializes the BiCodec model with the required components. Args: mel_params (dict): Parameters for the mel-spectrogram transformer. encoder (nn.Module): Encoder module. decoder (nn.Module): Decoder module. quantizer (nn.Module): Quantizer module. speaker_encoder (nn.Module): Speaker encoder module. prenet (nn.Module): Prenet network. postnet (nn.Module): Postnet network. N) ÚsuperÚ__init__rrrrrrÚinit_mel_transformer) ÚselfrrrrrrrÚkwargs©Ú __class__©úD/home/ubuntu/veenaModal/external/sparktts/sparktts/models/bicodec.pyr%s zBiCodec.__init__Ú model_dirc Ks|›d}t|›dƒd}|d}tdi|d¤Ž}tdi|d¤Ž}tdi|d¤Ž}tdi|d¤Ž} tdi|d ¤Ž} tdi|d ¤Ž} |||| || || d } t|ƒ} | j| d d \}}|D] }td|›ƒqb|D] }td|›ƒqn|   ¡|   ¡| S)zí Loads the model from a checkpoint. Args: model_dir (Path): Path to the model directory containing checkpoint and config. Returns: BiCodec: The initialized BiCodec model. z/model.safetensorsz /config.yamlÚaudio_tokenizerrrrrrrr)rrrrrrrF)ÚstrictzMissing tensor: zUnexpected tensor: Nr) rr r r r rrÚload_state_dictÚprintÚevalÚremove_weight_norm)ÚclsrrÚ ckpt_pathÚconfigrrrrrrrÚmodelÚ state_dictÚ missing_keysÚunexpected_keysÚkeyrrrÚload_from_checkpointEs6 ù zBiCodec.load_from_checkpointÚbatchc Cs¶|d}| |d¡ d¡}| | dd¡¡}| |¡}| | dd¡¡\}}|}d} | |d|¡} | | ¡} | | d¡} |  | ¡} |d|d |d | | |||d  d¡| d œ S) a% Performs a forward pass through the model. Args: batch (dict): A dictionary containing features, reference waveform, and target waveform. Returns: dict: A dictionary containing the reconstruction, features, and other metrics. ÚfeatÚref_wavééFÚz_qéÿÿÿÿÚvq_lossÚ perplexityÚ active_numÚwav) r6r7Ú cluster_sizeÚreconsÚ pred_featÚx_vectorÚd_vectorÚaudiosÚwith_speaker_loss) Úmel_transformerÚsqueezerÚ transposerrrrÚ unsqueezer) rr/r0ÚmelÚzÚ vq_outputsr=r>Ú conditionsr@Úxr<Ú wav_reconrrrÚforwardqs*     ÷zBiCodec.forwardcCsV|d}| |d¡ d¡}| | dd¡¡}|j |¡}|j | dd¡¡}||fS)zî Tokenizes the input audio into semantic and global tokens. Args: batch (dict): The input audio features and reference waveform. Returns: tuple: Semantic tokens and global tokens. r0r1r2r3)rArBrrCrÚtokenizer)rr/r0rErFÚsemantic_tokensÚ global_tokensrrrrL—s   zBiCodec.tokenizecCs@|j |¡}|j |¡}| ||¡}|| d¡}| |¡}|S)a Detokenizes the semantic and global tokens into a waveform. Args: semantic_tokens (tensor): Semantic tokens. global_tokens (tensor): Global tokens. Returns: tensor: Reconstructed waveform. r5)rÚ detokenizerrrDr)rrMrNr4r>rIrJrrrrO«s   zBiCodec.detokenizer(c CsLddlm}|j|d|d|d|d|d|d|d d d d d |_dS) z¸ Initializes the MelSpectrogram transformer based on the provided configuration. Args: config (dict): Configuration parameters for MelSpectrogram. rNÚ sample_rateÚn_fftÚ win_lengthÚ hop_lengthÚmel_fminÚmel_fmaxÚnum_melsr2Úslaney)Ún_melsÚpowerÚnormÚ mel_scale)Útorchaudio.transformsÚ transformsÚMelSpectrogramrA)rr(ÚTTrrrr¿s  özBiCodec.init_mel_transformercCsdd„}| |¡dS)z-Removes weight normalization from all layers.cSs*z tjj |¡WdStyYdSw)N)ÚtorchÚnnÚutilsr%Ú ValueError)ÚmrrrÚ_remove_weight_norm×s  ÿz7BiCodec.remove_weight_norm.._remove_weight_normN)Úapply)rrerrrr%ÕszBiCodec.remove_weight_norm)Ú__name__Ú __module__Ú __qualname__Ú__doc__rÚstrrraÚModulerÚ classmethodrr.rKr`Úno_gradrLrOrr%Ú __classcell__rrrrr s8 þýüûúùø ö "+& r Ú__main__z3pretrained_models/SparkTTS-0.5B/BiCodec/config.yamlz'pretrained_models/SparkTTS-0.5B/BiCodec)rg¸…ëQ¸î?ér2i€>é2i)r0r9r1r;zTest successfulz Test failed)-r`Útorch.nnraÚpathlibrÚtypingrrÚ omegaconfrÚsafetensors.torchrÚsparktts.utils.filerÚ(sparktts.modules.speaker.speaker_encoderrÚ-sparktts.modules.encoder_decoder.feat_encoderr Ú-sparktts.modules.encoder_decoder.feat_decoderr Ú/sparktts.modules.encoder_decoder.wave_generatorr Ú.sparktts.modules.vq.factorized_vector_quantizer rlr rgr(r.r)ÚdurationÚrandnÚintrIr0ÚinputsÚoutputsrLrMrNrOrJÚallcloseÚdetachr#rrrrÚs<          Cÿ    ê