o @Tib@sddlZddlmZddlZddlZddlmZddlmmZ ddl m Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZmZeZeGd d d ZeGd ddZGdddejZdS)N) dataclass)FiniteScalarQuantizer) GlobalEncoder)PostNet)SSLFeatureExtractor) Transformer)wav2vec2_model)freeze_modules get_loggerc@seZdZUdZeedfed<dZeedfed<dZe ed<dZ eed <d Z eed <dZ e ed <d Z eed<d Zeed<dZeed<dZeed<dZeed<dZeed<dZeed<dS)LinaCodecConfig) .local_ssl_layers)rglobal_ssl_layersTnormalize_ssl_featuresrdownsample_factormel_upsample_factoruse_conv_downsamplelinearlocal_interpolation_modemel_interpolation_modei] sample_ratein_fft hop_lengthdn_melscenterpaddingN)__name__ __module__ __qualname__rtupleint__annotations__rrboolrrrrstrrrrrrr!r*r*C/home/ubuntu/.local/lib/python3.10/site-packages/linacodec/model.pyr s           r c@sDeZdZUdZejdBed<dZejdBed<dZejdBed<dS)LinaCodecFeaturesNcontent_embeddingcontent_token_indicesglobal_embedding) r"r#r$r-torchTensorr'r.r/r*r*r*r+r,*s r,cseZdZdZdedededededBded ed ed e ffd d Z de fddZ dedefddZ dededededBfddZdefddZded ed ed e fddZdYdededefddZdedefd d!Zdedefd"d#Zd$eejd%eedejfd&d'ZdZd$ejd)edejfd*d+Z d[d,ejd-edBdeejejffd.d/Zd0ejdeejejejejfdBfd1d2Zd3ejdejfd4d5Zd6ejd7ejd8edejfd9d:Z d;ee de!e ejffde ddfd?d@Z$e#    d\dAe dBdBe dBd>e dBdCe dBddDf dEdFZ%e&d]d,ejdHedIede'fdJdKZ(dLejdejfdMdNZ)e&   d^dOejdPejdBdQejdBdRedBdejf dSdTZ*e&dUejdVejdejfdWdXZ+Z,S)_LinaCodecModelz?Model architecture and forward pass logic for Kanade tokenizer.configssl_feature_extractor local_encoderlocal_quantizerfeature_decoderNglobal_encoder mel_prenet mel_decoder mel_postnetc sJt||_||||||||||||||| dSN)super__init__r3_init_ssl_extractor_init_local_branch_init_global_branch_init_mel_decoder) selfr3r4r5r6r7r8r9r:r; __class__r*r+r>4s   zLinaCodecModel.__init__pathcCsFt|}tdi|d}|j|ddd}||_ddg|_dS) z8Loads distilled wavlm model, 970m params --> 250m paramsr3 state_dictFstrictr Nr*)r0loadr load_state_dictcuda wavlm_modeldistilled_layers)rCrFckptrNresultr*r*r+load_distilled_wavlmGs  z#LinaCodecModel.load_distilled_wavlmcCs||_t|jgtd|jjt|j|_t|jdkr0tdt|jd|jn td|jdd|jrDtdt|j |_ t|j dkrbtdt|j d |j d Std|j dd d S) z/Initialize and configure SSL feature extractor.z;SSL feature extractor initialized and frozen, feature dim: rzUsing average of z SSL layers for local branch: zUsing single SSL layer rz for local branchz.Normalizing local SSL features before encodingz SSL layers for global branch: z for global branchN) r4r loggerdebug feature_dimlistrlenrr)rCr3r4r*r*r+r?Os&     z"LinaCodecModel._init_ssl_extractorcCs||_||_||_|j|_|jdkrQtd|j|jrD|j}tj |||j|jd|_ tj |||j|jd|_ td|jdSd|_ d|_ tddSd|_ d|_ dS)zOInitialize local branch components (encoder, downsampling, quantizer, decoder).rz(Using temporal downsampling with factor  kernel_sizestridez6Using Conv1d downsampling/upsampling with kernel size NzJUsing average pooling and linear interpolation for downsampling/upsampling) r5r6r7rrSrTr output_dimnnConv1dconv_downsampleConvTranspose1d conv_upsample)rCr3r5r6r7rUr*r*r+r@ls(    z!LinaCodecModel._init_local_branchcCs ||_dS)z$Initialize global branch components.Nr8)rCr8r*r*r+rAs z"LinaCodecModel._init_global_branchcCsZ||_||_||_d|_|jdkr+|j}tj|||j|jd|_t d|jdSdS)zIInitialize mel decoder components (prenet, upsampling, decoder, postnet).NrrXz5Using Conv1DTranspose for mel upsampling with factor ) r9r:r;mel_conv_upsamplerr[r\r_rSrT)rCr3r9r:r; input_dimr*r*r+rBs  z LinaCodecModel._init_mel_decoderF audio_lengthensure_recon_lengthreturnc Cs~|j}|jj}|||j}t||j}|r(||j}dkr(||j|7}||}||j|} t| |d} | S)zWCalculate required padding for input waveform to ensure consistent SSL feature lengths.rr) r4r3rssl_sample_ratemathceilhop_sizerget_minimum_input_length) rCrdre extractorrnum_samples_after_resamplingexpected_ssl_output_length remainder%num_samples_required_after_resamplingnum_samples_requiredr!r*r*r+_calculate_waveform_paddings z*LinaCodecModel._calculate_waveform_padding token_lengthcCs:|j}|jj}||j}||}||j|}t|S)z:Calculate the original audio length based on token length.)r4r3rrrkrgrhri)rCrsrlrfeature_lengthrprqr*r*r+ _calculate_original_audio_lengths    z/LinaCodecModel._calculate_original_audio_lengthcCsL|jjdkr||jjdS|jjdkr||jjS||jj|jjdS)zBCalculate the target mel spectrogram length based on audio length.r rsame)r3r!rr)rCrdr*r*r+_calculate_target_mel_lengths   z+LinaCodecModel._calculate_target_mel_lengthfeatureslayerscsLt|dkrfdd|D}tj|ddjdd}|S|dd}|S)Nrcsg|]}|dqS)rr*).0irxr*r+ sz8LinaCodecModel._process_ssl_features..r)dim)rWr0stackmean)rCrxryselected_featuresmixed_featuresr*r|r+_process_ssl_featuress z$LinaCodecModel._process_ssl_features:0yE>epscCs<|jjs|Stj|ddd}tj|ddd}||||S)NrT)r~keepdim)r3rr0rstd)rCrxrrrr*r*r+_normalize_ssl_featuress z&LinaCodecModel._normalize_ssl_featureswaveformr!cCs|dkr |d}|dkrtj|||fdd}t"|j|dd}|j|}|jj |t |j dd}Wdn1sBwY| ||j }| |}| ||j}||fS) aForward pass to extract SSL features. (B, T, C) Args: waveform: Input waveform tensor of shape (B, channels, samples) padding: Optional padding to apply on both sides of the waveform. This is useful to ensure that the SSL feature extractor produces consistent output lengths. Returns: local_ssl_features: Local SSL features for local branch. (B, T, C) global_ssl_features: Global SSL features for global branch. (B, T, C) rrconstant)moder) num_layersN)r~squeezeFpadr0no_gradr4 resamplerrNextract_featuresmaxrOrrr)rCrr!acoustic_wavlm_featuresdistilled_wavlm_featureslocal_ssl_featuresglobal_ssl_featuresr*r*r+forward_ssl_featuress    z#LinaCodecModel.forward_ssl_featuresrc Cs||}|jdkr.|jjr||dddd}ntj|dd|j|jddd}d}t d}|j dur| |\}}|d}t |d}|}|jdkrz|jjre| |dddd}ntj|dd|jd|jjddd}| |}n|j |\}}||||fS) aForward pass to extract content embeddings from the local branch. Args: local_ssl_features: Local SSL features tensor of shape (B, T, C) Returns: local_quantized: Quantized local embeddings. (B, T/factor, C) indices: Content token indices. (B, T/factor) ssl_recon: Reconstructed SSL features (if feature decoder is present). (B, T, C) perplexity: Quantizer perplexity (if feature decoder is present). Scalar tensor. rrrXNgindices perplexitysizer)r5rr3rr^ transposer avg_pool1dr0tensorr7r6rr` interpolateshaperencode) rCr local_encoded ssl_reconrlocal_quantizedlocal_quantize_inforlocal_latent_for_sslr*r*r+forward_contents:       zLinaCodecModel.forward_contentrcCs||}|S)zForward pass to extract global embeddings from the global branch. Args: global_ssl_features: Global SSL features tensor of shape (B, T, C) Returns: global_encoded: Global embeddings. (B, C) ra)rCrglobal_encodedr*r*r+forward_global1s zLinaCodecModel.forward_globalcontent_embeddingsglobal_embeddings mel_lengthcCs||}|jdur||dddd}tj|dd||jjddd}|j||dd}|dd}| |}|S)aForward pass to generate mel spectrogram from content and global embeddings. Args: content_embeddings: Content embeddings tensor of shape (B, T, C) global_embeddings: Global embeddings tensor of shape (B, C) mel_length: Target mel spectrogram length (T_mel) Returns: mel_recon: Reconstructed mel spectrogram tensor of shape (B, n_mels, T_mel) Nrrr) condition) r9rbrrrr3rr: unsqueezer;)rCrrr local_latent mel_reconr*r*r+ forward_mel;s   zLinaCodecModel.forward_melinclude_modulescs,fdddDfdd|D}|S)zPGet model weights for saving. Excludes certain modules not needed for inference.csg|]}|vr|qSr*r*)rzm)rr*r+r}\s z2LinaCodecModel.weights_to_save..)r4r7r`cs,i|]\}tfddDs|qS)c3s|]}|VqdSr<) startswith)rzexclnamer*r+ dsz..)any)rzparam)excluded_modulesrr+ as z2LinaCodecModel.weights_to_save..)named_parameters)rCrrGr*)rrr+weights_to_saveZs  zLinaCodecModel.weights_to_save config_pathcCs4tjdd}|jdtd||}||}|jS)zInstantiate KanadeModel from config file. Args: config_path (str): Path to model configuration file (.yaml). Returns: KanadeModel: Instantiated KanadeModel. F) exit_on_errorz--model)type) jsonargparseArgumentParser add_argumentr2 parse_pathinstantiate_classesmodel)clsrparsercfgr*r*r+ from_hparamshs   zLinaCodecModel.from_hparamsrepo_idrevision weights_path KanadeModelc Cs|durddlm}||d|d}||d|d}n |dus!|dur%td||}ddlm}||d d }|j|d d td ||S)atLoad LinaCodec either from HuggingFace Hub or local config and weights files. Args: repo_id (str, optional): HuggingFace Hub repository ID. If provided, loads config and weights from the hub. revision (str, optional): Revision (branch, tag, commit) for the HuggingFace Hub repo. config_path (str, optional): Path to model configuration file (.yaml). Required if repo_id is not provided. weights_path (str, optional): Path to model weights file (.safetensors). Required if repo_id is not provided. Returns: LinaCodec: Loaded LinaCodec instance. Nr)hf_hub_downloadz config.yaml)rzmodel.safetensorszePlease provide either HuggingFace Hub repo_id or both config_path and weights_path for model loading.) load_filecpu)deviceFrHz&Loaded weights from safetensors file: ) huggingface_hubr ValueErrorrsafetensors.torchrrLrSinfo) rrrrrrrrrGr*r*r+from_pretrainedvs    zLinaCodecModel.from_pretrainedTreturn_content return_globalc Cs|d}||}|j|d|d\}}t}tjdtjdd5|r:||\} } } } | d|_ | d|_ |rO| |} | d|_ Wd|SWd|S1sZwY|S)aExtract content and/or global features from audio using Kanade model. Args: waveform (torch.Tensor): Input audio waveform tensor (samples,). The sample rate should match model config. return_content (bool): Whether to extract content features. return_global (bool): Whether to extract global features. Returns: dict[str, torch.Tensor]: Extracted features. r)r!rMT device_typedtypeenabledN)rrrrrr,r0autocastbfloat16rrr-r.rr/) rCrrrrdr!rrrQr- token_indices_r/r*r*r+rs&         zLinaCodecModel.encodercCs|j|}|S)zXGet content embeddings from content token indices. (..., seq_len) -> (..., seq_len, dim))r6decode)rCrr-r*r*r+decode_token_indicess z#LinaCodecModel.decode_token_indicesr/r.r-target_audio_lengthcCs|dur|dur td||}|dur|d}||}tjdtjdd||}|d}|d}|j |||d}Wdn1sJwY| dS)a)Synthesize audio from content and global features using LinaCodec model and Vocos. Args: global_embedding (torch.Tensor): Global embedding tensor (dim,). content_token_indices (torch.Tensor, optional): Optional content token indices tensor (seq_len). content_embedding (torch.Tensor, optional): Optional content embedding tensor (seq_len, dim). If both content_token_indices and content_embedding are provided, content_embedding takes precedence. target_audio_length (int, optional): Target length of the output audio in samples. If None, uses the original audio length estimated from the sequence length of content tokens. Returns: torch.Tensor: Generated mel spectrogram tensor (n_mels, T). NzCEither content_token_indices or content_embedding must be provided.rrMTr)r) rrrrur0rrrwrrr)rCr/r.r-rseq_lenrmel_spectrogramr*r*r+rs       zLinaCodecModel.decodesource_waveformreference_waveformcCs>|j|ddd}|j|ddd}|j|j|j|dd}|S)aConvert voice using LinaCodec model and Vocos, keeping content from source and global characteristics from reference. Only supports single audio input. Just a convenient wrapper around encode and decode methods. Args: source_waveform (torch.Tensor): Source audio waveform tensor (samples,). reference_waveform (torch.Tensor): Reference audio waveform tensor (samples_ref,). Returns: torch.Tensor: Converted mel spectrogram tensor (n_mels, T). TF)rrr)r-r/r)rrr-r/r)rCrrsource_featuresreference_featuresrr*r*r+voice_conversions zLinaCodecModel.voice_conversion)F)rr<)NNNN)TT)NNN)-r"r#r$__doc__r rrrrrr>r)rRr?r@rArBr&r(rrrurwrVr0r1rfloatrr%rrrrdictr classmethodrrinference_moder,rrrr __classcell__r*r*rDr+r21s   "  "      4    )  %&r2)rh dataclassesrrr0torch.nnr\torch.nn.functional functionalr module.fsqrmodule.global_encoderrmodule.postnetrmodule.ssl_extractorrmodule.transformerrmodule.distill_wavlmr utilr r rSr r,Moduler2r*r*r*r+s&