o mi@s^dZddlmZmZddlmZddlmZddlm Z ddl Z ddl Z Gdddej ZdS) z7 Created on Wed Aug 30 15:47:55 2023 @author: zhangxin ) SEANetEncoder SEANetDecoder)ResidualVectorQuantizerN) rearrangecseZdZfddZededefddZddgfd ejd e d e fd d Z dd ejd e fddZ  dd ejd e de fddZ ddejde fddZZS)SpeechTokenizerc s2tt|d|d|d|d|d|d|d|d|d d |_|d |_|d |_t|d|_ |d|d kr[t |d|d |_ nt |_ t|d|d |dd|_t|d|d|d|dd|d|d|d|d d |_dS)zi Parameters ---------- config : json Model Config. n_filters dimensionstrides lstm_layers bidirectional dilation_baseresidual_kernel_sizen_residual_layers activation) rr ratioslstmr r rrr sample_raten_qsemantic_dimension codebook_size)r rbinsFN)super__init__rgetencoderrrnpproddownsample_ratennLinear transformIdentityr quantizerrdecoder)selfconfig __class__5/home/ubuntu/SpeechTokenizer/speechtokenizer/model.pyrs8   "  zSpeechTokenizer.__init__ config_path ckpt_pathcCs^ddl}t| }||}Wdn1swY||}tj|dd}|||S)a Parameters ---------- config_path : str Path of model configuration file. ckpt_path : str Path of model checkpoint. Returns ------- model : SpeechTokenizer SpeechTokenizer model. rNcpu) map_location)jsonopenloadtorchload_state_dict)clsr+r,r/fcfgmodelparamsr)r)r*load_from_checkpoint4s   z$SpeechTokenizer.load_from_checkpointNrxrlayersc Cs\|r|n|j}||}|j|||d\}}}}t|dd} || } ||} | || fS)a Parameters ---------- x : torch.tensor Input wavs. Shape: (batch, channels, timesteps). n_q : int, optional Number of quantizers in RVQ used to encode. The default is all layers. layers : list[int], optional Layers of RVQ should return quantized result. The default is the first layer. Returns ------- o : torch.tensor Output wavs. Shape: (batch, channels, timesteps). commit_loss : torch.tensor Commitment loss from residual vector quantizers. feature : torch.tensor Output of RVQ's first layer. Shape: (batch, timesteps, dimension) )rr;rzb d t -> b t d)rrr#rr!r$) r%r:rr;e quantizedcodes commit_lossquantized_listfeatureor)r)r*forwardPs    zSpeechTokenizer.forwardcCs:||}|r |ntt|j}|j||d\}}}}|S)a Parameters ---------- x : torch.tensor Input wavs. Shape should be (batch, channels, timesteps). layers : list[int], optional Layers of RVQ should return quantized result. The default is all layers. Returns ------- quantized_list : list[torch.tensor] Quantized of required layers. )r;)rlistrangerr#)r%r:r;r<r=r>r?r@r)r)r*forward_featureqs zSpeechTokenizer.forward_featurestcCs:||}|dur d}|r|n|j}|jj|||d}|S)a Parameters ---------- x : torch.tensor Input wavs. Shape: (batch, channels, timesteps). n_q : int, optional Number of quantizers in RVQ used to encode. The default is all layers. st : int, optional Start quantizer index in RVQ. The default is 0. Returns ------- codes : torch.tensor Output indices for each quantizer. Shape: (n_q, batch, timesteps) Nr)rrG)rrr#encode)r%r:rrGr<r>r)r)r*rHs zSpeechTokenizer.encoder>cCs|jj||d}||}|S)at Parameters ---------- codes : torch.tensor Indices for each quantizer. Shape: (n_q, batch, timesteps). st : int, optional Start quantizer index in RVQ. The default is 0. Returns ------- o : torch.tensor Reconstruct wavs from codes. Shape: (batch, channels, timesteps) )rG)r#decoder$)r%r>rGr=rBr)r)r*rIs zSpeechTokenizer.decode)N)NN)r)__name__ __module__ __qualname__r classmethodstrr9r2tensorintrDrCrFrHrI __classcell__r)r)r'r*rsL %  #  r)__doc__modules.seanetrr quantizationrtorch.nnreinopsrr2numpyrModulerr)r)r)r*s