o :idN@szddlmZmZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZGdddejZdS) )ListOptionalN)DistributedDataParallel) EulerSolver) TTSZipformer)condition_time_maskget_tokens_index make_pad_mask pad_labelsprepare_avg_tokens_durationsc)seZdZdZgdgdgddddddddd d d d d dd d ddfdeedeedeedededededededededededededed ed!ed"ed#ef(fd$d% Z & &dWd'ejd(ejd)ejd*ejd+e ejd,e ejd-ejfd.d/Z d0eeefd1d2Z d3ejd4ejd5ejfd6d7Z d0eeed5ejfd8d9Z d0eeed5ejd:eeed;ejfdefd?d@Z AdXd0eeedBejd5ejdCejd'ejdDed-ejfdEdFZ & G G H I JdYd0eeed:eeedKejd;ejd5e ejd>edLedMedNed,ed-ejfdOdPZ Q &dZd0eeedBejd5ejdCejdRejdSedTedNed,ejd-ejfdUdVZZS)[ZipVoicezThe ZipVoice model.)rr )rrrrr)rriri 0drfm_decoder_downsampling_factorfm_decoder_num_layersfm_decoder_cnn_module_kernelfm_decoder_feedforward_dimfm_decoder_num_headsfm_decoder_dimtext_encoder_num_layerstext_encoder_feedforward_dimtext_encoder_cnn_module_kerneltext_encoder_num_headstext_encoder_dimtime_embed_dimtext_embed_dimquery_head_dimvalue_head_dim pos_head_dimpos_dimfeat_dim vocab_sizepad_idcstt|d|||||||||||d| d|_t| |d|| | || ||||dd |_||_| |_||_t || |_ t |dd|_ d S) a Initialize the model with specified configuration parameters. Args: fm_decoder_downsampling_factor: List of downsampling factors for each layer in the flow-matching decoder. fm_decoder_num_layers: List of the number of layers for each block in the flow-matching decoder. fm_decoder_cnn_module_kernel: List of kernel sizes for CNN modules in the flow-matching decoder. fm_decoder_feedforward_dim: Dimension of the feedforward network in the flow-matching decoder. fm_decoder_num_heads: Number of attention heads in the flow-matching decoder. fm_decoder_dim: Hidden dimension of the flow-matching decoder. text_encoder_num_layers: Number of layers in the text encoder. text_encoder_feedforward_dim: Dimension of the feedforward network in the text encoder. text_encoder_cnn_module_kernel: Kernel size for the CNN module in the text encoder. text_encoder_num_heads: Number of attention heads in the text encoder. text_encoder_dim: Hidden dimension of the text encoder. time_embed_dim: Dimension of the time embedding. text_embed_dim: Dimension of the text embedding. query_head_dim: Dimension of the query attention head. value_head_dim: Dimension of the value attention head. pos_head_dim: Dimension of the position attention head. pos_dim: Dimension of the positional encoding. feat_dim: Dimension of the acoustic features. vocab_size: Size of the vocabulary. pad_id: ID used for padding tokens. T)in_dimout_dimdownsampling_factornum_encoder_layerscnn_module_kernel encoder_dimfeedforward_dim num_headsr'r)r(r*use_time_embedr%r F) r/r0r1r2r3r4r5r6r'r)r(r*r7forward_fm_decoder) func_nameN) super__init__r fm_decoder text_encoderr+r&r-nn Embeddingembedrsolver)selfrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r- __class__//home/ubuntu/LuxTTS/zipvoice/models/zipvoice.pyr;&sJ 7zZipVoice.__init__Ntxttext_conditionspeech_condition padding_maskguidance_scalereturncCs tj|||gdd}|dvsJ|dkr1|ddkr1|d}|dkr1|ddks|dkr?||jd}|dur{|dkrb|ddkrb|d}|dkrb|ddksP|dkrp||jd}|j||||d}|S|j|||d }|S) aiCompute velocity. Args: t: A tensor of shape (N, 1, 1) or a tensor of a float, in the range of (0, 1). xt: the input of the current timestep, including condition embeddings and noisy acoustic features. text_condition: the text condition embeddings, with the shape (batch, seq_len, emb_dim). speech_condition: the speech condition embeddings, with the shape (batch, seq_len, emb_dim). padding_mask: The mask for padding, True means masked position, with the shape (N, T). guidance_scale: The guidance scale in classifier-free guidance, which is a tensor of shape (N, 1, 1) or a tensor of a float. Returns: predicted velocity, with the shape (batch, seq_len, emb_dim). r)dim)rr.r rN)xrGrKrLrPrGrK)torchcatrNsizesqueezerepeatshaper<)rBrGrHrIrJrKrLvtrErErFr8s&    zZipVoice.forward_fm_decodertokenscCs|t|tr|jnt|j}t||j|d}||}tj dd|Dtj |d}t ||j d}|j |d|d}||fS)a Get the text embeddings. Args: tokens: a list of list of token ids. Returns: embed: the text embeddings, shape (batch, seq_len, emb_dim). tokens_lens: the length of each token sequence, shape (batch,). )r-devicecSg|]}t|qSrElen.0tokenrErErF z/ZipVoice.forward_text_embed..dtyperZr NrQ) isinstanceDDPrZnext parametersr r-r@rRtensorint64r rWr=)rBrYrZ tokens_paddedr@ tokens_lenstokens_padding_maskrErErFforward_text_embeds  zZipVoice.forward_text_embedr@rl features_lensc Csht|}t||d}t||}t|||j}tj|d| d | d|| dd}||fS)aP Get the text condition with the same length of the acoustic feature. Args: embed: the text embeddings, shape (batch, token_seq_len, emb_dim). tokens_lens: the length of each token sequence, shape (batch,). features_lens: the length of each acoustic feature sequence, shape (batch,). Returns: text_condition: the text condition, shape (batch, feature_seq_len, emb_dim). padding_mask: the padding mask of text condition, shape (batch, feature_seq_len). )max_lenr rOr)rNindex) intmaxr r rtorZrRgather unsqueezeexpandrT) rBr@rlro num_framesrKtokens_durations tokens_indexrIrErErFforward_text_conditions     zZipVoice.forward_text_conditioncCs(||\}}||||\}}||fS)zX Process text for training, given text tokens and real feature lengths. )rnr{)rBrYror@rlrIrKrErErFforward_text_trainszZipVoice.forward_text_train prompt_tokensprompt_features_lensc CsDddt||D}||}||\}}||||\}}||fS)zb Process text for inference, given text tokens, real feature lengths and prompts. cSg|]\}}||qSrErEr_ prompt_tokenr`rErErFra z?ZipVoice.forward_text_inference_gt_duration..)ziprnr{) rBrYror}r~r@rlrIrKrErErF"forward_text_inference_gt_durations z+ZipVoice.forward_text_inference_gt_durationspeedcCst|tr|jnt|j}ddt||D}tjdd|Dtj|d}tjdd|Dtj|d}| |\} } |t ||||j tjd} | | | | \} } | | fS)z Process text for inference, given text tokens and prompts, feature lengths are predicted with the ratio of token numbers. cSrrErErrErErFra1rzBZipVoice.forward_text_inference_ratio_duration..cSr[rEr\r^rErErFra6rbrccSr[rEr\r^rErErFra<rb)rd) rerfrZrgrhrrRrirjrnceilrtr{)rBrYr}r~rrZ cat_tokensprompt_tokens_lensrl cat_embedcat_tokens_lensrorIrKrErErF%forward_text_inference_ratio_duration"s2   z.ZipVoice.forward_text_inference_ratio_durationfeaturesnoisecondition_drop_ratiocCs|j||d\}}t|d|dd} t| dd|} |dkr5t|ddd|j|k} || }|||d|} ||} |j || || |d}| |@}t ||| |d }|S) a3Forward pass of the model for training. Args: tokens: a list of list of token ids. features: the acoustic features, with the shape (batch, seq_len, feat_dim). features_lens: the length of each acoustic feature sequence, shape (batch,). noise: the intitial noise, with the shape (batch, seq_len, feat_dim). t: the time step, with the shape (batch, 1, 1). condition_drop_ratio: the ratio of dropped text condition. Returns: fm_loss: the flow-matching loss. rYro)gffffff??r )ro mask_percentrprOrr)rGrHrIrJrKr) r|rrTrRwherervrandrtrZr8mean)rBrYrrorrGrrIrKspeech_condition_maskrJ drop_maskrHutrX loss_maskfm_lossrErErFforwardLs8  zZipVoice.forwardrpredict?prompt_featurest_shiftdurationnum_stepc  Cs|dvsJ|dkr|j||||d\} } n|dusJ|j||||d\} } | j\} }}tjj|ddd||df}t||}t | dt ||}tj | ||d| j d }|jj|| || | | |d }| d|}tj|d||d |j d }tj|d||d |j d }t|dD]2}||||||||f||d||ddf<||d||f||d||ddf<q||||fS) a Generate acoustic features, given text tokens, prompts feature and prompt transcription's text tokens. Args: tokens: a list of list of text tokens. prompt_tokens: a list of list of prompt tokens. prompt_features: the prompt feature with the shape (batch_size, seq_len, feat_dim). prompt_features_lens: the length of each prompt feature, with the shape (batch_size,). features_lens: the length of the predicted eature, with the shape (batch_size,). It is used only when duration is "real". duration: "real" or "predict". If "real", the predicted feature length is given by features_lens. num_step: the number of steps to use in the ODE solver. guidance_scale: the guidance scale for classifier-free guidance. )realrr)rYr}r~rN)rYror}r~rr rO)rZ)rPrIrJrKrrLrr)rrrWrRr> functionalpadrTr rrv zeros_likerandnrZrAsamplesumzerosrsrange)rBrYr}rr~rorrrrrLrIrK batch_sizerx_rJrx0x1x1_wo_prompt_lens x1_prompt x1_wo_promptirErErFrs|       zZipVoice.sampler rt_startt_endc  CsV|j||d\} } t|dd|} |jj|| | | || ||d} | d}| |fS)a8 Generate acoustic features in intermediate timesteps. Args: tokens: List of list of token ids. features: The acoustic features, with the shape (batch, seq_len, feat_dim). features_lens: The length of each acoustic feature sequence, with the shape (batch,). noise: The initial noise, with the shape (batch, seq_len, feat_dim). speech_condition_mask: The mask for speech condition, True means non-condition positions, with the shape (batch, seq_len). t_start: The start timestep. t_end: The end timestep. num_step: The number of steps for sampling. guidance_scale: The scale for classifier-free guidance inference, with the shape (batch, 1, 1). rrOr)rPrIrJrKrrLrr)r|rRrrvrArr)rBrYrrorrrrrrLrIrKrJx_t_end x_t_end_lensrErErFsample_intermediates"  zZipVoice.sample_intermediate)NN)r)Nrrrrr)r N)__name__ __module__ __qualname____doc__rrrr;rRTensorrr8rnr{r|rfloatrrstrrr __classcell__rErErCrFr #sj     g 4   '        1  >      m    r )typingrrrRtorch.nnr>torch.nn.parallelrrfzipvoice.models.modules.solverr!zipvoice.models.modules.zipformerrzipvoice.utils.commonrrr r r Moduler rErErErFs