o oiL+@sdZddlZddlmZddlmZddlmZddlm Z m Z ddl Z ddl Z ddlmZeGd d d ZGd d d ZGd ddZdS)z0Core components for Kani-TTS-2 audio generation.N)AudioCodecModel) AutoTokenizer) dataclass)OptionalTuple)KaniTTS2ForCausalLMc@seZdZUdZdZeed<dZeed<dZ eed<dZ eed <d Z eed <d Z eed <dZ dZeeed<dZeeed<dZeeed<dZeeed<dZeeed<dZeeed<dZeeed<dS) TTSConfigzConfiguration for TTS model.auto device_mapitokeniser_lengthr start_of_text end_of_texti max_new_tokensz,nvidia/nemo-nano-codec-22khz-0.6kbps-12.5fpsnanocodec_modeli"VNtext_vocab_sizetokens_per_frame audio_stepuse_learnable_rope alpha_min alpha_maxspeaker_emb_dim)__name__ __module__ __qualname____doc__r str__annotations__r intr rrr sample_raterrrrfloatrboolrrrr#r#A/home/ubuntu/.local/lib/python3.10/site-packages/kani_tts/core.pyr s       r c@seZdZdZddedeeddfddZdej ddfd d Z dej de ej ej ffd d Z dej defd dZ dej de ejeeffddZdS)NemoAudioPlayerz1Handles audio codec operations using NVIDIA NeMo.Nconfigtext_tokenizer_namereturncCs||_t|jj|_tjrdnd|_ |j |j ||_ |j r+t |j |_ |jj|_|jj|_|jj|_|jd|_|jd|_|jd|_|jd|_|jd|_|jd|_|jd |_|jd |_d |_dS) Ncudacpurr i)confrfrom_pretrainedrevalnemo_codec_modeltorchr) is_availabledevicetor'r tokenizerr r rstart_of_speech end_of_speechstart_of_human end_of_human start_of_ai end_of_ai pad_tokenaudio_tokens_start codebook_size)selfr&r'r#r#r$__init__*s*            zNemoAudioPlayer.__init__out_idscCs(|j|v}|j|v}|r|stddS)z6Validate that output contains required speech markers.z Special speech tokens not exist!N)r:r; ValueError)rCrEstart_of_speech_flagend_of_speech_flagr#r#r$output_validationAs  z!NemoAudioPlayer.output_validationcs|jkjddd}|jkjddd}||kr"td||d|}t|dr4td|dd}|tfd d t dD}|j }|dk dkr]td |j d}t|jdg}||fS) z2Extract and process audio codes from model output.Tas_tuplerzInvalid audio codes sequence!rr,z3The length of the sequence must be a multiple of 4!csg|]}j|qSr#)rB).0irCr#r$ Ssz2NemoAudioPlayer.get_nano_codes..zInvalid audio tokens!)r:nonzeroitemr;rFlenreshaper5tensorrangerAsumT unsqueezeshape)rCrE start_a_idx end_a_idx audio_codeslen_r#rOr$get_nano_codesHs     zNemoAudioPlayer.get_nano_codescCsX||jkjddd}||jkjddd}|||d}|jj|dd}|S)z!Extract text from token sequence.TrJrr)skip_special_tokens)r rQrRrr9decode)rCrE start_t_idx end_t_idx txt_tokenstextr#r#r$get_text\s zNemoAudioPlayer.get_textcCs|}||||\}}||j||j}}t|jj||d\}}| }Wdn1s@wY|j rQ||}||fS|dfS)z.Convert model output tokens to audio waveform.)tokens tokens_lenN)flattenrIr_r8r7r5inference_moder4rar*detachnumpysqueezer'rf)rCrEr]r^reconstructed_audio_ output_audiorer#r#r$ get_waveformds   zNemoAudioPlayer.get_waveformN)rrrrr rrrDr5TensorrIrr_rfnpndarrayrqr#r#r#r$r%'s &r%c@seZdZdZdedededdfddZdd ed edee j e j ffd d Z   dde j de j de e j de de de de j fddZ    dded ede e j de de de deejeffddZdS) KaniModelz1Text-to-speech model using causal language model.r& model_nameplayerr(Nc Cs||_||_tjr dnd|_tj||jj|jj |jj |jj |jj |jj |jjtj|jjd |_t||_t|jjdd|_d|_g|_|jdur\|jd|_|jdg|_dSdS)Nr)r*) rArrrrrr torch_dtyper language_settingsno_language_tagsstatuslanguage_tags_list)r1rxr5r)r6r7rr2rArrrrrrbfloat16r modelrr9getattrr&rzr|r}get)rCr&rwrxr#r#r$rDxs.  zKaniModel.__init__ text_prompt language_tagc Cs|dur |d|}|jj}|jj}|jj}|j|ddj}tj|ggtj d}tj||ggtj d}tj |||gdd} tj d| j dtj d} | | fS)z*Prepare input tokens with special markers.Nz: pt)return_tensors)dtyper)dim) striprxr<rr=r9 input_idsr5rUint64catonesrZ) rCrrSTART_OF_HUMAN END_OF_TEXT END_OF_HUMANr start_token end_tokensmodified_input_idsattention_maskr#r#r$ get_input_idsszKaniModel.get_input_ids?ffffff?皙?rr speaker_emb temperaturetop_prepetition_penaltyc Cs||j}||j}|||jjd|||d|jjd }|dur*||j}||d<t|jj di|}Wdn1sBwY|dS)a Generate audio tokens from text tokens. Args: input_ids: Text token IDs attention_mask: Attention mask speaker_emb: Optional speaker embedding [batch_size, speaker_emb_dim] temperature: Sampling temperature (default: 1.0) top_p: Top-p sampling parameter (default: 0.95) repetition_penalty: Repetition penalty (default: 1.1) Tr) rrr do_samplerrrnum_return_sequences eos_token_idNrr*r#) r8r7r1rrxr;r5no_gradrgenerate) rCrrrrrr gen_kwargs generated_idsr#r#r$ model_requests&     zKaniModel.model_requestrec Cs|jdkr"|dur"tdtdtdt|jdditdn|jdkr7|dur7tdtd td|||\}}|j||||||d } |j| \} } | |fS) a Generate audio from text. Args: text: Input text to convert to speech language_tag: Optional language ID (for models that support language or accent tags) speaker_emb: Optional speaker embedding tensor [1, speaker_emb_dim] temperature: Sampling temperature (default: 1.0) top_p: Top-p sampling parameter (default: 0.95) repetition_penalty: Repetition penalty (default: 1.1) available_language_tagsNz(========================================z+!!! YOU NEED TO SELECT THE LANGUAGE TAG !!!zLanguages available:sep r{z:!!! This model does not support language tag selection !!!)rrrr)r|printr}rrrxrq) rCrerrrrrrr model_outputaudioror#r#r$ run_models& zKaniModel.run_modelrr)Nrrr)NNrrr)rrrrr rr%rDrr5rsrrr!rrtrurr#r#r#r$rvusN$  , rv)rr5nemo.collections.tts.modelsr transformersr dataclassesrtypingrrrlrtosrrr r%rvr#r#r#r$s    N