o ۷i@sddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlZddlZddlZddlZddlmZmZmZddlmZmZmZe eeje ejeffZ e ee efZ!eGdd d Z"Gd d d Z#dS) N) dataclass)AnyDictListOptionalTupleUnion)urlparse) AutoConfig AutoModel AutoProcessor)Qwen3TTSConfig Qwen3TTSForConditionalGenerationQwen3TTSProcessorc@sJeZdZUdZeejed<ejed<eed<eed<dZ ee ed<dS)VoiceClonePromptItemz Container for one sample's voice-clone prompt information that can be fed to the model. Fields are aligned with `Qwen3TTSForConditionalGeneration.generate(..., voice_clone_prompt=...)`. ref_coderef_spk_embeddingx_vector_only_modeicl_modeNref_text) __name__ __module__ __qualname____doc__rtorchTensor__annotations__boolrstrr r X/home/ubuntu/vllm_env/lib/python3.10/site-packages/qwen_tts/inference/qwen3_tts_model.pyr(s  rc@seZdZdZdTdedeeeeffddZ e deddfd d Z dee fd d Z dee fd dZdeeddfddZdeeeddfddZdedefddZdedefddZdedefddZdedeejeffddZd eeeefdeeejeffd!d"Zdedeefd#d$Z d%edefd&d'Z!d%edefd(d)Z"d*edefd+d,Z#d-eedee$j%fd.d/Z&          dUd0eed1eed2ee'd3ee'd4ee'd5eed6eed7ee'd8ee'd9eedeeeffd:d;Z(e$)  eeeeeefd?eeeefdee*fd@dAZ+dBee*deeeffdCdDZ,e$-    <  eeeeeefd?eeeefdFeeeeefee*fdGedeeejeffdHdIZ.e$-  JdXd%eeeefd*eeeefdEeeeefdGedeeejeff dKdLZ/e$-   JdYd%eeeefdMeeeefdEeeeefd*eeeeefdGedeeejeff dNdOZ0deeefdPdQZ1deeefdRdSZ2dS)Z Qwen3TTSModela A HuggingFace-style wrapper for Qwen3 TTS models (CustomVoice/VoiceDesign/Base) that provides: - from_pretrained() initialization via AutoModel/AutoProcessor - generation APIs for: * CustomVoice: generate_custom_voice() * VoiceDesign: generate_voice_design() * Base: generate_voice_clone() + create_voice_clone_prompt() - consistent output: (wavs: List[np.ndarray], sample_rate: int) Notes: - This wrapper expects the underlying model class to be `Qwen3TTSForConditionalGeneration` - Language / speaker validation is done via model methods: model.get_supported_languages(), model.get_supported_speakers() Nmodelgenerate_defaultscCsj||_||_|p i|_t|dd|_|jdur3z t|j|_WdSty2td|_YdSwdS)Ndevicecpu) r# processorr$getattrr%next parameters StopIterationr)selfr#r'r$r r r!__init__Fs   zQwen3TTSModel.__init__pretrained_model_name_or_pathreturncKsvtdttttttttj|fi|}t|ts*t dt |dtj|dd}|j }||||dS)a Load a Qwen3 TTS model and its processor in HuggingFace `from_pretrained` style. This method: 1) Loads config via AutoConfig (so your side can register model_type -> config/model). 2) Loads the model via AutoModel.from_pretrained(...), forwarding `kwargs` unchanged. 3) Loads the processor via AutoProcessor.from_pretrained(model_path). 4) Loads optional `generate_config.json` from the model directory/repo snapshot if present. Args: pretrained_model_name_or_path (str): HuggingFace repo id or local directory of the model. **kwargs: Forwarded as-is into `AutoModel.from_pretrained(...)`. Typical examples: device_map="cuda:0", dtype=torch.bfloat16, attn_implementation="flash_attention_2". Returns: Qwen3TTSModel: Wrapper instance containing `model`, `processor`, and generation defaults. qwen3_ttszAutoModel returned z-, expected Qwen3TTSForConditionalGeneration. T)fix_mistral_regex)r#r'r$) r registerrr rr rfrom_pretrained isinstance TypeErrortypegenerate_config)clsr.kwargsr#r'r$r r r!r3Rs    zQwen3TTSModel.from_pretrainedcC>t|jdd}t|r|}|durdStdd|DSdS)Nget_supported_languagescSg|]}t|qSr rlower.0xr r r! z:Qwen3TTSModel._supported_languages_set..r(r#callableset)r,langsvr r r!_supported_languages_set{z&Qwen3TTSModel._supported_languages_setcCr:)Nget_supported_speakerscSr<r r=r?r r r!rBrCz9Qwen3TTSModel._supported_speakers_set..rD)r,spksrHr r r!_supported_speakers_setrJz%Qwen3TTSModel._supported_speakers_set languagescCsp|}|dur dSg}|D]}|dur||qt||vr'||q|r6td|dt|dS)z Validate that requested languages are supported by the model. Args: languages (List[str]): Language names for each sample. Raises: ValueError: If any language is not supported. NzUnsupported languages: . Supported: )rIappendrr> ValueErrorsorted)r,rN supportedbadlangr r r!_validate_languagess   z!Qwen3TTSModel._validate_languagesspeakerscCsn|}|dur dSg}|D]}|dus|dkrqt||vr&||q|r5td|dt|dS)z Validate that requested speakers are supported by the Instruct model. Args: speakers (List[Optional[str]]): Speaker names for each sample. Raises: ValueError: If any speaker is not supported. NzUnsupported speakers: rO)rMrr>rPrQrR)r,rWrSrTspkr r r!_validate_speakerss  z Qwen3TTSModel._validate_speakersscCs2|drdSd|vrd|vrt|dkrdSdS)Nz data:audioT/\F) startswithlen)r,r[r r r!_is_probably_base64s z!Qwen3TTSModel._is_probably_base64cCs4zt|}|jdvot|jWStyYdSw)N)httphttpsF)r schemernetloc Exception)r,r[ur r r!_is_urls  zQwen3TTSModel._is_urlb64cCs0d|vr|dr|ddd}t|S)N,zdata:)stripr_splitbase64 b64decode)r,rir r r!_decode_base64_to_wav_bytess z)Qwen3TTSModel._decode_base64_to_wav_bytesrAcCs||r?tj| }|}Wdn1swYt|}tj|ddd\}}Wdn1s9wYn4||ri| |}t|}tj|ddd\}}Wdn1scwYn t j |ddd\}}|j dkrt j|dd}|t jt|fS) Nfloat32F)dtype always_2dT)srmonorkaxis)rhurllibrequesturlopenreadioBytesIOsfrarplibrosaloadndimnpmeanastyperqint)r,rAresp audio_bytesfaudiort wav_bytesr r r!_load_audio_to_nps$       zQwen3TTSModel._load_audio_to_npaudioscCst|tr|}n|g}g}|D]G}t|tr|||qt|trDt|dkrDt|dtjrD||d tj t |dfqt|tjrNt dt dt|t|D]$\}}|djdkrtj|ddd tj |d<|d|df||<q[|S)a Normalize audio inputs into a list of (waveform, sr). Supported forms: - str: wav path / URL / base64 audio string - (np.ndarray, sr): waveform + sampling rate - list of the above Args: audios: Audio input(s). Returns: List[Tuple[np.ndarray, int]]: List of (float32 waveform, original sr). Raises: ValueError: If a numpy waveform is provided without sr. r rrkz3For numpy waveform input, pass a tuple (audio, sr).zUnsupported audio input type: rvrw)r4listrrPrtupler`rndarrayrrqrrQr5r6 enumeraterr)r,ritemsoutair r r!_normalize_audio_inputss$  &$ z%Qwen3TTSModel._normalize_audio_inputscCst|tr|S|gSN)r4r)r,rAr r r! _ensure_list szQwen3TTSModel._ensure_listtextcC d|dS)N<|im_start|>assistant z!<|im_end|> <|im_start|>assistant r r,rr r r!_build_assistant_text  z#Qwen3TTSModel._build_assistant_textcCr)Nr <|im_end|> r rr r r!_build_ref_textrzQwen3TTSModel._build_ref_textinstructcCr)Nz<|im_start|>user rr )r,rr r r!_build_instruct_textrz"Qwen3TTSModel._build_instruct_texttextscCsVg}|D]$}|j|ddd}|d|j}|dkr!|dn|}||q|S)NptT)rreturn_tensorspadding input_idsrkr)r'tor%dim unsqueezerP)r,rrrinputinput_idr r r!_tokenize_textss zQwen3TTSModel._tokenize_texts do_sampletop_ktop_p temperaturerepetition_penaltysubtalker_dosamplesubtalker_top_ksubtalker_top_psubtalker_temperaturemax_new_tokensc  stddddddddddd dtd td tffd d } t| } | j| d || d|| d|| d|| d|| d|| d|| d|| d| | d| d | S)a Merge user-provided generation arguments with defaults from `generate_config.json`. Rule: - If the user explicitly passes a value (not None), use it. - Otherwise, use the value from generate_config.json if present. - Otherwise, fall back to the hard defaults. Args: do_sample, top_k, top_p, temperature, repetition_penalty, subtalker_dosample, subtalker_top_k, subtalker_top_p, subtalker_temperature, max_new_tokens: Common generation parameters. **kwargs: Other arguments forwarded to model.generate(). Returns: Dict[str, Any]: Final kwargs to pass into model.generate(). T2g?g?g?i) rrrrrrrrrrnameuser_valr/cs(|dur|S|jvrj|S|Sr)r$)rr hard_defaultsr,r r!pickLs   z2Qwen3TTSModel._merge_generate_kwargs..pickrrrrrrrrrr)dictrrupdate)r,rrrrrrrrrrr9rmergedr rr!_merge_generate_kwargss6   z$Qwen3TTSModel._merge_generate_kwargsF ref_audiorrc Cs|jjdkrtd|jjd|jjd|jjd||}t|tr)||n|gt|}t|tr:||n|gt|}t|t|ksQt|t|krdtdt|dt|dt|| |}g}g} |D]\} } | | | | qott | d kr|jj j || d d } | j} ng} |D]\} } | |jj j | | d jd qg}tt|| ||D]R\}\\} } }}}|s|d us|d krtd|| }| |jjkrtj|tjt| |jjd}|jj||jjd}| t|rd n||t|t| |dq|S)a Build voice-clone prompt items from reference audio (and optionally reference text) using Base model. Modes: - x_vector_only_mode=True: Only speaker embedding is used to clone voice; ref_text/ref_code are ignored. This is mutually exclusive with ICL. - x_vector_only_mode=False: ICL mode is enabled automatically (icl_mode=True). In this case ref_text is required, because the model continues/conditions on the reference text + reference speech codes. Batch behavior: - ref_audio can be a single item or a list. - ref_text and x_vector_only_mode can be scalars or lists. - If any of them are lists with length > 1, lengths must match. Audio input: - str: local wav path / URL / base64 - (np.ndarray, sr): waveform + sampling rate Args: ref_audio: Reference audio(s) used to extract: - ref_code via `model.speech_tokenizer.encode(...)` - ref_spk_embedding via `model.extract_speaker_embedding(...)` (resampled to 24k) ref_text: Reference transcript(s). Required when x_vector_only_mode=False (ICL mode). x_vector_only_mode: Whether to use speaker embedding only. If False, ICL mode will be used. Returns: List[VoiceClonePromptItem]: List of prompt items that can be converted into `voice_clone_prompt` dict. Raises: ValueError: - If x_vector_only_mode=False but ref_text is missing. - If batch lengths mismatch. basemodel with tokenizer_type:  tts_model_size:  tts_model_type: z` does not support create_voice_clone_prompt, Please check Model Card or Readme for more details.zBatch size mismatch: ref_audio=z , ref_text=z, x_vector_only_mode=rkr)rtNrXzIref_text is required when x_vector_only_mode=False (ICL mode). Bad index=)yorig_sr target_sr)rrt)rrrrr)r#tts_model_typerQtokenizer_typetts_model_sizerr4rr`rrPrFspeech_tokenizerencode audio_codesrzipspeaker_encoder_sample_raterresamplerrrqrextract_speaker_embeddingrr)r,rrrref_audio_list ref_text_list xvec_list normalizedref_wavs_for_coderef_sr_for_codewavrtenc ref_codesrrcodertext xvec_only wav_resamplespk_embr r r!create_voice_clone_promptcsf .  ""        &   z'Qwen3TTSModel.create_voice_clone_promptrcCs8tdd|Ddd|Ddd|Ddd|DdS)NcSg|]}|jqSr )rr@itr r r!rBzEQwen3TTSModel._prompt_items_to_voice_clone_prompt..cSrr )rrr r r!rBrcSrr )rrr r r!rBrcSrr )rrr r r!rBr)rrrr)r)r,rr r r!#_prompt_items_to_voice_clone_prompts     z1Qwen3TTSModel._prompt_items_to_voice_clone_promptlanguagevoice_clone_promptnon_streaming_modec! sjjdkrtdjjdjjdjjd|} t|tr)|n|dur4|gt| ndgt| } t| dkrMt| dkrM| t| } t| t| krctd t| d t|  | |dur|durttd j |||d } t| dkrt| dkr| t| } t| t| krtd t| dt|  | } dd| D} n@t|tr|} t| dkrt| dkr| t| } t| t| krtd t| dt|  | } dd| D} n|} d} fdd| D} |}d}| dur3g}t | D]&\}}|dus|dkr!|dq |gd}||q jdi|}jjd||| | |d|\}}g}t |D]0\}}| dd}|dur|||dur||tj|||j|gddqR||qRjjdd|D\}}g}t |D]G\}}| dd}|dur||durt||jd}t||jd}t|t|d|jd} ||| dq||q||fS)a Voice clone speech using the Base model. You can provide either: - (ref_audio, ref_text, x_vector_only_mode) and let this method build the prompt, OR - `VoiceClonePromptItem` returned by `create_voice_clone_prompt`, OR - a list of `VoiceClonePromptItem` returned by `create_voice_clone_prompt`. `ref_audio` Supported forms: - str: wav path / URL / base64 audio string - (np.ndarray, sr): waveform + sampling rate - list of the above Input flexibility: - text/language can be scalar or list. - prompt can be single or batch. - If batch mode (len(text)>1), lengths must match. Args: text: Text(s) to synthesize. language: Language(s) for each sample. ref_audio: Reference audio(s) for prompt building. Required if voice_clone_prompt is not provided. ref_text: Reference text(s) used for ICL mode (required when x_vector_only_mode=False). x_vector_only_mode: If True, only speaker embedding is used (ignores ref_text/ref_code). If False, ICL mode is used automatically. voice_clone_prompt: list[VoiceClonePromptItem] from `create_voice_clone_prompt`. non_streaming_mode: Using non-streaming text input, this option currently only simulates streaming text input when set to `false`, rather than enabling true streaming input or streaming generation. do_sample: Whether to use sampling, recommended to be set to `true` for most use cases. top_k: Top-k sampling parameter. top_p: Top-p sampling parameter. temperature: Sampling temperature; higher => more random. repetition_penalty: Penalty to reduce repeated tokens/codes. subtalker_dosample: Sampling switch for the sub-talker (only valid for qwen3-tts-tokenizer-v2) if applicable. subtalker_top_k: Top-k for sub-talker sampling (only valid for qwen3-tts-tokenizer-v2). subtalker_top_p: Top-p for sub-talker sampling (only valid for qwen3-tts-tokenizer-v2). subtalker_temperature: Temperature for sub-talker sampling (only valid for qwen3-tts-tokenizer-v2). max_new_tokens: Maximum number of new codec tokens to generate. **kwargs: Any other keyword arguments supported by HuggingFace Transformers `generate()` can be passed. They will be forwarded to the underlying `Qwen3TTSForConditionalGeneration.generate(...)`. Returns: Tuple[List[np.ndarray], int]: (wavs, sample_rate) Raises: ValueError: If batch sizes mismatch or required prompt inputs are missing. rrrrz[ does not support generate_voice_clone, Please check Model Card or Readme for more details.NAutorkBatch size mismatch: text= , language=zrz6Qwen3TTSModel.generate_voice_clone..cSrr rrr r r!rBGrcg|]}|qSr rr@tr,r r!rBLrXr)rref_idsrrNrr)rcSg|]}d|iqSrr r@cr r r!rBlr )r#rrQrrrr4rr`rVrrrrrPrrgenerategetrcatrr%rdecodershapemax)!r,rrrrrrrr9rrN prompt_itemsvoice_clone_prompt_dictref_texts_for_ids input_textsrrrrtref_tok gen_kwargstalker_codes_list_codes_for_decodecodes ref_code_listwavs_allfswavs_outrref_len total_lencutr rr!generate_voice_clones O  8            ( z"Qwen3TTSModel.generate_voice_cloneTc sjjdkrtdjjdjjdjjd|}t|tr)|n|dur4|gt|ndgt|}|}t|dkrRt|dkrR|t|}t|dkrdt|dkrd|t|}t|t|krtt|ksntd t|d t|d t| | fd d |D} g} |D]} | dus| dkr| dq| | gdqj di|} jjd| | ||d| \} }jjdd | D\}}||fS)a Generate speech with the VoiceDesign model using natural-language style instructions. Args: text: Text(s) to synthesize. language: Language(s) for each sample. instruct: Instruction(s) describing desired voice/style. Empty string is allowed (treated as no instruction). non_streaming_mode: Using non-streaming text input, this option currently only simulates streaming text input when set to `false`, rather than enabling true streaming input or streaming generation. do_sample: Whether to use sampling, recommended to be set to `true` for most use cases. top_k: Top-k sampling parameter. top_p: Top-p sampling parameter. temperature: Sampling temperature; higher => more random. repetition_penalty: Penalty to reduce repeated tokens/codes. subtalker_dosample: Sampling switch for the sub-talker (only valid for qwen3-tts-tokenizer-v2) if applicable. subtalker_top_k: Top-k for sub-talker sampling (only valid for qwen3-tts-tokenizer-v2). subtalker_top_p: Top-p for sub-talker sampling (only valid for qwen3-tts-tokenizer-v2). subtalker_temperature: Temperature for sub-talker sampling (only valid for qwen3-tts-tokenizer-v2). max_new_tokens: Maximum number of new codec tokens to generate. **kwargs: Any other keyword arguments supported by HuggingFace Transformers `generate()` can be passed. They will be forwarded to the underlying `Qwen3TTSForConditionalGeneration.generate(...)`. Returns: Tuple[List[np.ndarray], int]: (wavs, sample_rate) voice_designrrrz\ does not support generate_voice_design, Please check Model Card or Readme for more details.Nrrkrr , instruct=crr rrrr r!rBrz7Qwen3TTSModel.generate_voice_design..rXr)r instruct_idsrNrcSrrr rr r r!rBrr )r#rrQrrrr4rr`rVrrPrrrrr)r,rrrrr9rrN instructsrrinsrrr wavsrr rr!generate_voice_design|sF 2  8   "&   z#Qwen3TTSModel.generate_voice_designspeakerc s\jjdkrtdjjdjjdjjd|}t|tr)|n|dur4|gt|ndgt|}|} jjdvrHd}t|trR|n|dur]|gt|nd gt|} t|d krvt|d krv|t|}t| d krt|d kr| t|} t| d krt|d kr| t|} t|t|krt| krt| ksntd t|d t|d t| dt|  | |  fdd|D} g} | D]} | dus| d kr| dq| | gdqjdi|}jjd| | || |d|\}}jjdd|D\}}||fS)a Generate speech with the CustomVoice model using a predefined speaker id, optionally controlled by instruction text. Args: text: Text(s) to synthesize. language: Language(s) for each sample. speaker: Speaker name(s). Will be validated against `model.get_supported_speakers()` (case-insensitive). instruct: Optional instruction(s). If None, treated as empty (no instruction). non_streaming_mode: Using non-streaming text input, this option currently only simulates streaming text input when set to `false`, rather than enabling true streaming input or streaming generation. do_sample: Whether to use sampling, recommended to be set to `true` for most use cases. top_k: Top-k sampling parameter. top_p: Top-p sampling parameter. temperature: Sampling temperature; higher => more random. repetition_penalty: Penalty to reduce repeated tokens/codes. subtalker_dosample: Sampling switch for the sub-talker (only valid for qwen3-tts-tokenizer-v2) if applicable. subtalker_top_k: Top-k for sub-talker sampling (only valid for qwen3-tts-tokenizer-v2). subtalker_top_p: Top-p for sub-talker sampling (only valid for qwen3-tts-tokenizer-v2). subtalker_temperature: Temperature for sub-talker sampling (only valid for qwen3-tts-tokenizer-v2). max_new_tokens: Maximum number of new codec tokens to generate. **kwargs: Any other keyword arguments supported by HuggingFace Transformers `generate()` can be passed. They will be forwarded to the underlying `Qwen3TTSForConditionalGeneration.generate(...)`. Returns: Tuple[List[np.ndarray], int]: (wavs, sample_rate) Raises: ValueError: If any speaker/language is unsupported or batch sizes mismatch. custom_voicerrrz\ does not support generate_custom_voice, Please check Model Card or Readme for more details.Nr0b6rXrkrrz , speaker=rcrr rrrr r!rB2rz7Qwen3TTSModel.generate_custom_voice..r)rrrNrWrcSrrr rr r r!rBFrr )r#rrQrrrr4rr`rVrZrrPrrrrr)r,rrrrrr9rrNrWrrrrrrr rrr rr!generate_custom_voicesX 9  8  8   0*     z#Qwen3TTSModel.generate_custom_voicecC|}|dur dSt|S)a List supported speaker names for the current model. This is a convenience wrapper around `model.get_supported_speakers()`. If the underlying model does not expose speaker constraints (returns None), this method also returns None. Returns: Optional[List[str]]: - A sorted list of supported speaker names (lowercased), if available. - None if the model does not provide supported speakers. N)rMrRr,rSr r r!rKJ z$Qwen3TTSModel.get_supported_speakerscCr)a List supported language names for the current model. This is a convenience wrapper around `model.get_supported_languages()`. If the underlying model does not expose language constraints (returns None), this method also returns None. Returns: Optional[List[str]]: - A sorted list of supported language names (lowercased), if available. - None if the model does not provide supported languages. N)rIrRr r r r!r;]r!z%Qwen3TTSModel.get_supported_languagesr) NNNNNNNNNN)NF)NNNFNF)NT)NNT)3rrrrrrrrrr- classmethodr3rFrIrMrrVrZrrarhbytesrprrrrrr AudioLiker MaybeListrrrrrrrfloatrinference_moderrrno_gradrrrrKr;r r r r!r"6s   (  ,)     D h  ' ^ nr")$rnr}urllib.requestry dataclassesrtypingrrrrrr urllib.parser rnumpyr soundfilerr transformersr r r core.modelsrrrrrrr$r%rr"r r r r!s,