o
۷i+ @ sv d dl mZmZmZ ddlmZ ddlmZ ddlm Z e r/d dl
Z
ddl m Z dd l
mZ d
ZG d d d e ZdS )
)AnyUnionoverload )GenerationConfig)is_torch_available )PipelineN)%MODEL_FOR_TEXT_TO_SPECTROGRAM_MAPPING)SpeechT5HifiGanzmicrosoft/speecht5_hifiganc
s e Zd ZdZdZdZdZdZdZdZe ddZ
dddd fdd
Z d
d Z d d
Z
edededeeef fddZedee dedeeeef fddZdeeee f deeeef eeeef f f fdd Z dddZdd Z ZS )TextToAudioPipelinea
Text-to-audio generation pipeline using any `AutoModelForTextToWaveform` or `AutoModelForTextToSpectrogram`. This
pipeline generates an audio file from an input text and optional other conditional inputs.
Unless the model you're using explicitly sets these generation parameters in its configuration files
(`generation_config.json`), the following default values will be used:
- max_new_tokens: 256
Example:
```python
>>> from transformers import pipeline
>>> pipe = pipeline(model="suno/bark-small")
>>> output = pipe("Hey it's HuggingFace on the phone!")
>>> audio = output["audio"]
>>> sampling_rate = output["sampling_rate"]
```
Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial)
You can specify parameters passed to the model by using [`TextToAudioPipeline.__call__.forward_params`] or
[`TextToAudioPipeline.__call__.generate_kwargs`].
Example:
```python
>>> from transformers import pipeline
>>> music_generator = pipeline(task="text-to-audio", model="facebook/musicgen-small", framework="pt")
>>> # diversify the music generation by adding randomness with a high temperature and set a maximum music length
>>> generate_kwargs = {
... "do_sample": True,
... "temperature": 0.7,
... "max_new_tokens": 35,
... }
>>> outputs = music_generator("Techno music with high melodic riffs", generate_kwargs=generate_kwargs)
```
This pipeline can currently be loaded from [`pipeline`] using the following task identifiers: `"text-to-speech"` or
`"text-to-audio"`.
See the list of available models on [huggingface.co/models](https://huggingface.co/models?filter=text-to-speech).
TF )max_new_tokensN)vocoder
sampling_rate no_processorc s@ t j|i | || _| jdkrtdd | _| jjt v r1|d u r.t
t
| jjn|| _|| _| jd ur?| jjj| _| jd u r| jj}| jjdd }|d ur[|| dD ]&}t||d }|d urm|| _q]t|dd d urt|j|d }|d ur|| _q]| jd u r| jst| jdr| jjj| _d S d S d S d S )Ntfz5The TextToAudioPipeline is only available in PyTorch.generation_config) sample_rater codec_configfeature_extractor)super__init__r framework
ValueErrorr model __class__r
valuesr from_pretrainedDEFAULT_VOCODER_IDtodevicer config__dict__getupdateto_dictgetattrr hasattr processorr ) selfr r r argskwargsr"
gen_configsampling_rate_namer Z/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/pipelines/text_to_audio.pyr a s<
z TextToAudioPipeline.__init__c K sv t |tr|g}| jjjdkr$| jjddddddd}|| |}| j r*| j
n| j }||fi |dd i}|S )
Nbarkmax_input_semantic_lengthr
FT
max_length)r4 add_special_tokensreturn_attention_maskreturn_token_type_idspaddingreturn_tensorspt)
isinstancestrr r"
model_typer semantic_configr$ r% r tokenizerr) )r* textr,
new_kwargs preprocessoroutputr0 r0 r1
preprocess s
z TextToAudioPipeline.preprocessc K s | j || jd}|d }|d }| j r7| j || jd}d|vr&| j|d< || | jjdi ||}nt|rDtd| | jdi ||d }| j
d urZ|
|}|S )N)r! forward_paramsgenerate_kwargsr zYou're using the `TextToAudioPipeline` with a forward-only model, but `generate_kwargs` is non empty. For forward-only TTA models, please use `forward_params` instead of `generate_kwargs`. For reference, the `generate_kwargs` used here are: r r0 ) _ensure_tensor_on_devicer! r can_generater r% generatelenr keysr )r* model_inputsr, rE rF rC r0 r0 r1 _forward s&
z TextToAudioPipeline._forward text_inputsrE returnc K d S Nr0 r* rN rE r0 r0 r1 __call__ z TextToAudioPipeline.__call__c K rP rQ r0 rR r0 r0 r1 rS rT c s t j|fi |S )a
Generates speech/audio from the inputs. See the [`TextToAudioPipeline`] documentation for more information.
Args:
text_inputs (`str` or `list[str]`):
The text(s) to generate.
forward_params (`dict`, *optional*):
Parameters passed to the model generation/forward method. `forward_params` are always passed to the
underlying model.
generate_kwargs (`dict`, *optional*):
The dictionary of ad-hoc parametrization of `generate_config` to be used for the generation call. For a
complete overview of generate, check the [following
guide](https://huggingface.co/docs/transformers/en/main_classes/text_generation). `generate_kwargs` are
only passed to the underlying model if the latter is a generative model.
Return:
A `dict` or a list of `dict`: The dictionaries have two keys:
- **audio** (`np.ndarray` of shape `(nb_channels, audio_length)`) -- The generated audio waveform.
- **sampling_rate** (`int`) -- The sampling rate of the generated audio waveform.
)r rS rR r/ r0 r1 rS s c C sr t | dd d ur
| j|d< t | dd d ur | j|d< | j|d< |r#|ni |r(|ni d}|d u r2i }i }|||fS )Nassistant_modelassistant_tokenizerr? )rE rF )r' rU r? rV )r* preprocess_paramsrE rF paramspostprocess_paramsr0 r0 r1 _sanitize_parameters s
z(TextToAudioPipeline._sanitize_parametersc C s i }| j jjdkr d}nd}| jr(t|tr|| }nt|tr%|d }n |}n| j|}t|t r=dd |D |d< n |j
dt j d
|d< | j|d < |S )
Ncsmaudiowaveformr c S s g | ] }|j d tjd qS )cpur! dtype)r torchfloatnumpy).0elr0 r0 r1
s z3TextToAudioPipeline.postprocess..r^ r_ r )r r" r= r r; dicttupler) decodelistr ra rb rc r )r* r\ output_dict waveform_keyr] r0 r0 r1 postprocess s
z TextToAudioPipeline.postprocess)NNN)__name__
__module__ __qualname____doc___load_processor_pipeline_calls_generate_load_image_processor_load_feature_extractor_load_tokenizerr _default_generation_configr rD rM r r< r rg rS rj r rZ rm
__classcell__r0 r0 r/ r1 r s8 5*! (
r )typingr r r
generationr utilsr baser ra models.auto.modeling_autor
!models.speecht5.modeling_speecht5r r r r0 r0 r0 r1 s