o
ei/3 @ s d dl mZmZmZ ddlmZ ddlmZ ddlm Z ddl
m Z m Z ddl
mZ e r=d d lZdd
lmZ dd lmZ d ZG d
d deddZG dd deZd S ) )Any TypedDictoverload )
AudioInput)GenerationConfig)is_torch_available)ChatChatType )PipelineN)%MODEL_FOR_TEXT_TO_SPECTROGRAM_MAPPING)SpeechT5HifiGanzmicrosoft/speecht5_hifiganc @ s" e Zd ZU dZeed< eed< dS ) AudioOutputz
audio (`AudioInput`):
The generated audio waveform.
sampling_rate (`int`):
The sampling rate of the generated audio waveform.
audio
sampling_rateN)__name__
__module__ __qualname____doc__r __annotations__int r r b/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/pipelines/text_to_audio.pyr ! s
r F)totalc s e Zd ZdZdZdZdZdZdZe ddZ
ddd fdd
Z d
d Z d d
Z
edededefddZedee dedee fddZedededefddZedee dedee fddZ fddZ dddZdd Z ZS ) TextToAudioPipelinea
Text-to-audio generation pipeline using any `AutoModelForTextToWaveform` or `AutoModelForTextToSpectrogram`. This
pipeline generates an audio file from an input text and optional other conditional inputs.
Unless the model you're using explicitly sets these generation parameters in its configuration files
(`generation_config.json`), the following default values will be used:
- max_new_tokens: 256
Example:
```python
>>> from transformers import pipeline
>>> pipe = pipeline(model="suno/bark-small")
>>> output = pipe("Hey it's HuggingFace on the phone!")
>>> audio = output["audio"]
>>> sampling_rate = output["sampling_rate"]
```
Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial)
You can specify parameters passed to the model by using [`TextToAudioPipeline.__call__.forward_params`] or
[`TextToAudioPipeline.__call__.generate_kwargs`].
Example:
```python
>>> from transformers import pipeline
>>> music_generator = pipeline(task="text-to-audio", model="facebook/musicgen-small")
>>> # diversify the music generation by adding randomness with a high temperature and set a maximum music length
>>> generate_kwargs = {
... "do_sample": True,
... "temperature": 0.7,
... "max_new_tokens": 35,
... }
>>> outputs = music_generator("Techno music with high melodic riffs", generate_kwargs=generate_kwargs)
```
This pipeline can currently be loaded from [`pipeline`] using the following task identifiers: `"text-to-speech"` or
`"text-to-audio"`.
See the list of available models on [huggingface.co/models](https://huggingface.co/models?filter=text-to-speech).
TNF )max_new_tokens)vocoderr c s@ t j|i | d | _| jjt v r%|d u r"tt
| jj n|| _| jj j
dv r/d | _|| _| jd ur=| jj j| _| jd u r| jj }| jjdd }|d urY|| dD ]&}t||d }|d urk|| _q[t|dd d urt|j|d }|d ur|| _q[| jd u r| jd urt| jdr| jjj| _d S d S d S d S )N)musicgenspeecht5generation_config) sample_rater codec_configfeature_extractor)super__init__r model __class__r
valuesr from_pretrainedDEFAULT_VOCODER_IDtodeviceconfig
model_type processorr __dict__getupdateto_dictgetattrr# hasattrr$ )selfr r argskwargsr.
gen_configsampling_rate_namer( r r r& m s:
z TextToAudioPipeline.__init__c K s t |tr|g}| jjjdkr-d}t| jdr t| jjdd}|dddd}| | |}| j
d ur5| j
n| j }t |t rL|j
|jfddd|}|S | jjjd krZd
d |D }| jjjd krhd
d |D }||fi |ddi}|S )Nbarkr semantic_configmax_input_semantic_lengthFT)
max_lengthadd_special_tokensreturn_attention_maskreturn_token_type_ids)tokenize return_dictcsmc S $ g | ]}| d sd| n|qS )[z[0]
startswith.0tr r r
$ z2TextToAudioPipeline.preprocess..diac S rG )rH z[S1] rI rK r r r rN rO return_tensorspt)
isinstancestrr' r. r/ r6 r! r5 r> r3 r0 tokenizerr apply_chat_templatemessages)r7 textr9 r@
new_kwargs preprocessoroutputr r r
preprocess s<
z TextToAudioPipeline.preprocessc K s | j || jd}|d }|d }| j rM| j || jd}d|vr&| j|d< || |ddi | jjjdv rAd|vrAd|d< | jjd i ||}nt |rZt
d | | jd i ||d
}| j d urp| |}|S ) N)r- forward_paramsgenerate_kwargsr! return_dict_in_generateT)rF output_audiozYou're using the `TextToAudioPipeline` with a forward-only model, but `generate_kwargs` is non empty. For forward-only TTA models, please use `forward_params` instead of `generate_kwargs`. For reference, the `generate_kwargs` used here are: r r )
_ensure_tensor_on_devicer- r' can_generater! r3 r. r/ generatelen
ValueErrorkeysr )r7 model_inputsr9 r] r^ r[ r r r _forward s.
z TextToAudioPipeline._forward text_inputsr] returnc K d S Nr r7 ri r] r r r __call__ z TextToAudioPipeline.__call__c K rk rl r rm r r r rn ro c K rk rl r rm r r r rn ro c K rk rl r rm r r r rn ro c s t j|fi |S )aL
Generates speech/audio from the inputs. See the [`TextToAudioPipeline`] documentation for more information.
Args:
text_inputs (`str`, `list[str]`, `ChatType`, or `list[ChatType]`):
One or several texts to generate. If strings or a list of string are passed, this pipeline will
generate the corresponding text. Alternatively, a "chat", in the form of a list of dicts with "role"
and "content" keys, can be passed, or a list of such chats. When chats are passed, the model's chat
template will be used to format them before passing them to the model.
forward_params (`dict`, *optional*):
Parameters passed to the model generation/forward method. `forward_params` are always passed to the
underlying model.
generate_kwargs (`dict`, *optional*):
The dictionary of ad-hoc parametrization of `generate_config` to be used for the generation call. For a
complete overview of generate, check the [following
guide](https://huggingface.co/docs/transformers/en/main_classes/text_generation). `generate_kwargs` are
only passed to the underlying model if the latter is a generative model.
Return:
`AudioOutput` or a list of `AudioOutput`, which is a `TypedDict` with two keys:
- **audio** (`np.ndarray` of shape `(nb_channels, audio_length)`) -- The generated audio waveform.
- **sampling_rate** (`int`) -- The sampling rate of the generated audio waveform.
)r% rn rm r< r r rn s c C sr t | dd d ur
| j|d< t | dd d ur | j|d< | j|d< |r#|ni |r(|ni d}|d u r2i }i }|||fS )Nassistant_modelassistant_tokenizerrU )r] r^ )r5 rp rU rq )r7 preprocess_paramsr] r^ paramspostprocess_paramsr r r _sanitize_parameters
s
z(TextToAudioPipeline._sanitize_parametersc C s d}t |trd|v r|d }nd}|d }n t |tr |d }|r-| jd ur-| j|}t |trFdd |D }t|dkrA|n|d }n |jd tj d
}t || j
d S ) NFr T sequencesr c S s$ g | ]}|j d tjd qS )cpur- dtype)r, torchfloatnumpysqueeze)rL elr r r rN 3 rO z3TextToAudioPipeline.postprocess..r rw rx )r r )rS dicttupler0 decodelistrd r, rz r{ r| r} r r )r7 r needs_decodingr r r postprocess$ s$
z TextToAudioPipeline.postprocess)NNN)r r r r _pipeline_calls_generate_load_processor_load_image_processor_load_feature_extractor_load_tokenizerr _default_generation_configr& r\ rh r rT r r rn r r
ru r
__classcell__r r r< r r - s6 4('*
r )typingr r r audio_utilsr
generationr utilsr utils.chat_template_utilsr r
baser rz models.auto.modeling_autor
!models.speecht5.modeling_speecht5r r+ r r r r r r s