o ۷if@sddlZddlmZmZddlZddlZddlmm Z ddl m Z m Z mZmZddlmZmZddlmZddlmZmZmZddlmZd d lmZmZmZmZerbddl m!m"Z#d Z$nd Z$e%e&Z'd Z(GdddeeeZ)dS)N)AnyCallable)ClapTextModelWithProjectionRobertaTokenizerRobertaTokenizerFastSpeechT5HifiGan) AutoencoderKLUNet2DConditionModel)KarrasDiffusionSchedulers)is_torch_xla_availableloggingreplace_example_docstring) randn_tensor)AudioPipelineOutputDeprecatedPipelineMixinDiffusionPipelineStableDiffusionMixinTFaj Examples: ```py >>> from diffusers import AudioLDMPipeline >>> import torch >>> import scipy >>> repo_id = "cvssp/audioldm-s-full-v2" >>> pipe = AudioLDMPipeline.from_pretrained(repo_id, torch_dtype=torch.float16) >>> pipe = pipe.to("cuda") >>> prompt = "Techno music with a strong, upbeat tempo and high melodic riffs" >>> audio = pipe(prompt, num_inference_steps=10, audio_length_in_s=5.0).audios[0] >>> # save the audio sample as a .wav file >>> scipy.io.wavfile.write("techno.wav", rate=16000, data=audio) ``` c$seZdZdZdZdZdededee Bde de d e f fd d Z d1d ejd Bdejd BfddZddZddZddZ d1ddZd2ddZeee       d3d!eeeBd"ed Bd#ed$ed%eeeBd Bd&ed Bd'ed(ejeejBd Bd)ejd Bd ejd Bdejd Bd*ed+eeeejgd fd Bd,ed Bd-e ee!fd Bd.ed Bf d/d0Z"Z#S)4AudioLDMPipelinea Pipeline for text-to-audio generation using AudioLDM. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.). Args: vae ([`AutoencoderKL`]): Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations. text_encoder ([`~transformers.ClapTextModelWithProjection`]): Frozen text-encoder (`ClapTextModelWithProjection`, specifically the [laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused) variant. tokenizer ([`PreTrainedTokenizer`]): A [`~transformers.RobertaTokenizer`] to tokenize text. unet ([`UNet2DConditionModel`]): A `UNet2DConditionModel` to denoise the encoded audio latents. scheduler ([`SchedulerMixin`]): A scheduler to be used in combination with `unet` to denoise the encoded audio latents. Can be one of [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`]. vocoder ([`~transformers.SpeechT5HifiGan`]): Vocoder of class `SpeechT5HifiGan`. z0.33.1ztext_encoder->unet->vaevae text_encoder tokenizerunet schedulervocodercsRt|j||||||dt|ddr$dt|jjjd|_dSd|_dS)N)rrrrrrrr) super__init__register_modulesgetattrlenrconfigblock_out_channelsvae_scale_factor)selfrrrrrr __class__d/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/pipelines/audioldm/pipeline_audioldm.pyrWs 2zAudioLDMPipeline.__init__N prompt_embedsnegative_prompt_embedsc Cs|dur t|tr d}n|durt|trt|}n|jd}|dur|j|d|jjddd} | j} | j} |j|ddd j} | jd | jd krmt | | sm|j | dd|jjdd f} t d |jjd | |j| || |d }|j}tj|d d}|j|jj|d}|j\}}|d|}||||}|r|dur|durdg|}n;t|t|urtdt|dt|dt|tr|g}n|t|krtd|dt|d|d|d |}|jd}|j|d|ddd}|j|}|j|} |j|| d }|j}tj|d d}|r?|jd}|j|jj|d}|d|}||||}t ||g}|S)a` Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `list[str]`, *optional*): prompt to be encoded device (`torch.device`): torch device num_waveforms_per_prompt (`int`): number of waveforms that should be generated per prompt do_classifier_free_guidance (`bool`): whether to use classifier free guidance or not negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the audio generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. Nrr max_lengthTpt)paddingr- truncationreturn_tensorslongest)r/r1z\The following part of your input was truncated because CLAP can only handle sequences up to z tokens: )attention_mask)dim)dtypedevicez?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.) isinstancestrlistr"shapermodel_max_length input_idsr4torchequal batch_decodeloggerwarningrto text_embedsF normalizer6repeatviewtype TypeError ValueErrorcat)r&promptr7num_waveforms_per_promptdo_classifier_free_guidancenegative_promptr+r, batch_size text_inputstext_input_idsr4untruncated_ids removed_textbs_embedseq_len uncond_tokensr- uncond_inputuncond_input_idsr)r)r*_encode_promptls"            zAudioLDMPipeline._encode_promptcCs$d|jjj|}|j|j}|S)Nr)rr#scaling_factordecodesample)r&latentsmel_spectrogramr)r)r*decode_latentsszAudioLDMPipeline.decode_latentscCs0|dkr |d}||}|}|S)Nr)r5squeezercpufloat)r&rbwaveformr)r)r*mel_spectrogram_to_waveforms    z,AudioLDMPipeline.mel_spectrogram_to_waveformcCsXdtt|jjjv}i}|r||d<dtt|jjjv}|r*||d<|S)Neta generator)setinspect signaturerstep parameterskeys)r&rkrj accepts_etaextra_step_kwargsaccepts_generatorr)r)r*prepare_extra_step_kwargssz*AudioLDMPipeline.prepare_extra_step_kwargsc Csj||j}||krtd|d|d|jjj|jdkr-td|jjjd|jd|dus>|durKt|tr>|dkrKtd|dt|d|dur^|dur^td |d |d |durj|durjtd |durt|tst|t std t||dur|durtd|d|d |dur|dur|j |j krtd|j d|j ddSdSdS)NzH`audio_length_in_s` has to be a positive value greater than or equal to z , but is r9rzwThe number of frequency bins in the vocoder's log-mel spectrogram has to be divisible by the VAE scale factor, but got z bins and a scale factor of z5`callback_steps` has to be a positive integer but is z of type zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` ) r%rMrr# model_in_dimr:intrKr;r<r=) r&rOaudio_length_in_svocoder_upsample_factorcallback_stepsrRr+r,min_audio_length_in_sr)r)r* check_inputssb  zAudioLDMPipeline.check_inputsc Cs||t||jt|jjj|jf}t|tr+t||kr+tdt|d|d|dur8t ||||d}n| |}||j j }|S)Nz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.)rkr7r6) rwr%rr#rvr:r<r"rMrrErinit_noise_sigma) r&rSnum_channels_latentsheightr6r7rkrar=r)r)r*prepare_latentsRs     z AudioLDMPipeline.prepare_latents @rTnprOrxnum_inference_stepsguidance_scalerRrPrjrkra return_dictcallbackrzcross_attention_kwargs output_typec% Cst|jjj|jjj}|dur|jjj|j|}t ||}t ||jjj}||jdkrOt t ||j|j}t d|d||d|d| |||||| | |durft|trfd}n|durtt|trtt|}n| jd}|j}|dk}|j|||||| | d } |jj||d |jj}|jjj}|||||| j||| } |||}t|||jj}|j|d }t|D]\}}|rt | gd n| }|j!||}|j||d| |d j"}|r|#d \} }!| ||!| }|jj$||| fi|j%} |t|dks!|d|kr@|d|jjdkr@|&| dur@||dkr@|t'|jdd}"| |"|| t(rGt)*qWdn 1sSwY|+| }#|,|#}$|$ddd|f}$|dkru|$-}$| s{|$fSt.|$dS)u The call function to the pipeline for generation. Args: prompt (`str` or `list[str]`, *optional*): The prompt or prompts to guide audio generation. If not defined, you need to pass `prompt_embeds`. audio_length_in_s (`int`, *optional*, defaults to 5.12): The length of the generated audio sample in seconds. num_inference_steps (`int`, *optional*, defaults to 10): The number of denoising steps. More denoising steps usually lead to a higher quality audio at the expense of slower inference. guidance_scale (`float`, *optional*, defaults to 2.5): A higher guidance scale value encourages the model to generate audio that is closely linked to the text `prompt` at the expense of lower sound quality. Guidance scale is enabled when `guidance_scale > 1`. negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts to guide what to not include in audio generation. If not defined, you need to pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`). num_waveforms_per_prompt (`int`, *optional*, defaults to 1): The number of waveforms to generate per prompt. eta (`float`, *optional*, defaults to 0.0): Corresponds to parameter eta (η) from the [DDIM](https://huggingface.co/papers/2010.02502) paper. Only applies to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers. generator (`torch.Generator` or `list[torch.Generator]`, *optional*): A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor is generated by sampling using the supplied random `generator`. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not provided, text embeddings are generated from the `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not provided, `negative_prompt_embeds` are generated from the `negative_prompt` input argument. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.AudioPipelineOutput`] instead of a plain tuple. callback (`Callable`, *optional*): A function that calls every `callback_steps` steps during inference. The function is called with the following arguments: `callback(step: int, timestep: int, latents: torch.Tensor)`. callback_steps (`int`, *optional*, defaults to 1): The frequency at which the `callback` function is called. If not specified, the callback is called at every step. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the [`AttentionProcessor`] as defined in [`self.processor`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). output_type (`str`, *optional*, defaults to `"np"`): The output format of the generated image. Choose between `"np"` to return a NumPy `np.ndarray` or `"pt"` to return a PyTorch `torch.Tensor` object. Examples: Returns: [`~pipelines.AudioPipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.AudioPipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list with the generated audio. NrzAudio length in seconds z is increased to z; so that it can be handled by the model. It will be cut to z after the denoising process.rg?)r+r,)r7)totalr)encoder_hidden_states class_labelsrorderr)audios)/rprodrr#upsample_rates sampling_rater sample_sizer%rwceilrCinfor|r:r;r<r"r=_execution_devicer]r set_timesteps timesteps in_channelsrr6rur progress_bar enumerater@rNscale_model_inputr`chunkro prev_sampleupdater! XLA_AVAILABLExm mark_steprcrinumpyr)%r&rOrxrrrRrPrjrkrar+r,rrrzrrryroriginal_waveform_lengthrSr7rQrr~rsnum_warmup_stepsritlatent_model_input noise_prednoise_pred_uncondnoise_pred_textstep_idxrbaudior)r)r*__call__hsO        6  "   zAudioLDMPipeline.__call__)NNN)N)NNrrNrrNNNNTNrNr)$__name__ __module__ __qualname____doc___last_supported_versionmodel_cpu_offload_seqr rrrr r rrr@Tensorr]rcrirur|rno_gradrEXAMPLE_DOC_STRINGr;r<rgrw Generatorboolrdictrr __classcell__r)r)r'r*r<s    ;      r)*rmtypingrrrrr@torch.nn.functionalnn functionalrG transformersrrrrmodelsr r schedulersr utilsr r rutils.torch_utilsrpipeline_utilsrrrrtorch_xla.core.xla_modelcore xla_modelrr get_loggerrrCrrr)r)r)r*s$