APEX FusedRMSNorm not available, using native implementation
/home/ubuntu/vibevoice/vibevoice/processor/vibevoice_asr_processor.py:23: UserWarning: audio_utils not available, will fall back to soundfile for audio loading
  warnings.warn("audio_utils not available, will fall back to soundfile for audio loading")
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'Qwen2Tokenizer'. 
The class this function is called from is 'VibeVoiceTextTokenizerFast'.

Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:00<00:01,  1.75it/s]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:01<00:00,  1.94it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:01<00:00,  2.06it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:01<00:00,  2.00it/s]
Warming up baseline...

[1] Baseline: no graphs, no compile, 10 steps, cfg=1.3
    RTF=0.945x  TTFB=137ms  Audio=11.60s

[2] CUDA graph diffusion head, 10 steps, cfg=1.3
Setting up CUDA graph for diffusion head...
Traceback (most recent call last):
  File "/home/ubuntu/vibevoice/cuda_graph_engine.py", line 73, in _capture
    eps = self.prediction_head(self.static_speech, self.static_t, condition=self.static_condition)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/vibevoice/vibevoice/modular/modular_vibevoice_diffusion_head.py", line 272, in forward
    t = self.t_embedder(timesteps)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/vibevoice/vibevoice/modular/modular_vibevoice_diffusion_head.py", line 91, in forward
    t_freq = self.timestep_embedding(t, self.frequency_embedding_size)
  File "/home/ubuntu/vibevoice/vibevoice/modular/modular_vibevoice_diffusion_head.py", line 83, in timestep_embedding
    ).to(t.device)
RuntimeError: CUDA error: operation not permitted when stream is capturing
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/vibevoice/cuda_graph_engine.py", line 348, in <module>
    main()
  File "/home/ubuntu/vibevoice/cuda_graph_engine.py", line 312, in main
    r, out = benchmark_cuda_graphs(model, processor, voice_path, text, ddpm_steps=10, cfg_scale=1.3)
  File "/home/ubuntu/vibevoice/cuda_graph_engine.py", line 194, in benchmark_cuda_graphs
    cg_diffusion = CUDAGraphDiffusion(
  File "/home/ubuntu/vibevoice/cuda_graph_engine.py", line 50, in __init__
    self._capture()
  File "/home/ubuntu/vibevoice/cuda_graph_engine.py", line 72, in _capture
    with torch.cuda.graph(g):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/graphs.py", line 186, in __exit__
    self.cuda_graph.capture_end()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/graphs.py", line 84, in capture_end
    super().capture_end()
RuntimeError: CUDA error: operation failed due to a previous error during capture
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.