/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:275: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
copy checkpoint for finetune

vocab :  543

vocoder :  vocos
Using logger: None
Gradient accumulation checkpointing with per_updates now, old logic per_steps used with before f992c4e
Loading dataset ...

Sorting with sampler... if slow, check whether dataset is provided with duration:   0%|          | 0/2330 [00:00<?, ?it/s]
Sorting with sampler... if slow, check whether dataset is provided with duration: 100%|██████████| 2330/2330 [00:00<00:00, 1367580.23it/s]

Creating dynamic batches with 9600 audio frames per gpu:   0%|          | 0/2330 [00:00<?, ?it/s]
Creating dynamic batches with 9600 audio frames per gpu: 100%|██████████| 2330/2330 [00:00<00:00, 2000968.12it/s]

Epoch 1/10:   0%|          | 0/97 [00:00<?, ?update/s]
Epoch 1/10:   1%|          | 1/97 [00:09<15:10,  9.49s/update]
Epoch 1/10:   1%|          | 1/97 [00:09<15:10,  9.49s/update, loss=0.637, update=1]
Epoch 1/10:   2%|▏         | 2/97 [00:17<13:20,  8.43s/update, loss=0.637, update=1]
Epoch 1/10:   2%|▏         | 2/97 [00:17<13:20,  8.43s/update, loss=nan, update=2]  
Epoch 1/10:   3%|▎         | 3/97 [00:24<12:44,  8.13s/update, loss=nan, update=2]
Epoch 1/10:   3%|▎         | 3/97 [00:24<12:44,  8.13s/update, loss=nan, update=3]
Epoch 1/10:   4%|▍         | 4/97 [00:32<12:22,  7.99s/update, loss=nan, update=3]
Epoch 1/10:   4%|▍         | 4/97 [00:32<12:22,  7.99s/update, loss=nan, update=4]
Epoch 1/10:   5%|▌         | 5/97 [00:40<12:07,  7.91s/update, loss=nan, update=4]
Epoch 1/10:   5%|▌         | 5/97 [00:40<12:07,  7.91s/update, loss=nan, update=5]
Epoch 1/10:   6%|▌         | 6/97 [00:48<11:55,  7.86s/update, loss=nan, update=5]
Epoch 1/10:   6%|▌         | 6/97 [00:48<11:55,  7.86s/update, loss=nan, update=6]
Epoch 1/10:   7%|▋         | 7/97 [00:55<11:32,  7.70s/update, loss=nan, update=6]
Epoch 1/10:   7%|▋         | 7/97 [00:55<11:32,  7.70s/update, loss=nan, update=7]
Epoch 1/10:   8%|▊         | 8/97 [01:02<11:09,  7.52s/update, loss=nan, update=7]
Epoch 1/10:   8%|▊         | 8/97 [01:02<11:09,  7.52s/update, loss=nan, update=8]
Epoch 1/10:   9%|▉         | 9/97 [01:10<11:13,  7.65s/update, loss=nan, update=8]
Epoch 1/10:   9%|▉         | 9/97 [01:10<11:13,  7.65s/update, loss=nan, update=9]
Epoch 1/10:  10%|█         | 10/97 [01:18<11:06,  7.66s/update, loss=nan, update=9]
Epoch 1/10:  10%|█         | 10/97 [01:18<11:06,  7.66s/update, loss=nan, update=10]
Epoch 1/10:  11%|█▏        | 11/97 [01:25<10:57,  7.65s/update, loss=nan, update=10]
Epoch 1/10:  11%|█▏        | 11/97 [01:25<10:57,  7.65s/update, loss=nan, update=11]
Epoch 1/10:  12%|█▏        | 12/97 [01:33<10:52,  7.67s/update, loss=nan, update=11]
Epoch 1/10:  12%|█▏        | 12/97 [01:33<10:52,  7.67s/update, loss=nan, update=12]
Epoch 1/10:  13%|█▎        | 13/97 [01:41<10:46,  7.70s/update, loss=nan, update=12]
Epoch 1/10:  13%|█▎        | 13/97 [01:41<10:46,  7.70s/update, loss=nan, update=13]