'/home/ubuntu/lewm-tts/train_v3c.log' run sampels on latest check[oint check ur memory


The user wants me to run samples on the latest checkpoint from their training run. Let me check my memory first, then look at the log file and find the latest checkpoint.


ToolSearch

Question: select:Read,Bash,Glob


Question: select:Read,Bash,Glob


Let me check my memory file for the LeWM TTS project and look at the training log to find the latest checkpoint.


     1→Device: cuda
     2→LeWM TTS model: 21.11M parameters
     3→Dataset: 6959 samples loaded
     4→Training: 6959 samples, 27 batches/epoch
     5→Epoch 1/200 | avg_loss=11.9772 pred=1.4899 kl=0.2497 recon=2.3543 spec=3.8524 | 42.9s
     6→  New best model (loss=11.9772)
     7→  [2/200] step 50 | loss=11.6261 pred=1.2251 kl=0.2035 recon=2.3549 spec=3.7941 | grad=3.04 | 35.2s
     8→Epoch 2/200 | avg_loss=11.7592 pred=1.3041 kl=0.2161 recon=2.3541 spec=3.8312 | 41.1s
     9→  New best model (loss=11.7592)
    10→Epoch 3/200 | avg_loss=11.3993 pred=1.0685 kl=0.1916 recon=2.3521 spec=3.7509 | 40.2s
    11→  New best model (loss=11.3993)
    12→  [4/200] step 100 | loss=10.4622 pred=0.8959 kl=0.2493 recon=2.3514 spec=3.2423 | grad=3.24 | 28.7s
    13→Epoch 4/200 | avg_loss=10.6588 pred=0.9144 kl=0.2344 recon=2.3339 spec=3.3843 | 40.8s
    14→  New best model (loss=10.6588)
    15→Epoch 5/200 | avg_loss=9.0671 pred=0.8269 kl=0.3122 recon=2.2122 spec=2.5436 | 40.3s
    16→  New best model (loss=9.0671)
    17→  [6/200] step 150 | loss=7.2285 pred=0.8280 kl=0.2964 recon=1.6269 spec=2.0975 | grad=7.31 | 22.7s
    18→Epoch 6/200 | avg_loss=7.2588 pred=0.7820 kl=0.2954 recon=1.6706 spec=2.0902 | 40.6s
    19→  New best model (loss=7.2588)
    20→Epoch 7/200 | avg_loss=5.9076 pred=0.6632 kl=0.3169 recon=1.3503 spec=1.6955 | 40.6s
    21→  New best model (loss=5.9076)
    22→  [8/200] step 200 | loss=4.9208 pred=0.5572 kl=0.3621 recon=1.1829 spec=1.3314 | grad=4.71 | 16.9s
    23→Epoch 8/200 | avg_loss=4.8213 pred=0.5439 kl=0.3731 recon=1.1619 spec=1.3019 | 40.8s
    24→  New best model (loss=4.8213)
    25→Epoch 9/200 | avg_loss=4.0268 pred=0.4353 kl=0.4836 recon=1.0010 spec=1.0590 | 40.4s
    26→  New best model (loss=4.0268)
    27→  [10/200] step 250 | loss=3.8342 pred=0.3734 kl=0.5448 recon=0.9780 spec=1.0024 | grad=32.58 | 10.9s
    28→Epoch 10/200 | avg_loss=3.7727 pred=0.3652 kl=0.5560 recon=0.9683 spec=0.9797 | 40.8s
    29→  Saved checkpoint: /home/ubuntu/lewm-tts/output_v3c/checkpoints/epoch_0010.pt
    30→  New best model (loss=3.7727)
    31→Epoch 11/200 | avg_loss=3.6309 pred=0.3330 kl=0.5902 recon=0.9486 spec=0.9328 | 40.5s
    32→  New best model (loss=3.6309)
    33→  [12/200] step 300 | loss=3.6239 pred=0.3101 kl=0.6098 recon=0.9584 spec=0.9303 | grad=26.33 | 4.9s
    34→Epoch 12/200 | avg_loss=3.4695 pred=0.2884 kl=0.6412 recon=0.9204 spec=0.8922 | 40.6s
    35→  New best model (loss=3.4695)
    36→  [13/200] step 350 | loss=3.2157 pred=0.2090 kl=0.7828 recon=0.8850 spec=0.8227 | grad=14.26 | 39.2s
    37→Epoch 13/200 | avg_loss=3.3178 pred=0.2341 kl=0.7302 recon=0.8977 spec=0.8574 | 40.8s
    38→  New best model (loss=3.3178)
    39→Epoch 14/200 | avg_loss=3.2315 pred=0.1837 kl=0.8377 recon=0.8898 spec=0.8436 | 40.9s
    40→  New best model (loss=3.2315)
    41→  [15/200] step 400 | loss=3.1591 pred=0.1422 kl=0.9733 recon=0.8933 spec=0.8179 | grad=22.70 | 33.5s
    42→Epoch 15/200 | avg_loss=3.1433 pred=0.1514 kl=0.9374 recon=0.8741 spec=0.8269 | 40.9s
    43→  New best model (loss=3.1433)
    44→Epoch 16/200 | avg_loss=3.0754 pred=0.1303 kl=1.0223 recon=0.8569 spec=0.8182 | 40.5s
    45→  New best model (loss=3.0754)
    46→  [17/200] step 450 | loss=2.9534 pred=0.1163 kl=1.1075 recon=0.8214 spec=0.7931 | grad=13.75 | 27.6s
    47→Epoch 17/200 | avg_loss=2.9850 pred=0.1171 kl=1.0986 recon=0.8310 spec=0.8009 | 40.9s
    48→  New best model (loss=2.9850)
    49→Epoch 18/200 | avg_loss=2.9984 pred=0.1021 kl=1.1966 recon=0.8366 spec=0.8119 | 40.5s
    50→  [19/200] step 500 | loss=2.9102 pred=0.0872 kl=1.2991 recon=0.7968 spec=0.8156 | grad=12.00 | 21.1s
    51→Epoch 19/200 | avg_loss=2.8936 pred=0.0886 kl=1.2971 recon=0.8056 spec=0.7920 | 40.6s
    52→  New best model (loss=2.8936)
    53→Epoch 20/200 | avg_loss=2.8444 pred=0.0758 kl=1.4101 recon=0.7931 spec=0.7837 | 40.7s
    54→  Saved checkpoint: /home/ubuntu/lewm-tts/output_v3c/checkpoints/epoch_0020.pt
    55→  New best model (loss=2.8444)
    56→  [21/200] step 550 | loss=2.7382 pred=0.0664 kl=1.5054 recon=0.7477 spec=0.7792 | grad=7.86 | 15.4s
    57→Epoch 21/200 | avg_loss=2.8097 pred=0.0648 kl=1.5170 recon=0.7812 spec=0.7832 | 40.7s
    58→  New best model (loss=2.8097)
    59→Epoch 22/200 | avg_loss=2.7211 pred=0.0561 kl=1.6157 recon=0.7555 spec=0.7635 | 40.8s
    60→  New best model (loss=2.7211)
    61→  [23/200] step 600 | loss=2.7167 pred=0.0525 kl=1.6758 recon=0.7557 spec=0.7624 | grad=10.03 | 9.4s
    62→Epoch 23/200 | avg_loss=2.6877 pred=0.0490 kl=1.7129 recon=0.7431 spec=0.7620 | 40.4s
    63→  New best model (loss=2.6877)
    64→Epoch 24/200 | avg_loss=2.6451 pred=0.0435 kl=1.8101 recon=0.7303 spec=0.7536 | 40.8s
    65→  New best model (loss=2.6451)
    66→  [25/200] step 650 | loss=2.6094 pred=0.0415 kl=1.8704 recon=0.7259 spec=0.7366 | grad=9.57 | 3.6s
    67→Epoch 25/200 | avg_loss=2.6333 pred=0.0396 kl=1.9033 recon=0.7274 spec=0.7515 | 40.7s
    68→  New best model (loss=2.6333)
    69→  [26/200] step 700 | loss=2.6726 pred=0.0363 kl=2.0206 recon=0.7374 spec=0.7656 | grad=9.21 | 37.5s
    70→Epoch 26/200 | avg_loss=2.6300 pred=0.0374 kl=1.9832 recon=0.7267 spec=0.7511 | 40.5s
    71→  New best model (loss=2.6300)
    72→Epoch 27/200 | avg_loss=2.6164 pred=0.0357 kl=2.0697 recon=0.7212 spec=0.7497 | 40.7s
    73→  New best model (loss=2.6164)
    74→  [28/200] step 750 | loss=2.5593 pred=0.0320 kl=2.1418 recon=0.7061 spec=0.7335 | grad=13.84 | 31.9s
    75→Epoch 28/200 | avg_loss=2.6100 pred=0.0332 kl=2.1325 recon=0.7218 spec=0.7457 | 40.8s
    76→  New best model (loss=2.6100)
    77→Epoch 29/200 | avg_loss=2.5911 pred=0.0311 kl=2.1768 recon=0.7150 spec=0.7429 | 40.5s
    78→  New best model (loss=2.5911)
    79→  [30/200] step 800 | loss=2.5686 pred=0.0305 kl=2.2163 recon=0.7134 spec=0.7299 | grad=10.11 | 25.6s
    80→Epoch 30/200 | avg_loss=2.5796 pred=0.0301 kl=2.2186 recon=0.7101 spec=0.7420 | 40.7s
    81→  Saved checkpoint: /home/ubuntu/lewm-tts/output_v3c/checkpoints/epoch_0030.pt
    82→  New best model (loss=2.5796)
    83→Epoch 31/200 | avg_loss=2.5567 pred=0.0290 kl=2.2608 recon=0.7027 spec=0.7367 | 40.5s
    84→  New best model (loss=2.5567)
    85→  [32/200] step 850 | loss=2.5136 pred=0.0283 kl=2.2881 recon=0.6939 spec=0.7197 | grad=6.13 | 20.0s
    86→Epoch 32/200 | avg_loss=2.5509 pred=0.0285 kl=2.2890 recon=0.7021 spec=0.7334 | 40.8s
    87→  New best model (loss=2.5509)
    88→Epoch 33/200 | avg_loss=2.5990 pred=0.0287 kl=2.3147 recon=0.7185 spec=0.7430 | 40.6s
    89→  [34/200] step 900 | loss=2.6683 pred=0.0284 kl=2.3074 recon=0.7330 spec=0.7698 | grad=14.52 | 13.8s
    90→Epoch 34/200 | avg_loss=2.6028 pred=0.0286 kl=2.3078 recon=0.7215 spec=0.7412 | 40.8s
    91→Epoch 35/200 | avg_loss=2.5449 pred=0.0293 kl=2.3040 recon=0.6980 spec=0.7332 | 40.6s
    92→  New best model (loss=2.5449)
    93→  [36/200] step 950 | loss=2.5300 pred=0.0277 kl=2.3304 recon=0.6851 spec=0.7410 | grad=5.92 | 8.0s
    94→Epoch 36/200 | avg_loss=2.5258 pred=0.0275 kl=2.3351 recon=0.6917 spec=0.7295 | 40.6s
    95→  New best model (loss=2.5258)
    96→Epoch 37/200 | avg_loss=2.5108 pred=0.0268 kl=2.3397 recon=0.6855 spec=0.7278 | 40.4s
    97→  New best model (loss=2.5108)
    98→  [38/200] step 1000 | loss=2.5287 pred=0.0251 kl=2.3660 recon=0.6945 spec=0.7285 | grad=6.34 | 2.0s
    99→Epoch 38/200 | avg_loss=2.5070 pred=0.0257 kl=2.3479 recon=0.6866 spec=0.7241 | 40.3s
   100→  New best model (loss=2.5070)

Device: cuda
LeWM TTS model: 21.11M parameters
Dataset: 6959 samples loaded
Training: 6959 samples, 27 batches/epoch
Epoch 1/200 | avg_loss=11.9772 pred=1.4899 kl=0.2497 recon=2.3543 spec=3.8524 | 42.9s
  New best model (loss=11.9772)
  [2/200] step 50 | loss=11.6261 pred=1.2251 kl=0.2035 recon=2.3549 spec=3.7941 | grad=3.04 | 35.2s
Epoch 2/200 | avg_loss=11.7592 pred=1.3041 kl=0.2161 recon=2.3541 spec=3.8312 | 41.1s
  New best model (loss=11.7592)
Epoch 3/200 | avg_loss=11.3993 pred=1.0685 kl=0.1916 recon=2.3521 spec=3.7509 | 40.2s
  New best model (loss=11.3993)
  [4/200] step 100 | loss=10.4622 pred=0.8959 kl=0.2493 recon=2.3514 spec=3.2423 | grad=3.24 | 28.7s
Epoch 4/200 | avg_loss=10.6588 pred=0.9144 kl=0.2344 recon=2.3339 spec=3.3843 | 40.8s
  New best model (loss=10.6588)
Epoch 5/200 | avg_loss=9.0671 pred=0.8269 kl=0.3122 recon=2.2122 spec=2.5436 | 40.3s
  New best model (loss=9.0671)
  [6/200] step 150 | loss=7.2285 pred=0.8280 kl=0.2964 recon=1.6269 spec=2.0975 | grad=7.31 | 22.7s
Epoch 6/200 | avg_loss=7.2588 pred=0.7820 kl=0.2954 recon=1.6706 spec=2.0902 | 40.6s
  New best model (loss=7.2588)
Epoch 7/200 | avg_loss=5.9076 pred=0.6632 kl=0.3169 recon=1.3503 spec=1.6955 | 40.6s
  New best model (loss=5.9076)
  [8/200] step 200 | loss=4.9208 pred=0.5572 kl=0.3621 recon=1.1829 spec=1.3314 | grad=4.71 | 16.9s
Epoch 8/200 | avg_loss=4.8213 pred=0.5439 kl=0.3731 recon=1.1619 spec=1.3019 | 40.8s
  New best model (loss=4.8213)
Epoch 9/200 | avg_loss=4.0268 pred=0.4353 kl=0.4836 recon=1.0010 spec=1.0590 | 40.4s
  New best model (loss=4.0268)
  [10/200] step 250 | loss=3.8342 pred=0.3734 kl=0.5448 recon=0.9780 spec=1.0024 | grad=32.58 | 10.9s
Epoch 10/200 | avg_loss=3.7727 pred=0.3652 kl=0.5560 recon=0.9683 spec=0.9797 | 40.8s
  Saved checkpoint: /home/ubuntu/lewm-tts/output_v3c/checkpoints/epoch_0010.pt
  New best model (loss=3.7727)
Epoch 11/200 | avg_loss=3.6309 pred=0.3330 kl=0.5902 recon=0.9486 spec=0.9328 | 40.5s
  New best model (loss=3.6309)
  [12/200] step 300 | loss=3.6239 pred=0.3101 kl=0.6098 recon=0.9584 spec=0.9303 | grad=26.33 | 4.9s
Epoch 12/200 | avg_loss=3.4695 pred=0.2884 kl=0.6412 recon=0.9204 spec=0.8922 | 40.6s
  New best model (loss=3.4695)
  [13/200] step 350 | loss=3.2157 pred=0.2090 kl=0.7828 recon=0.8850 spec=0.8227 | grad=14.26 | 39.2s
Epoch 13/200 | avg_loss=3.3178 pred=0.2341 kl=0.7302 recon=0.8977 spec=0.8574 | 40.8s
  New best model (loss=3.3178)
Epoch 14/200 | avg_loss=3.2315 pred=0.1837 kl=0.8377 recon=0.8898 spec=0.8436 | 40.9s
  New best model (loss=3.2315)
  [15/200] step 400 | loss=3.1591 pred=0.1422 kl=0.9733 recon=0.8933 spec=0.8179 | grad=22.70 | 33.5s
Epoch 15/200 | avg_loss=3.1433 pred=0.1514 kl=0.9374 recon=0.8741 spec=0.8269 | 40.9s
  New best model (loss=3.1433)
Epoch 16/200 | avg_loss=3.0754 pred=0.1303 kl=1.0223 recon=0.8569 spec=0.8182 | 40.5s
  New best model (loss=3.0754)
  [17/200] step 450 | loss=2.9534 pred=0.1163 kl=1.1075 recon=0.8214 spec=0.7931 | grad=13.75 | 27.6s
Epoch 17/200 | avg_loss=2.9850 pred=0.1171 kl=1.0986 recon=0.8310 spec=0.8009 | 40.9s
  New best model (loss=2.9850)
Epoch 18/200 | avg_loss=2.9984 pred=0.1021 kl=1.1966 recon=0.8366 spec=0.8119 | 40.5s
  [19/200] step 500 | loss=2.9102 pred=0.0872 kl=1.2991 recon=0.7968 spec=0.8156 | grad=12.00 | 21.1s
Epoch 19/200 | avg_loss=2.8936 pred=0.0886 kl=1.2971 recon=0.8056 spec=0.7920 | 40.6s
  New best model (loss=2.8936)
Epoch 20/200 | avg_loss=2.8444 pred=0.0758 kl=1.4101 recon=0.7931 spec=0.7837 | 40.7s
  Saved checkpoint: /home/ubuntu/lewm-tts/output_v3c/checkpoints/epoch_0020.pt
  New best model (loss=2.8444)
  [21/200] step 550 | loss=2.7382 pred=0.0664 kl=1.5054 recon=0.7477 spec=0.7792 | grad=7.86 | 15.4s
Epoch 21/200 | avg_loss=2.8097 pred=0.0648 kl=1.5170 recon=0.7812 spec=0.7832 | 40.7s
  New best model (loss=2.8097)
Epoch 22/200 | avg_loss=2.7211 pred=0.0561 kl=1.6157 recon=0.7555 spec=0.7635 | 40.8s
  New best model (loss=2.7211)
  [23/200] step 600 | loss=2.7167 pred=0.0525 kl=1.6758 recon=0.7557 spec=0.7624 | grad=10.03 | 9.4s
Epoch 23/200 | avg_loss=2.6877 pred=0.0490 kl=1.7129 recon=0.7431 spec=0.7620 | 40.4s
  New best model (loss=2.6877)
Epoch 24/200 | avg_loss=2.6451 pred=0.0435 kl=1.8101 recon=0.7303 spec=0.7536 | 40.8s
  New best model (loss=2.6451)
  [25/200] step 650 | loss=2.6094 pred=0.0415 kl=1.8704 recon=0.7259 spec=0.7366 | grad=9.57 | 3.6s
Epoch 25/200 | avg_loss=2.6333 pred=0.0396 kl=1.9033 recon=0.7274 spec=0.7515 | 40.7s
  New best model (loss=2.6333)
  [26/200] step 700 | loss=2.6726 pred=0.0363 kl=2.0206 recon=0.7374 spec=0.7656 | grad=9.21 | 37.5s
Epoch 26/200 | avg_loss=2.6300 pred=0.0374 kl=1.9832 recon=0.7267 spec=0.7511 | 40.5s
  New best model (loss=2.6300)
Epoch 27/200 | avg_loss=2.6164 pred=0.0357 kl=2.0697 recon=0.7212 spec=0.7497 | 40.7s
  New best model (loss=2.6164)
  [28/200] step 750 | loss=2.5593 pred=0.0320 kl=2.1418 recon=0.7061 spec=0.7335 | grad=13.84 | 31.9s
Epoch 28/200 | avg_loss=2.6100 pred=0.0332 kl=2.1325 recon=0.7218 spec=0.7457 | 40.8s
  New best model (loss=2.6100)
Epoch 29/200 | avg_loss=2.5911 pred=0.0311 kl=2.1768 recon=0.7150 spec=0.7429 | 40.5s
  New best model (loss=2.5911)
  [30/200] step 800 | loss=2.5686 pred=0.0305 kl=2.2163 recon=0.7134 spec=0.7299 | grad=10.11 | 25.6s
Epoch 30/200 | avg_loss=2.5796 pred=0.0301 kl=2.2186 recon=0.7101 spec=0.7420 | 40.7s
  Saved checkpoint: /home/ubuntu/lewm-tts/output_v3c/checkpoints/epoch_0030.pt
  New best model (loss=2.5796)
Epoch 31/200 | avg_loss=2.5567 pred=0.0290 kl=2.2608 recon=0.7027 spec=0.7367 | 40.5s
  New best model (loss=2.5567)
  [32/200] step 850 | loss=2.5136 pred=0.0283 kl=2.2881 recon=0.6939 spec=0.7197 | grad=6.13 | 20.0s
Epoch 32/200 | avg_loss=2.5509 pred=0.0285 kl=2.2890 recon=0.7021 spec=0.7334 | 40.8s
  New best model (loss=2.5509)
Epoch 33/200 | avg_loss=2.5990 pred=0.0287 kl=2.3147 recon=0.7185 spec=0.7430 | 40.6s
  [34/200] step 900 | loss=2.6683 pred=0.0284 kl=2.3074 recon=0.7330 spec=0.7698 | grad=14.52 | 13.8s
Epoch 34/200 | avg_loss=2.6028 pred=0.0286 kl=2.3078 recon=0.7215 spec=0.7412 | 40.8s
Epoch 35/200 | avg_loss=2.5449 pred=0.0293 kl=2.3040 recon=0.6980 spec=0.7332 | 40.6s
  New best model (loss=2.5449)
  [36/200] step 950 | loss=2.5300 pred=0.0277 kl=2.3304 recon=0.6851 spec=0.7410 | grad=5.92 | 8.0s
Epoch 36/200 | avg_loss=2.5258 pred=0.0275 kl=2.3351 recon=0.6917 spec=0.7295 | 40.6s
  New best model (loss=2.5258)
Epoch 37/200 | avg_loss=2.5108 pred=0.0268 kl=2.3397 recon=0.6855 spec=0.7278 | 40.4s
  New best model (loss=2.5108)
  [38/200] step 1000 | loss=2.5287 pred=0.0251 kl=2.3660 recon=0.6945 spec=0.7285 | grad=6.34 | 2.0s
Epoch 38/200 | avg_loss=2.5070 pred=0.0257 kl=2.3479 recon=0.6866 spec=0.7241 | 40.3s
  New best model (loss=2.5070)


     1→# LeWM TTS Project
     2→
     3→## Overview
     4→Adapting LeWM (LeWorldModel) JEPA architecture for Text-to-Speech synthesis.
     5→- Paper: https://le-wm.github.io/
     6→- Project folder: `/home/ubuntu/lewm-tts`
     7→- Goal: 10-20 speaker multi-speaker TTS on this 14M→24M model
     8→
     9→## Architecture (v3c — Current)
    10→- **TextEncoder**: Byte-level char transformer (4 layers, 256d, 4 heads)
    11→- **AudioEncoder**: 1D CNN + Transformer (4 layers) → mu/logvar → reparameterization
    12→  - **downsample_factor**: configurable (2 or 4), stored in config dict
    13→- **JEPAPredictor**: Transformer decoder (6 layers) with causal self-attn + cross-attn to text + **KV-cache**
    14→- **MelDecoder**: Linear→512 + ConvTranspose1d + ResConvBlock residual blocks + refinement conv
    15→  - **upsample_factor**: matches downsample_factor
    16→- **Vocoder**: Direct Vocos decode (model outputs 100 mels matching `charactr/vocos-mel-24khz`)
    17→- **4 losses**: prediction (MSE), KL (Gaussian reg), reconstruction (L1 mel), multi-resolution spectral loss
    18→- **Speaker embedding**: support for n_speakers > 1 (not yet trained)
    19→- **KL annealing**: ramps up over first 20% of training
    20→- Mel preprocessing matches Vocos exactly (verified: mel diff = 0.0000)
    21→
    22→## Training Runs
    23→- **v3** (4x downsample, kl=0.1, recon=1.0, spec=0.5): 24M params. Robotic + noise.
    24→- **v3b** (4x downsample, kl=0.01, recon=2.0, spec=1.5): 24M params. Voice audible but noisy at ep90 (recon=0.56). Killed at ep96.
    25→- **v3c** (2x downsample, kl=0.01, recon=2.0, spec=1.5): 21.1M params. **Currently training** at output_v3c/. ~41s/epoch. At ep110 recon≈0.54, voice coming through with noise.
    26→
    27→## Key Findings (This Session)
    28→- **KL weight 0.1 too aggressive** — crushes latent space, mel decoder can't reconstruct sharp formants
    29→- **4x downsample bottleneck** — decoder must hallucinate fine detail for 4 frames per latent. 2x is better.
    30→- **Model error is structured** (blurred formants), not random — L1=0.51 model recon sounds worse than L1=0.84 random noise on GT mel
    31→- At L1≈0.51 both random noise and model recon produce voice with noise — decoder IS learning, just needs lower error
    32→- **AR synthesis collapses** — predictor pred_loss=0.02 (TF) but AR free-running converges to silence. Needs scheduled sampling.
    33→- TF prediction ≈ reconstruction quality (predictor is fine, bottleneck is mel decoder)
    34→
    35→## Files
    36→- `preprocess.py` — segments audio, extracts mels, saves manifest
    37→- `model.py` — all model components, now supports configurable downsample_factor (2 or 4)
    38→- `dataset.py` — TTSDataset, collate_fn, build_dataloader
    39→- `train.py` — training loop, now supports --downsample_factor arg
    40→- `inference.py` — AR generation with Vocos vocoder (single request)
    41→- `inference_engine.py` — Production engine: batched inference, KV-cache, FP16, voice prompts
    42→
    43→## Dataset
    44→- **Raw**: `/home/ubuntu/modi_dataset` (2,330 clips, 18.25h, Hindi, 24kHz)
    45→- **100-mel processed**: `/home/ubuntu/lewm-tts/processed_data_100mel` (7,009 segments, 17.88h)
    46→- Mel config: n_fft=1024, hop=256, n_mels=100, sr=24000, f_max=None, power=1.0, clip_val=1e-7
    47→
    48→## Next Steps
    49→1. Let v3c finish 200 epochs, test quality
    50→2. Add **scheduled sampling** to training (fix AR collapse)
    51→3. If mel quality sufficient, proceed to multi-speaker training
    52→

# LeWM TTS Project

## Overview
Adapting LeWM (LeWorldModel) JEPA architecture for Text-to-Speech synthesis.
- Paper: https://le-wm.github.io/
- Project folder: `/home/ubuntu/lewm-tts`
- Goal: 10-20 speaker multi-speaker TTS on this 14M→24M model

## Architecture (v3c — Current)
- **TextEncoder**: Byte-level char transformer (4 layers, 256d, 4 heads)
- **AudioEncoder**: 1D CNN + Transformer (4 layers) → mu/logvar → reparameterization
  - **downsample_factor**: configurable (2 or 4), stored in config dict
- **JEPAPredictor**: Transformer decoder (6 layers) with causal self-attn + cross-attn to text + **KV-cache**
- **MelDecoder**: Linear→512 + ConvTranspose1d + ResConvBlock residual blocks + refinement conv
  - **upsample_factor**: matches downsample_factor
- **Vocoder**: Direct Vocos decode (model outputs 100 mels matching `charactr/vocos-mel-24khz`)
- **4 losses**: prediction (MSE), KL (Gaussian reg), reconstruction (L1 mel), multi-resolution spectral loss
- **Speaker embedding**: support for n_speakers > 1 (not yet trained)
- **KL annealing**: ramps up over first 20% of training
- Mel preprocessing matches Vocos exactly (verified: mel diff = 0.0000)

## Training Runs
- **v3** (4x downsample, kl=0.1, recon=1.0, spec=0.5): 24M params. Robotic + noise.
- **v3b** (4x downsample, kl=0.01, recon=2.0, spec=1.5): 24M params. Voice audible but noisy at ep90 (recon=0.56). Killed at ep96.
- **v3c** (2x downsample, kl=0.01, recon=2.0, spec=1.5): 21.1M params. **Currently training** at output_v3c/. ~41s/epoch. At ep110 recon≈0.54, voice coming through with noise.

## Key Findings (This Session)
- **KL weight 0.1 too aggressive** — crushes latent space, mel decoder can't reconstruct sharp formants
- **4x downsample bottleneck** — decoder must hallucinate fine detail for 4 frames per latent. 2x is better.
- **Model error is structured** (blurred formants), not random — L1=0.51 model recon sounds worse than L1=0.84 random noise on GT mel
- At L1≈0.51 both random noise and model recon produce voice with noise — decoder IS learning, just needs lower error
- **AR synthesis collapses** — predictor pred_loss=0.02 (TF) but AR free-running converges to silence. Needs scheduled sampling.
- TF prediction ≈ reconstruction quality (predictor is fine, bottleneck is mel decoder)

## Files
- `preprocess.py` — segments audio, extracts mels, saves manifest
- `model.py` — all model components, now supports configurable downsample_factor (2 or 4)
- `dataset.py` — TTSDataset, collate_fn, build_dataloader
- `train.py` — training loop, now supports --downsample_factor arg
- `inference.py` — AR generation with Vocos vocoder (single request)
- `inference_engine.py` — Production engine: batched inference, KV-cache, FP16, voice prompts

## Dataset
- **Raw**: `/home/ubuntu/modi_dataset` (2,330 clips, 18.25h, Hindi, 24kHz)
- **100-mel processed**: `/home/ubuntu/lewm-tts/processed_data_100mel` (7,009 segments, 17.88h)
- Mel config: n_fft=1024, hop=256, n_mels=100, sr=24000, f_max=None, power=1.0, clip_val=1e-7

## Next Steps
1. Let v3c finish 200 epochs, test quality
2. Add **scheduled sampling** to training (fix AR collapse)
3. If mel quality sufficient, proceed to multi-speaker training


/home/ubuntu/lewm-tts/dataset.py
/home/ubuntu/lewm-tts/v2/dataset.py
/home/ubuntu/lewm-tts/v2/preprocess.py
/home/ubuntu/lewm-tts/v2/inference.py
/home/ubuntu/lewm-tts/v2/model.py
/home/ubuntu/lewm-tts/v2/train.py
/home/ubuntu/lewm-tts/preprocess.py
/home/ubuntu/lewm-tts/inference.py
/home/ubuntu/lewm-tts/inference_engine.py
/home/ubuntu/lewm-tts/model.py
/home/ubuntu/lewm-tts/train.py


Let me check the end of the log to find the latest epoch, and look at the inference script.


Bash

Get last 30 lines of training log


ls -la /home/ubuntu/lewm-tts/output_v3c/checkpoints/ 2>/dev/null

List available checkpoints


     1→"""
     2→Inference pipeline for LeWM TTS.
     3→Text → JEPA predictor → mel reconstruction → Vocos vocoder → waveform
     4→"""
     5→
     6→import torch
     7→import torchaudio
     8→import numpy as np
     9→import argparse
    10→from pathlib import Path
    11→
    12→from model import LeWMTTS
    13→
    14→
    15→class LeWMTTSInference:
    16→    """Full inference pipeline."""
    17→
    18→    def __init__(self, checkpoint_path, device="cuda"):
    19→        self.device = torch.device(device if torch.cuda.is_available() else "cpu")
    20→
    21→        # Load checkpoint
    22→        ckpt = torch.load(checkpoint_path, map_location=self.device, weights_only=False)
    23→        config = ckpt["config"]
    24→
    25→        # Build model
    26→        self.model = LeWMTTS(config).to(self.device)
    27→        self.model.load_state_dict(ckpt["model"])
    28→        self.model.eval()
    29→
    30→        # Vocos vocoder (directly decodes 100-mel)
    31→        from vocos import Vocos
    32→        self.vocos = Vocos.from_pretrained("charactr/vocos-mel-24khz")
    33→        self.vocos.eval()
    34→
    35→        self.config = config
    36→        self._start_emb_scale = 5.0
    37→        print(f"Model loaded from {checkpoint_path}")
    38→
    39→    def vocos_decode(self, mel):
    40→        """Decode mel [B, 100, T] → waveform via Vocos."""
    41→        with torch.no_grad():
    42→            waveform = self.vocos.decode(mel)  # [B, T_audio]
    43→        return waveform
    44→
    45→    def text_to_tokens(self, text):
    46→        tokens = list(text.encode("utf-8"))
    47→        return torch.tensor(tokens, dtype=torch.long).unsqueeze(0).to(self.device)
    48→
    49→    @torch.no_grad()
    50→    def synthesize(self, text, max_steps=300, temperature=0.0):
    51→        """Generate speech from text using AR prediction with KV-cache."""
    52→        text_tokens = self.text_to_tokens(text)
    53→        text_emb = self.model.text_encoder(text_tokens)
    54→
    55→        d = self.config["d_model"]
    56→        start_emb = torch.randn(1, 1, d, device=self.device) * self._start_emb_scale
    57→        all_embeddings = [start_emb]
    58→
    59→        cache = self.model.init_ar_cache()
    60→        next_emb, cache = self.model.predict_next_cached(start_emb, text_emb, 0, cache)
    61→        if temperature > 0:
    62→            next_emb = next_emb + torch.randn_like(next_emb) * temperature
    63→        all_embeddings.append(next_emb)
    64→
    65→        prev_norms = [next_emb.norm().item()]
    66→
    67→        for step in range(1, max_steps):
    68→            next_emb, cache = self.model.predict_next_cached(next_emb, text_emb, step, cache)
    69→            if temperature > 0:
    70→                next_emb = next_emb + torch.randn_like(next_emb) * temperature
    71→            all_embeddings.append(next_emb)
    72→
    73→            norm = next_emb.norm().item()
    74→            prev_norms.append(norm)
    75→            if len(prev_norms) > 20:
    76→                recent = prev_norms[-20:]
    77→                if np.std(recent) < 0.01 * np.mean(recent) and step > 30:
    78→                    break
    79→
    80→        audio_embs = torch.cat(all_embeddings, dim=1)
    81→        mel = self.model.mel_decoder(audio_embs)
    82→        waveform = self.vocos_decode(mel.cpu())
    83→        return waveform.squeeze().numpy(), 24000
    84→
    85→    @torch.no_grad()
    86→    def reconstruct(self, mel_path):
    87→        """Teacher-forced reconstruction: encode real mel → decode back."""
    88→        mel = torch.load(mel_path, weights_only=True).unsqueeze(0).to(self.device)
    89→
    90→        z, mu, logvar = self.model.audio_encoder(mel)
    91→        mel_recon = self.model.mel_decoder(mu)
    92→
    93→        waveform = self.vocos_decode(mel_recon.cpu())
    94→        return waveform.squeeze().numpy(), 24000
    95→
    96→    @torch.no_grad()
    97→    def reconstruct_with_prediction(self, mel_path):
    98→        """Hybrid: encode real mel → predictor → decode predicted embeddings."""
    99→        mel = torch.load(mel_path, weights_only=True).unsqueeze(0).to(self.device)
   100→
   101→        z, mu, logvar = self.model.audio_encoder(mel)
   102→
   103→        text_tokens = torch.zeros(1, 1, dtype=torch.long, device=self.device)
   104→        text_emb = self.model.text_encoder(text_tokens)
   105→
   106→        predicted = self.model.predictor(mu[:, :-1], text_emb)
   107→        combined = torch.cat([mu[:, :1], predicted], dim=1)
   108→
   109→        mel_recon = self.model.mel_decoder(combined)
   110→        waveform = self.vocos_decode(mel_recon.cpu())
   111→        return waveform.squeeze().numpy(), 24000
   112→
   113→    def save_audio(self, waveform, sr, output_path):
   114→        # Normalize
   115→        mx = np.abs(waveform).max()
   116→        if mx > 0:
   117→            waveform = waveform / mx * 0.95
   118→        wav_tensor = torch.from_numpy(waveform).unsqueeze(0)
   119→        torchaudio.save(output_path, wav_tensor, sr)
   120→        print(f"Saved: {output_path} ({len(waveform)/sr:.2f}s)")
   121→
   122→
   123→def main():
   124→    parser = argparse.ArgumentParser()
   125→    parser.add_argument("--checkpoint", required=True)
   126→    parser.add_argument("--text", default=None, help="Text to synthesize")
   127→    parser.add_argument("--mel", default=None, help="Mel path for reconstruction test")
   128→    parser.add_argument("--mode", choices=["synthesize", "reconstruct", "predict"],
   129→                        default="synthesize")
   130→    parser.add_argument("--output", default="output.wav")
   131→    parser.add_argument("--max_steps", type=int, default=300)
   132→    parser.add_argument("--temperature", type=float, default=0.0)
   133→    args = parser.parse_args()
   134→
   135→    tts = LeWMTTSInference(args.checkpoint)
   136→
   137→    if args.mode == "synthesize":
   138→        assert args.text, "Need --text for synthesis"
   139→        waveform, sr = tts.synthesize(args.text, args.max_steps, args.temperature)
   140→    elif args.mode == "reconstruct":
   141→        assert args.mel, "Need --mel for reconstruction"
   142→        waveform, sr = tts.reconstruct(args.mel)
   143→    elif args.mode == "predict":
   144→        assert args.mel, "Need --mel for prediction test"
   145→        waveform, sr = tts.reconstruct_with_prediction(args.mel)
   146→
   147→    tts.save_audio(waveform, sr, args.output)
   148→
   149→
   150→if __name__ == "__main__":
   151→    main()
   152→

"""
Inference pipeline for LeWM TTS.
Text → JEPA predictor → mel reconstruction → Vocos vocoder → waveform
"""

import torch
import torchaudio
import numpy as np
import argparse
from pathlib import Path

from model import LeWMTTS

class LeWMTTSInference:
    """Full inference pipeline."""

    def __init__(self, checkpoint_path, device="cuda"):
        self.device = torch.device(device if torch.cuda.is_available() else "cpu")

        # Load checkpoint
        ckpt = torch.load(checkpoint_path, map_location=self.device, weights_only=False)
        config = ckpt["config"]

        # Build model
        self.model = LeWMTTS(config).to(self.device)
        self.model.load_state_dict(ckpt["model"])
        self.model.eval()

        # Vocos vocoder (directly decodes 100-mel)
        from vocos import Vocos
        self.vocos = Vocos.from_pretrained("charactr/vocos-mel-24khz")
        self.vocos.eval()

        self.config = config
        self._start_emb_scale = 5.0
        print(f"Model loaded from {checkpoint_path}")

    def vocos_decode(self, mel):
        """Decode mel [B, 100, T] → waveform via Vocos."""
        with torch.no_grad():
            waveform = self.vocos.decode(mel)  # [B, T_audio]
        return waveform

    def text_to_tokens(self, text):
        tokens = list(text.encode("utf-8"))
        return torch.tensor(tokens, dtype=torch.long).unsqueeze(0).to(self.device)

    @torch.no_grad()
    def synthesize(self, text, max_steps=300, temperature=0.0):
        """Generate speech from text using AR prediction with KV-cache."""
        text_tokens = self.text_to_tokens(text)
        text_emb = self.model.text_encoder(text_tokens)

        d = self.config["d_model"]
        start_emb = torch.randn(1, 1, d, device=self.device) * self._start_emb_scale
        all_embeddings = [start_emb]

        cache = self.model.init_ar_cache()
        next_emb, cache = self.model.predict_next_cached(start_emb, text_emb, 0, cache)
        if temperature > 0:
            next_emb = next_emb + torch.randn_like(next_emb) * temperature
        all_embeddings.append(next_emb)

        prev_norms = [next_emb.norm().item()]

        for step in range(1, max_steps):
            next_emb, cache = self.model.predict_next_cached(next_emb, text_emb, step, cache)
            if temperature > 0:
                next_emb = next_emb + torch.randn_like(next_emb) * temperature
            all_embeddings.append(next_emb)

            norm = next_emb.norm().item()
            prev_norms.append(norm)
            if len(prev_norms) > 20:
                recent = prev_norms[-20:]
                if np.std(recent) < 0.01 * np.mean(recent) and step > 30:
                    break

        audio_embs = torch.cat(all_embeddings, dim=1)
        mel = self.model.mel_decoder(audio_embs)
        waveform = self.vocos_decode(mel.cpu())
        return waveform.squeeze().numpy(), 24000

    @torch.no_grad()
    def reconstruct(self, mel_path):
        """Teacher-forced reconstruction: encode real mel → decode back."""
        mel = torch.load(mel_path, weights_only=True).unsqueeze(0).to(self.device)

        z, mu, logvar = self.model.audio_encoder(mel)
        mel_recon = self.model.mel_decoder(mu)

        waveform = self.vocos_decode(mel_recon.cpu())
        return waveform.squeeze().numpy(), 24000

    @torch.no_grad()
    def reconstruct_with_prediction(self, mel_path):
        """Hybrid: encode real mel → predictor → decode predicted embeddings."""
        mel = torch.load(mel_path, weights_only=True).unsqueeze(0).to(self.device)

        z, mu, logvar = self.model.audio_encoder(mel)

        text_tokens = torch.zeros(1, 1, dtype=torch.long, device=self.device)
        text_emb = self.model.text_encoder(text_tokens)

        predicted = self.model.predictor(mu[:, :-1], text_emb)
        combined = torch.cat([mu[:, :1], predicted], dim=1)

        mel_recon = self.model.mel_decoder(combined)
        waveform = self.vocos_decode(mel_recon.cpu())
        return waveform.squeeze().numpy(), 24000

    def save_audio(self, waveform, sr, output_path):
        # Normalize
        mx = np.abs(waveform).max()
        if mx > 0:
            waveform = waveform / mx * 0.95
        wav_tensor = torch.from_numpy(waveform).unsqueeze(0)
        torchaudio.save(output_path, wav_tensor, sr)
        print(f"Saved: {output_path} ({len(waveform)/sr:.2f}s)")

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--checkpoint", required=True)
    parser.add_argument("--text", default=None, help="Text to synthesize")
    parser.add_argument("--mel", default=None, help="Mel path for reconstruction test")
    parser.add_argument("--mode", choices=["synthesize", "reconstruct", "predict"],
                        default="synthesize")
    parser.add_argument("--output", default="output.wav")
    parser.add_argument("--max_steps", type=int, default=300)
    parser.add_argument("--temperature", type=float, default=0.0)
    args = parser.parse_args()

    tts = LeWMTTSInference(args.checkpoint)

    if args.mode == "synthesize":
        assert args.text, "Need --text for synthesis"
        waveform, sr = tts.synthesize(args.text, args.max_steps, args.temperature)
    elif args.mode == "reconstruct":
        assert args.mel, "Need --mel for reconstruction"
        waveform, sr = tts.reconstruct(args.mel)
    elif args.mode == "predict":
        assert args.mel, "Need --mel for prediction test"
        waveform, sr = tts.reconstruct_with_prediction(args.mel)

    tts.save_audio(waveform, sr, args.output)

if __name__ == "__main__":
    main()


Epoch 187/200 | avg_loss=1.9771 pred=0.0171 kl=2.4528 recon=0.5028 spec=0.6199 | 40.8s
  New best model (loss=1.9771)
  [188/200] step 5050 | loss=1.9892 pred=0.0170 kl=2.4546 recon=0.5050 spec=0.6252 | grad=0.89 | 2.0s
Epoch 188/200 | avg_loss=1.9775 pred=0.0172 kl=2.4517 recon=0.5027 spec=0.6203 | 40.7s
  [189/200] step 5100 | loss=1.9634 pred=0.0172 kl=2.4512 recon=0.5033 spec=0.6101 | grad=3.41 | 36.1s
Epoch 189/200 | avg_loss=1.9782 pred=0.0172 kl=2.4519 recon=0.5027 spec=0.6207 | 40.5s
Epoch 190/200 | avg_loss=1.9801 pred=0.0171 kl=2.4521 recon=0.5027 spec=0.6219 | 40.7s
  Saved checkpoint: /home/ubuntu/lewm-tts/output_v3c/checkpoints/epoch_0190.pt
  [191/200] step 5150 | loss=1.9591 pred=0.0170 kl=2.4533 recon=0.5012 spec=0.6101 | grad=1.52 | 30.2s
Epoch 191/200 | avg_loss=1.9790 pred=0.0171 kl=2.4524 recon=0.5026 spec=0.6214 | 40.6s
Epoch 192/200 | avg_loss=1.9759 pred=0.0171 kl=2.4522 recon=0.5026 spec=0.6193 | 40.9s
  New best model (loss=1.9759)
  [193/200] step 5200 | loss=1.9515 pred=0.0170 kl=2.4544 recon=0.5019 spec=0.6041 | grad=1.74 | 24.2s
Epoch 193/200 | avg_loss=1.9763 pred=0.0171 kl=2.4524 recon=0.5025 spec=0.6197 | 40.5s
Epoch 194/200 | avg_loss=1.9755 pred=0.0171 kl=2.4521 recon=0.5025 spec=0.6192 | 40.6s
  New best model (loss=1.9755)
  [195/200] step 5250 | loss=1.9719 pred=0.0171 kl=2.4506 recon=0.5019 spec=0.6177 | grad=1.07 | 18.4s
Epoch 195/200 | avg_loss=1.9780 pred=0.0171 kl=2.4521 recon=0.5026 spec=0.6208 | 40.6s
Epoch 196/200 | avg_loss=1.9765 pred=0.0171 kl=2.4522 recon=0.5025 spec=0.6199 | 40.7s
  [197/200] step 5300 | loss=1.9615 pred=0.0171 kl=2.4543 recon=0.5012 spec=0.6117 | grad=0.89 | 12.5s
Epoch 197/200 | avg_loss=1.9785 pred=0.0171 kl=2.4519 recon=0.5026 spec=0.6211 | 40.6s
Epoch 198/200 | avg_loss=1.9734 pred=0.0171 kl=2.4519 recon=0.5026 spec=0.6177 | 40.9s
  New best model (loss=1.9734)
  [199/200] step 5350 | loss=1.9805 pred=0.0172 kl=2.4502 recon=0.5032 spec=0.6216 | grad=1.39 | 6.6s
Epoch 199/200 | avg_loss=1.9756 pred=0.0171 kl=2.4519 recon=0.5025 spec=0.6193 | 40.9s
  [200/200] step 5400 | loss=1.9890 pred=0.0171 kl=2.4549 recon=0.5023 spec=0.6285 | grad=1.26 | 40.6s
Epoch 200/200 | avg_loss=1.9766 pred=0.0171 kl=2.4520 recon=0.5026 spec=0.6199 | 40.6s
  Saved checkpoint: /home/ubuntu/lewm-tts/output_v3c/checkpoints/epoch_0200.pt

Training complete. Best loss: 1.9734


total 5251548
drwxrwxr-x 2 ubuntu ubuntu      4096 Mar 24 09:19 .
drwxrwxr-x 4 ubuntu ubuntu      4096 Mar 24 06:54 ..
-rw-rw-r-- 1 ubuntu ubuntu  95012157 Mar 24 09:18 best.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 07:10 epoch_0010.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 07:17 epoch_0020.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 07:24 epoch_0030.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 07:30 epoch_0040.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 07:37 epoch_0050.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 07:44 epoch_0060.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 07:51 epoch_0070.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 07:58 epoch_0080.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 08:04 epoch_0090.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 08:11 epoch_0100.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 08:18 epoch_0110.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 08:25 epoch_0120.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 08:32 epoch_0130.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 08:39 epoch_0140.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 08:45 epoch_0150.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 08:52 epoch_0160.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 08:59 epoch_0170.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 09:06 epoch_0180.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 09:12 epoch_0190.pt
-rw-rw-r-- 1 ubuntu ubuntu 264126088 Mar 24 09:19 epoch_0200.pt


Training is complete at epoch 200. Latest checkpoint is `epoch_0200.pt` and there's also a `best.pt`. Let me run samples using the inference script. From memory, AR synthesis collapses, so I should run both reconstruction and synthesis modes. Let me get a sample mel file and run inference.


Training completed — 200/200 epochs, best loss 1.9734 (recon=0.50, spec=0.62). Latest checkpoint is `epoch_0200.pt`, and `best.pt` is also available.

Let me grab a sample mel and run inference in all 3 modes.


ls /home/ubuntu/lewm-tts/processed_data_100mel/ | head -5

List processed data directory structure


audio
config.json
manifest.json
mels


ls /home/ubuntu/lewm-tts/processed_data_100mel/mels/ | head -5

List mel files


000000.pt
000001.pt
000002.pt
000003.pt
000004.pt


cd /home/ubuntu/lewm-tts && python -c "
import json
with open('processed_data_100mel/manifest.json') as f:
    manifest = json.load(f)
# Find a mid-length sample
for i, entry in enumerate(manifest[:10]):
    print(f'{i}: mel={entry.get(\"mel_path\",\"?\")}, text={entry.get(\"text\",\"?\")[:80]}, dur={entry.get(\"duration\",\"?\")}')" 2>&1

Check manifest for sample entries with text


Error: Exit code 127
/bin/bash: line 7: python: command not found


which python3 && cd /home/ubuntu/lewm-tts && python3 -c "
import json
with open('processed_data_100mel/manifest.json') as f:
    manifest = json.load(f)
for i, entry in enumerate(manifest[:10]):
    print(f'{i}: {entry}')" 2>&1

Check manifest entries


/usr/bin/python3
0: {'id': 0, 'audio_path': '/home/ubuntu/lewm-tts/processed_data_100mel/audio/000000.wav', 'mel_path': '/home/ubuntu/lewm-tts/processed_data_100mel/mels/000000.pt', 'text': 'मेरे प्यारे देशवासियों, नमस्कार। मन की बात में आपका स्वागत है, अभिन', 'duration': 9.52, 'mel_frames': 893}
1: {'id': 1, 'audio_path': '/home/ubuntu/lewm-tts/processed_data_100mel/audio/000001.wav', 'mel_path': '/home/ubuntu/lewm-tts/processed_data_100mel/mels/000001.pt', 'text': 'ंदन है। मन की बात देश और देशवासियों की उपलब्धियों को सामने लान', 'duration': 8.8, 'mel_frames': 826}
2: {'id': 2, 'audio_path': '/home/ubuntu/lewm-tts/processed_data_100mel/audio/000002.wav', 'mel_path': '/home/ubuntu/lewm-tts/processed_data_100mel/mels/000002.pt', 'text': 'े का एक मजबूत platform है। देश ने ऐसी ही उपलब्धि अभी दिल्ली में ह', 'duration': 9.14, 'mel_frames': 857}
3: {'id': 3, 'audio_path': '/home/ubuntu/lewm-tts/processed_data_100mel/audio/000003.wav', 'mel_path': '/home/ubuntu/lewm-tts/processed_data_100mel/mels/000003.pt', 'text': 'Global AI Impact Summit के दौरान देखी। कई देशों के नेता, उद्योग जगत के leade', 'duration': 9.09, 'mel_frames': 853}
4: {'id': 4, 'audio_path': '/home/ubuntu/lewm-tts/processed_data_100mel/audio/000004.wav', 'mel_path': '/home/ubuntu/lewm-tts/processed_data_100mel/mels/000004.pt', 'text': 'rs, innovators, और startup sector से जुड़े लोग, AI Impact Summit के लिए भारत मंडपम मे', 'duration': 10.14, 'mel_frames': 951}
5: {'id': 5, 'audio_path': '/home/ubuntu/lewm-tts/processed_data_100mel/audio/000005.wav', 'mel_path': '/home/ubuntu/lewm-tts/processed_data_100mel/mels/000005.pt', 'text': 'ं एकत्र हुए। आने वाले समय में, AI की शक्ति का उपयोग दुनिया किस प्रकार करेग', 'duration': 8.83, 'mel_frames': 828}
6: {'id': 6, 'audio_path': '/home/ubuntu/lewm-tts/processed_data_100mel/audio/000006.wav', 'mel_path': '/home/ubuntu/lewm-tts/processed_data_100mel/mels/000006.pt', 'text': 'दुनिया भर के leaders को बहुत प्रभावित किया। पहला product Amul के booth पर था। इसमें', 'duration': 10.38, 'mel_frames': 974}
7: {'id': 7, 'audio_path': '/home/ubuntu/lewm-tts/processed_data_100mel/audio/000007.wav', 'mel_path': '/home/ubuntu/lewm-tts/processed_data_100mel/mels/000007.pt', 'text': 'बताया गया कि कैसे AI जानवरों का इलाज करने में हमारी मदद कर रही है। और कैसे', 'duration': 9.37, 'mel_frames': 879}
8: {'id': 8, 'audio_path': '/home/ubuntu/lewm-tts/processed_data_100mel/audio/000008.wav', 'mel_path': '/home/ubuntu/lewm-tts/processed_data_100mel/mels/000008.pt', 'text': '24/7 AI assistant की मदद से किसान अपनी dairy और जानवर का हिसाब रखते हैं', 'duration': 8.79, 'mel_frames': 825}
9: {'id': 9, 'audio_path': '/home/ubuntu/lewm-tts/processed_data_100mel/audio/000009.wav', 'mel_path': '/home/ubuntu/lewm-tts/processed_data_100mel/mels/000009.pt', 'text': 'साथियों, दूसरा product हमारी संस्कृति से संबंधित था। दुनिया भर के leader', 'duration': 8.15, 'mel_frames': 765}


Good. Let me run all 3 inference modes on the first sample using the best checkpoint.


mkdir -p samples_v3c && python3 inference.py --checkpoint output_v3c/checkpoints/best.pt --mel processed_data_100mel/mels/000000.pt --mode reconstruct --output samples_v3c/recon_best.wav 2>&1

Run reconstruction on sample 0 with best checkpoint


Model loaded from output_v3c/checkpoints/best.pt
Saved: samples_v3c/recon_best.wav (9.50s)


python3 inference.py --checkpoint output_v3c/checkpoints/best.pt --mel processed_data_100mel/mels/000000.pt --mode predict --output samples_v3c/predict_best.wav 2>&1

Run predict mode (TF predictor + decode) on sample 0


python3 inference.py --checkpoint output_v3c/checkpoints/best.pt --text "मेरे प्यारे देशवासियों, नमस्कार।" --mode synthesize --output samples_v3c/synth_best.wav --max_steps 300 --temperature 0.0 2>&1

Run AR synthesis from text with best checkpoint


Model loaded from output_v3c/checkpoints/best.pt
Saved: samples_v3c/predict_best.wav (9.50s)


[W324 10:25:05.680560209 NNPACK.cpp:56] Could not initialize NNPACK! Reason: Unsupported hardware.
Model loaded from output_v3c/checkpoints/best.pt
Saved: samples_v3c/synth_best.wav (1.25s)


Also let me run a synthesis with temperature and try the epoch_0200 checkpoint for comparison.


python3 inference.py --checkpoint output_v3c/checkpoints/best.pt --text "मेरे प्यारे देशवासियों, नमस्कार।" --mode synthesize --output samples_v3c/synth_best_t05.wav --max_steps 300 --temperature 0.5 2>&1

AR synthesis with temperature 0.5


python3 inference.py --checkpoint output_v3c/checkpoints/epoch_0200.pt --mel processed_data_100mel/mels/000000.pt --mode reconstruct --output samples_v3c/recon_ep200.wav 2>&1

Reconstruction with epoch 200 checkpoint


Model loaded from output_v3c/checkpoints/best.pt
Saved: samples_v3c/synth_best_t05.wav (6.41s)


Model loaded from output_v3c/checkpoints/epoch_0200.pt
Saved: samples_v3c/recon_ep200.wav (9.50s)


List all generated samples


total 1716
drwxrwxr-x  2 ubuntu ubuntu   4096 Mar 24 10:25 .
drwxrwxr-x 11 ubuntu ubuntu   4096 Mar 24 10:24 ..
-rw-rw-r--  1 ubuntu ubuntu 456270 Mar 24 10:25 predict_best.wav
-rw-rw-r--  1 ubuntu ubuntu 456270 Mar 24 10:24 recon_best.wav
-rw-rw-r--  1 ubuntu ubuntu 456270 Mar 24 10:25 recon_ep200.wav
-rw-rw-r--  1 ubuntu ubuntu 307790 Mar 24 10:25 synth_best_t05.wav
-rw-rw-r--  1 ubuntu ubuntu  59982 Mar 24 10:25 synth_best.wav


Here's a summary of the v3c samples generated from the **best checkpoint** (epoch 198, loss=1.9734):

| File | Mode | Duration | Notes |
|------|------|----------|-------|
| `recon_best.wav` | Reconstruct (encode→decode) | 9.50s | Direct mel decoder quality test |
| `recon_ep200.wav` | Reconstruct (ep200) | 9.50s | Comparison with final epoch |
| `predict_best.wav` | Predict (TF predictor→decode) | 9.50s | Predictor + decoder together |
| `synth_best.wav` | AR synthesis (temp=0.0) | **1.25s** | Collapsed quickly — AR still broken |
| `synth_best_t05.wav` | AR synthesis (temp=0.5) | 6.41s | Temperature helps sustain generation |

**Key observations from the run:**
- **Reconstruction** works (9.5s output matching input length) — recon loss is 0.50 now vs 0.54 at ep110
- **AR synthesis still collapses** at temp=0.0 (only 1.25s output) — confirms the finding from memory that scheduled sampling is needed
- Temperature 0.5 keeps AR going for 6.4s but quality is likely noisy

The samples are in `/home/ubuntu/lewm-tts/samples_v3c/` — give them a listen. The core issue remains: the predictor trains well with teacher forcing (pred_loss=0.017) but free-running AR drifts to silence without scheduled sampling.


'/home/ubuntu/lewm-tts/train_v3c.log' run sampels on latest check[oint check ur memory